Questions of procedure

As a result of comments on my post Is Massively Collaborative Mathematics Possible? and also as a result of thinking about the proposal a little further I have a few extra remarks to make, and a slight redrafting of the procedural rules. (As I said before, these rules are just my first guess about what would work, and if a consensus emerges that they should change then they can of course be changed.)

Michael Nielsen, in this interesting response to my original post, felt that the rule I stated there as rule 10 was too restrictive. The rule said that one should keep posts focused and not change the subject. Now that I’ve thought of a mechanism, albeit a crude one, for starting new threads to the discussion, I’m inclined to agree and to want to be more flexible. So now if someone says something like, “The discussion has given me an idea for a different but related problem that we might try to tackle,” then as long as there is a genuine connection and people are interested in the related project, I see no harm in starting a new thread for discussing it. 

I said earlier that I would create all these threads in advance as empty posts. But now that Terry has pointed out to me that if I just put everything related to this project into one category then it will be easy for people to collect the relevant posts together, I’ve decided that I’ll just create new-thread posts as the need arises.

I’ve dropped the use of the word “stupid” in the rules (see rules 3-5 of the original post). That’s because I don’t want to encourage comments that are stupid in a tiresome way, as opposed to “intelligently stupid”. I hope the new rules 3-4 express what I meant slightly better. (I’m afraid the numbers have had to change. Let us say that if you refer to rule n then you mean rule n in this post unless you specify otherwise.)

The newly formulated rules.

1. The aim will be to produce a proof in a top-down manner. Thus, at least to start with, comments should be short and not too technical: they would be more like feasibility studies of various ideas.

2. Comments should be as easy to understand as you can possibly make them. For a truly collaborative project it is not enough to have a good idea: you have to express it in such a way that others can build on it.

3. When you do research, you are more likely to succeed if you try out lots of ideas, even if it is often the case that soon afterwards you can see why they never had a chance of working. Similarly, you are encouraged to express your mathematical ideas here, even if you have not spent time checking whether simple arguments show that they don’t work.

4. If they don’t work for an obvious reason, then almost certainly (if a lot of people are working together) someone will point that reason out very quickly. That is more efficient than you spending time in advance checking whether your ideas are good ones. Similarly, if you can’t immediately see how to make your idea precise, it may be more efficient to give it in a vague form and wait for somebody else to bring it into focus.

5. The ideal outcome would be a solution of the problem with no single individual having to think all that hard. The hard thought would be done by a sort of super-mathematician whose brain is distributed amongst bits of the brains of lots of interlinked people. So try to resist the temptation to go away and think about something and come back with carefully polished thoughts: just give quick reactions to what you read and hope that the conversation will develop in good directions. A good general rule to apply is this: don’t try to write a comment that requires you to think with a piece of paper (or a blackboard). 

6. If you are convinced that you could answer a question but that it would just need a couple of weeks to go away and try a few things out, then still resist the temptation to do that. Instead, explain briefly, but as precisely as you can, why you think it is feasible to answer the question and see if the collective approach gets to the answer more quickly. (The hope is that every big idea can be broken down into a sequence of small ideas. The job of any individual collaborator is to have these small ideas until the big idea becomes obvious — and therefore just a small addition to what has gone before.) Only go off on your own if there is a general consensus that that is what you should do.

7. Similarly, suppose that somebody has an imprecise idea and you think that you can write out a fully precise version. This could be extremely valuable to the project, but don’t rush ahead and do it. First, announce in a comment what you think you can do. If the responses to your comment suggest that others would welcome a fully detailed proof of some substatement, then write a further comment with a fully motivated explanation of what it is you can prove, and give a link to a pdf file that contains the proof.

8. Actual technical work, as described in 7, will mainly be of use if it can be treated as a module. That is, one would ideally like the result to be a short statement that others can use without understanding its proof.

9. One can summarize much of the above by saying that each comment should represent a “quantum of progress”. That is, the discussion should have been advanced in some small way that is not obviously complex or divisible.  

10. Rule 9 is a lower bound as well as an upper bound. A comment such as “I don’t think that idea will work” does not really advance the discussion. But if you say “I don’t think that idea will work because you would need a very strong analogue of such-and-such a result and nobody has any idea how to do that”  then it does, because it suggests possible ways for the conversation to continue. (People could then attempt to back up or alleviate your worries.)

11. Sometimes, comments such as “What you said in comment 32 sounded interesting, but I don’t quite see what you are driving at, and I think others probably don’t either,” advance the discussion. Again, the reason is that such a comment clearly demands a response, and the understanding of the problem is likely to be slightly greater after the response than it was before.

12. If at some point a clearly defined subdiscussion starts, then a new post should be written to summarize what has been said so far, and the subdiscussion should continue as a series of comments on that post.

13. Suppose the experiment actually results in something publishable. Even if only a very small number of people contribute the lion’s share of the ideas, the paper will still be submitted under a collective pseudonym with a link to the entire online discussion.

14. If it becomes clear that the discussion has run out of steam, then anything that is worth writing up will be written up (this may well be a collaborative process) and submitted to the arXiv, for use by anybody who wishes to use it.

15. Comments on the collaborative procedure should be carefully kept apart from the mathematical comments. Procedural comments should be attached to this post, and mathematical comments should be attached to the relevant mathematical posts.

A few more words about what is expected.

I am not 100% confident that this experiment will work, but I am very confident that something like this could in principle work. It is clear from the responses to my original post that many people share this confidence, probably because we have all read similar popular science books about artificial intelligence, large networks, and so on. I also have a more personal reason for this confidence, which is that I feel as though the kind of conversation I am advocating is very similar to the kind of conversation I have with myself when I am doing research on my own. (Although different people go about research in different ways, I would guess that many others do something similar to what I am about to describe.) For as long as possible I try to avoid doing any technical calculations: if the temptation arises, I try instead to look for heuristic arguments that will allow me to predict what the results of the calculations will be, or else for reasons to expect them not to be useful after all. Only if it has become very clear that actually doing a calculation is likely to lead to an insight that I can’t see how to obtain without doing it will I go ahead and do it (or try to do it). The aim of all this is to build up a plausible sketch of an entire argument, reducing the original problem from one big mystery to a series of exercises. In practice, it rarely works as smoothly as that, because very often I find a step plausible that I later discover to be wrong, sometimes after having used it as the foundation for quite a lot else. So the hard work of doing calculations, when I eventually get down to it, keeps feeding into the main sketch and forcing me to change it, sometimes quite drastically. But that just means that the result of a calculation can be to throw me back to the searching-for-a-plausible-sketch stage.

Now the precise calculations might not be all that easy to do collaboratively (though I could be wrong about this — perhaps there could be agreement that a certain lemma needs to be proved, and there could be some dialogue about how best to present the proof, after which the proof would more or less write itself, with people suggesting the next line, revising earlier lines, etc.). But everything else — the search for a sketch, backed up by heuristic arguments — does seem very naturally suited to a big collaboration. And I think it is fairly obvious from the description above that a big collaboration could potentially carry out a search of this kind much more efficiently than a single person. In particular, each heuristic argument would be subjected to quick scrutiny from a lot of people, so it if survived then it would have to be pretty good. So the experience I have often had, of doing detailed work on a sketch that was doomed to fail, would be much less likely.

I’ve written these last two paragraphs just to try to explain in a different way what I hope the contributions to this project will be like. The way I’ve presented my initial thoughts on the density Hales-Jewett theorem gives examples of the kind of thing I have in mind.

30 Responses to “Questions of procedure”

  1. luca Says:

    In rule (8) the reference to “actual technical work” should point to rule (7)

  2. luca Says:

    In rule ( 8 ), that is.

    [Many thanks — I’ve changed it. And the accidental smiley is too good to edit.]

  3. Procedural Question Says:

    I think this project is a great experiment, and I am eagerly waiting for the discussion to begin. However, away from the discussion of the math, I have an question about how we envision massively collaborative projects working. Say, Student X, is in year 3 of working on problem Y for her PhD dissertation. She has invested a lot of time and effort into the project and has none trivial progress, however isn’t near a complete result. Then, by coincidence, polymath13 takes this on as a project.

    Of course, being scooped is always a risk with research projects. However, this new massively collaborative situation presents a different sort of dilemma. Student X could contribute her progress to the polymath project. This seems to have two drawbacks. First if the polymath13 solves the problem using her ideas, she won’t get the CV line. As has been discussed, there is the advantage that she can direct prospective employers to the blog thread to review exactly what her contribution was. However, this may be a bit much to expect of hiring committees. Secondly, if polymath13 fizzles, but she goes on to resolve the problem herself, is her initial progress hers or polymath13’s?

    Conversely, assume that Student X elects not to participate in polymath13. Is she then on her honor to not read the polymath13 discussion? If she solves the problem using ideas very similar to polymath13 and submits the result for publication, should the referee just take her word that she didn’t consult the polymath13 forum?

    It seems in some ways, by making the project public (in contrast to a large private collaboration), it would (in practice) force individuals working on the problem into participating. It is an unfortunate aspect of our industry that, at least early in one’s career, credit for one’s work is essential to survival.

    I’m genuinely excited about this project (and have never thought about this problem myself before), however thought these might be useful issues to consider.

  4. A massively collaborative mathematical project « What’s new Says:

    […] opposed to the traditional model of a few very large contributions by a small number of people (see this article for more on the “rules of the game”).  I think this is an interesting experiment, and […]

  5. gowers Says:

    Dear Procedural Question, That’s definitely an important point. I have plans for more projects of this kind, and I’m fairly sure that this difficulty won’t arise for them. However, to be on the safe side I’ll try to leave a longer gap between saying roughly what the problem is and actually starting the discussion, and I’ll invite people to email me if there is a danger of putting someone in the difficult situation you describe.

  6. gowers Says:

    A quick report on my own experience. I have already found reading other people’s comments extremely stimulating, and I feel that my understanding of the problem has noticeably advanced in under 24 hours. Still absolutely no idea where it will all end, but I find the start promising.

    If things carry on as they are, then at some point I think I’ll write a post in which I make various suggestions about how to organize further comments (such as starting dedicated threads devoted to subproblems) and invite comments on those suggestions.

    And now I’ve got to go and type in yet another thought provoked by one of the existing comments.

  7. gowers Says:

    4.35pm: Everything I said in my previous comment applies, only more so. It feels as though this process is to normal research as driving is to pushing a car. However, the number of comments is already getting quite large, and I can imagine some people being put off by the thought of having to wade through them all. But already several subquestions have emerged. The question is, should we already be writing new posts about those subquestions and continuing discussion of them in properly separated threads? I wasn’t expecting to have to face this question so soon, but it’s probably a good idea to think about it quickly before the problem gets worse. Jason Dyer has already kindly had a go at classifying the various comments into different threads, which would make the organizational task easier. Opinions would be welcome, both on whether this needs to be done and on what the subthreads should be.

  8. Terence Tao Says:

    Dear Tim,

    This is indeed a promising start. It feels like the dynamic I have experienced with multi-author collaborations in the same room, but also resembles collaborations I have had with authors in different time zones (as one gets the uncanny feeling that progress is being made on the problem while one sleeps).

    I think we are beginning to bump up against the limits of the wordpress environment though. For instance, the only way to migrate comments from one thread to another is by hand, which is quite tedious. I would be hesitant to “fix” what isn’t “broken” yet, and so would leave the single massive thread as it is, at least until there is a clear divergence into two largely non-interacting subthreads, with one of them ripe for a recap post.

    One of the beauties of a digital medium such as this, though, is that any cleaning up of the organisational structure can always be done ex post facto; the important thing, at this stage, is just to get the thoughts and ideas entered into the system somewhere, and it can be sorted out later.

  9. Terence Tao Says:

    Incidentally, the following famous quote of Erdos seems apropos to our current project (though without the aliens):

    Aliens invade the earth and threaten to obliterate it in a year’s time unless human beings can find the Ramsey number for red five and blue five. We could marshal the world’s best minds and fastest computers, and within a year we could probably calculate the value. If the aliens demanded the Ramsey number for red six and blue six, however, we would have no choice but to launch a preemptive attack.

  10. Nathaniel Thurston Says:

    A quick note on the Procedural Question: If an experiment of this level of prominence is successful (and especially if the follow-on experiments meet with success), I would imagine that news of such a development would travel quickly within the mathematical community, other collaborative experiments will begin, and before too long it will become unusual to solve difficult problems alone.

    The problem of the CV line remains both important and unsolved.

  11. Mark Betnel Says:

    I’m trying to follow this even though I don’t understand the problem — the experiment is _awesome_. I hope that you will write some kind of analysis/review of the method and its evolution for a general audience — I can’t wait to try something similar in physics!

  12. Nathaniel Thurston Says:

    One thing which struck me as I tried to follow the polymath discussion itself it that it’s practically screaming for ‘threaded comments’, and ‘comment titles’, as at Groklaw. In the absence of these tools to allow me to focus my attention on those parts of the discussion I can follow, I’m finding it impractical to see the big picture without investing the time required to read and understand every comment.

  13. gowers Says:

    OK from now on I’m going to attach what one might think of as “thread titles” to various comments. In particular, all new comments of mine will have some kind of thread title, and I hope others will follow suite. This ought to make navigating through the comments easier.

    A number of interesting mathematical questions have been thrown up as a result of the discussion. With some of these there might be a case for a new post that briefly explains the question and why it came up, with the expectation that comments about that question would then go with that post. The advantage of that is that it would bring related comments closer together. A potential disadvantage of doing it in general is that the subconversations would become more isolated from each other and it would be harder to discuss connections between them. At this stage it’s not clear to me which of these considerations outweighs the other. (And of course they may not be the only considerations.)

  14. gowers Says:

    A very general comment about how things are going. I’d say that at the moment the collaboration is not “massive”, though it is much larger than anything I’ve ever been involved in before. That may be because the problem is insufficiently elementary. (I have one or two ideas about problems that are both elementary and extremely interesting, but that’s for the future.) But it occurs to me that even if it never becomes a massive collaboration, even the idea of doing research “in public” is one that could be valuable if it became common practice. For example, I often tell my research students that a large part of what one does when trying to solve a problem is formulate other questions. However, that’s a hard thing to explain properly because there is a lack of examples: by the time a result gets written up, all those questions along the way get forgotten about, except for the ones that ended up as part of the eventual solution. But if people routinely wrote public accounts (either collaboratively or individually) of how their ideas were developing, it would not only be fascinating to watch (especially on the rare occasions that somebody suddenly said, “Aha!” and a problem one was following became solved), but it would also provide a useful resource for someone who had that “I just don’t really know how to go about it” feeling about mathematical research.

  15. Nathaniel Thurston Says:

    I’m starting a discussion modeled after this discussion, with the basic goal of designing and creating a collaboration tool well suited to this sort of thing. In particular, I hope that the “collaborative programmer” can be of assistance in choosing and customizing the next home for the “collaborative mathematician”.

    I’ll be watching here, but if anyone has ideas about what’s missing from the simple blog format that would be useful in this sort of collaborative effort, or information about current forums/blogs/wikis/other tools that might be suited to the task, your input would be most welcome.

  16. gowers Says:

    I’m in a bit of a hurry now, but here, just to make the discussion more concrete, is a list off the top of my head of what some of the threads are.

    Upper and lower bounds for c_n.

    Trying to find different proofs of Sperner’s theorem with better chances of generalizing to density Hales-Jewett.

    Obstructions to uniformity and density arguments.

    Is there a new regularity lemma that would work inside the is-disjoint-from graph?

    General discussion of strategies for proving the theorem.

    Using Szemerédi’s original proof as a model.

    Some of the above are connected, and some of them (such as e.g. the Sperner thread) could be split into further subthreads (e.g. pair removal, pushing slices around, etc.).

    So two questions here. First, what should the threads be if we split into threads? Secondly, who’s prepared to write summaries? (I’m happy to do some of them — it’s probably fairly obvious which ones I feel most involved with.)

  17. Nielsen’s posts tagged Science2.0 | The Daily Clique Says:

    […] Questions of procedure from Gowers’s Weblog: Timothy Gowers updates the rules to his massively collaborative mathematics experiment. He posted the first project for this experiment a few days ago and has already received 145 comments. […]

  18. Upper and lower bounds for the density Hales-Jewett problem « What’s new Says:

    […] As with the rest of the project, this is supposed to be an open collaboration: please feel free to pose a question or a comment, even if (or especially if) it is just barely non-trivial.  (For more on the “rules of the game”, see this post.) […]

  19. Michael Nielsen » Update on the polymath project Says:

    […] I’m reminded of the famous Hardy-Littlewood rules for collaboration. Tim Gowers’ rules of collaboration have something of the same […]

  20. Gil Kalai Says:

    Perhaps an optimal way to do this is as follows: Have just two threads “background” and “forground”. “Forground” should represent the major avenue (s) in attacking the problem, and “background” should encourage related ideas, problems (and also amusing anekdotes, etc). From time to time each of these threads should be refreshed, with some summary and glossary (mainly because when they are too long the tex-ing takes ages) and the moderator can bring attention of people in the foreground thread to possibly relevant suggestions in the background one. It is useful to have a main thread reflecting the most promising and active effort and also to allow for wider contributions, but keeping more than two threads alive seems very difficult.

  21. Kristal Cantwell Says:

    The issue of computation has come up in a couple of the threads. I know that there are open source efforts in computer programming. It makes me wonder weather there eventually might be a combination of open source computer programming and massively collaborative mathematics in order to do difficult precise calculations that come up in efforts to try and solve the problem.

  22. A quick review of the polymath project « What Is Research? Says:

    […] to a combinatorial solution for the problem. Gowers wrote a background post about the problem and a post about the procedure, where he incorporated feedback from Michael Nielsen and others. These rules stipulated, among […]

  23. Michael Nielsen » The Polymath project: scope of participation Says:

    […] is a very large number for a mathematical collaboration – a few days into the project, Tim Gowers remarked that “this process is to normal research as driving is to pushing a car” – but it also falls […]

  24. Michael Nielsen » Introduction to the Polymath Project and “Density Hales-Jewett and Moser Numbers” Says:

    […] began the Polymath Project with a description of the problem to be attacked (see below), a list of rules of collaboration, and a list of 38 brief observations he’d made about the problem, intended to serve as […]

  25. KS3 // Ten in 2010 | Ways to work Says:

    […] * In Spring, 2009, a British mathematician Timothy Gowers used the comments feature of his blog to solve a complex mathematical problem collaboratively. After 7 weeks, Gowers announced the problem was solved with contributions from around 23 people. Read more. […]

  26. Discovery Science Research Group Says:

    Collaborative Mathematics in the Polymath Project…

    As part of my ongoing research to explore models for online collaboration I have recently been reviewing the Polymath project1 which exploits collaborative principals in resolving mathematical problems. The Polymath project created by the British …

  27. F1000 review: Open science is a research accelerator | Mario's Entangled Bank Says:

    […] {1} Gowers T, “Comment 1701” in “Gower’s Blog”, “ Feb 2009 (accessed 14 Nov 2011). {2} Raymond ES “The Cathedral and the Bazaar: Musings on Linux and Open […]

  28. Collaborative Protein Folding « wefold Says:

    […] how to choose the models for submission, etc. Let’s start with some of the ground rules set by Gowers for his Polymath project and then add more specific rules for this […]

  29. Science in the Open » Blog Archive » Network Enabled Research: Maximise scale and connectivity, minimise friction Says:

    […] and too hard. And had solved it by a route other than the one he had originally proposed. Gower’s commented: “It feels as though this is to normal research as driving is to pushing a […]

  30. Africa: Large Volumes of Data Are Challenging Open Science | AfricaHot Says:

    […] Tim Gowers Comment on the blog: Questions of procedure (, 2 February […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: