The results were complicated slightly by the fact that after a couple of days the post was linked to from the front page of Hacker News, and suddenly the number of people who voted more than tripled in a few hours. Furthermore, there was a bit of discussion about the polls, so it is not clear that the votes were completely independent. Also, the profile of an average Hacker News reader is probably somewhat different from the profile of an average reader of this blog.

Fortunately, I had recorded the votes shortly before this happened, so below we (“we” means Mohan Ganesalingam and I) present both sets of results. As it happens, the proportions didn’t change too much. We begin with some bar charts. U stands for “undergraduate”, G for “graduate”, C for “computer” and D for “don’t know”. The portion coloured in blue represents people who claimed to be sure that they were correct, and the portion coloured in red represents those who were unsure.

At the end of the post we give the exact numbers.

A brief remark before we present the results is that none of the three “contestants” was explicitly trying to write proofs that would appear as human as possible. The two humans were asked to provide proofs and not told why. And the program was designed to produce passable write-ups, but not to fool people into thinking that those write-ups were written by a human. (There are some easy improvements that could have been made if we had intended to do that, but we did not originally envisage carrying out this second experiment.)

The general picture can be summarized as follows.

1. The computer was typically identified by around 50% of all those who voted.

2. Typically around half of those were confident that they were correct, and half not so confident.

3. A non-negligible percentage of respondents claimed to be sure that a write-up that was not by the computer was by the computer.

The results in bar-chart form

Mohan Ganesalingam

There is quite a lot to say about these results, and also about the program. Most of the rest of this post is written jointly with my collaborator Mohan Ganesalingam (as indeed were the posts with the two experiments — we chose the wording very carefully), but first let me say a little about who Mohan is.

Officially he is a computer scientist who got a PhD in computational linguistics about three years ago. However, as an undergraduate he was a mathematician, and not just any old mathematician: he was Senior Wrangler in his year, which puts him in some illustrious company, and he could certainly have gone on to do research in mathematics if he had been so inclined. However, he took a different and more interesting path, beginning with an MA in Anglo-Saxon, where he won a university prize for the best results in the Cambridge English Faculty, and continuing with a PhD in computer science, where he gave a complete analysis of formal mathematical language. Rather than say what “complete analysis” means, I’ll just say that with Tom Barnet-Lamb (another former Senior Wrangler, who will be known to algebraic number theorists for some major recent papers with Toby Gee, David Geraghty and Richard Taylor — I suppose I should mention that Toby Gee is yet another former Senior Wrangler; I don’t know about Richard Taylor) he has written a program that can take as its input mathematical text such as

“A closed subset of a compact set is compact.”

and convert it into a logical format such as

set
……………compact
……………subset_of
……………closed_in
……………
……………compact

Mohan and Tom were interested in mathematical language primarily from the point of view of linguistics: the language that mathematicians use is simple enough to be tractable, but complex enough to involve many of the interesting difficulties that linguists face when studying natural language. However, as will be obvious, their program has many interesting potential applications.

Regular readers of this blog and, before the blog existed, of my web page may be aware that for a long time I have believed that at least the easy parts of mathematics could be done by a computer without the need for major leaps forward in artificial intelligence. I always thought that I would love to be able to do more than just say, “This surely ought to be possible,” but knew that I would never get round to doing it on my own. When Mohan became a research fellow at Trinity College, it was clear that if I ever wanted to take part in producing an actual program, I had an opportunity that was unlikely to be repeated, since Mohan was unbelievably well qualified for the project. Better still, he too was very interested in the idea of writing a theorem-proving program, and his ideas about how such a program should work were very similar to mine.

How the collaboration has worked so far

So we started collaborating about three years ago (shortly before Mohan got his PhD). Initially, most of the collaboration was by email, but after a while we moved over to what one might call “Duomath,” by which I mean a private blog where we wrote posts and commented on each other’s posts. At the last count, that blog had reached over a million words. We also met regularly for face-to-face conversations about where things were going.

During the three years of our collaboration, we talked about “the program” where what we meant was an algorithm that we had specified fairly precisely but not actually written as a program. We ended up radically redesigning this “program” at least twice: typically it would work well to start with, but then we would find ourselves dissatisfied with certain aspects of it — for example, sometimes there would be a rule governing its behaviour that we chose not because we had a good theoretical justification for it, but because it seemed to deal well with all the example problems that we were considering, and then we would think of examples where the rule led to “stupid” behaviour on the part of the program. But towards the end of 2012, Mohan, who was coming to the end of his fellowship and therefore badly needed publications, insisted that we should write an actual program and not be too perfectionist about it. We started work in January, and after ten days the program was up and running. We then worked on the writing-up side of things for another ten days, so the whole thing took three weeks. I say this to emphasize that the hard work had gone on during the preceding three years.

Another point we are keen to make is that we put almost no effort into the write-up part of the program. That is, we spent those 10 days on it, but the three years of preparation were devoted exclusively to the problem-solving part. The reason we care about this point is that we want to be able to claim that our program tackles problems in a human way. The fact that rather crude methods could be used to convert the program’s main output into a human-style write-up helps us to back up that claim.

It is also worth repeating that when we created the write-up part of the program, we were not doing so with the second experiment in mind: that is, we were not planning to ask people to see whether they could distinguish between human write-ups and our program’s write-ups. However, after the first experiment we felt an obligation to do so. If we had intended to make our write-ups indistinguishable from human write-ups, then at the very least we would made them more varied — for example, not ending them all in the same way with the words “we are done”. So the fact that a lot of people still got the answer wrong took us by surprise.

A rough idea of how the program works

To begin with we would like to be clear that nothing in this program is radically new: almost all the techniques we use would, for instance, have been familiar to Woody Bledsoe, a pioneer in the use of “human methods” in automatic theorem proving. We believe that the way we have come up with the program is new, but for somewhat subtle reasons that we will not go into in this blog post. One effect of this is that it is easy for us to produce human-readable write-ups, which as far as we are aware has not been done before. We hope that in the not too distant future our new approach will also be rewarded by a program that can solve problems that no previous program has managed to solve, but we do not make that claim of the program we have so far.

Broadly speaking, what the program does is start with a collection of statements, some of which we call hypotheses (that is, things it is allowed to assume) and some of which we call targets (that is, things it is trying to prove). It then continually modifies this collection of statements until it reaches an obvious stopping point. Depending on precisely how we decide to implement the algorithm, this stopping point could be when it has no targets left, or when it has replaced all targets by DONE, or when all targets are also present as hypotheses: at any rate, it stops when the problem is clearly solved.

Of course, it won’t solve every problem it is given, so another major reason for it to halt is that it can’t see anything to do.

Our main priority when writing the program was that the steps it took should be ones that a human would naturally take. One could express that in the form of a slogan: the program should not do silly things.

One thing that humans do not do is rewrite all their statements in low-level language and work directly from the axioms of set theory. So our program doesn’t do that either. It is happy to deal with high-level concepts, and it also has a (so far very small) library of statements that it is allowed to assume.

A worked example

If we give an example of how the program works through a problem, it will greatly clarify what we have just said. Let us take Problem 2 from the list, the one that asks you to show that if is continuous and is open then is open. The initial input to the program would be this.

is continuous
is open
———————
is open

There would also be some “background information” saying that and are metric spaces and that . And the program has a library that includes things like the epsilon-delta definition of continuity and the definition of open sets. Many people commented on the fact that the program didn’t use open balls. This is not some fundamental fact about the program — we could have got it to reason using open balls by changing not the program but the data it worked with. To do this, we would have had to tell it the open-balls definitions of continuity and open sets, and a few basic facts about open balls such as that if then .

However, let us accept that in this instance the program is aiming to behave like a human who wants to go back to epsilons and deltas. In that case, the natural first step is to expand the target — that is, to say in lower-level language what needs to be proved if we want to show that is open. The answer is that it needs to show that if , then there is some such that whenever . So the program moves to the following collection of statements. (Of course, the way it does that can be broken up further, but it is sufficiently obvious that that is a purely mechanical process that we will not go into that level of detail.)

is continuous
is open

———————

Humans typically rewrite (or rethink) statements of the form in the form . We therefore have a rule that says that statements of the form should be rewritten in that way. So the program quickly does that.

is continuous
is open

———————

However, it doesn’t do it to the substatement because in our judgment a typical human wouldn’t be thinking about that yet. Instead, it spots that the statement is exactly the kind of “input” that is demanded by the statement “ is open”, and it goes ahead and applies that statement.

is continuous

———————

A few comments are in order here. First, one might think that the new hypothesis should begin with , but the program automatically applies an operation that we call “peeling” to get rid of that existential quantifier. This operation corresponds to the kind of reasoning that humans do without thinking: we establish a statement like and from that moment we refer to as though it has been chosen, without bothering to say, “Let be such that .” We want the program to behave in a human way, so it too quietly gets rid of existential quantifiers that begin hypotheses. (In a similar way it “peels” universal quantifiers in front of targets. This corresponds to the human practice of taking a target of the form , saying, “Let be such that ,” and concentrating on proving .)

Another point to mention is that the hypotheses and “ is open” have been perceived by the program as “used up” and therefore deleted. This is because a typical human would be aware that these statements are very unlikely to be needed again. Deletion of hypotheses is not an entirely straightforward matter, because there are many circumstances in which a statement is used more than once. However, we have developed some rules that do a reasonable job of deleting hypotheses when a human would think they were obviously used up. (These rules work OK for now, but will undoubtedly need to be replaced by more sophisticated rules when we start to tackle harder problems.)

Deleting hypotheses is not something that the program needs in order to solve these problems successfully. This is an example of how our aims are different from conventional automatic theorem provers: we are very keen to model human mathematicians, so we have thought hard about how it is that they are aware that statements are “used up”, even though the advantage of doing so (in terms of time saved by not looking at those statements again when searching for further steps to take) is negligible.

The program is now slightly stuck, so it performs an operation we call “suspension”: it pretends that it has chosen , so that it can get inside the existential quantifier that begins the target and start breaking up the conditional statement inside. To indicate that is yet to be chosen, it is marked with a bullet. (Actually, this is not precisely what the program does, but it is similar, and to explain what the program does that is slightly different from this would take us to an unnecessary level of detail.) After suspension, the problem looks like this.

is continuous

——————————–

However, the program immediately peels and splits up the conditional, so we get to this.

is continuous


———————

It is important that, like a human, the program should be aware that is not allowed to depend on . That is what is indicated by the notation .

Now that the statement has been isolated, it is automatically rewritten.

is continuous


———————

The program is now in a position to do some “backwards reasoning”: that is, replacement of a target by a new target when there is a hypothesis . So it does it and deletes the second hypothesis.

is continuous

———————

Now it can do some more backwards reasoning (of a slightly more complicated nature because there are more quantifiers involved) using the continuity hypothesis, so it does that. It knows the expansion of “ is continuous” in terms of epsilons and deltas and uses that and a matching algorithm to spot that it can do this, but it doesn’t actually expand the definition, since an experienced human normally wouldn’t bother to do so.

———————-

Here the notation indicates that depends on and . (It seems that humans are more conscious of dependencies for variables that arise in certain ways and independencies for variables that arise in other ways, and that turns out to be convenient when writing the program too.)

Finally, the program notes that it can get the one remaining hypothesis to match the target if it sets equal to . It also checks that is allowed to depend on and , which it is. (All that happens here is that it checks that the list of variables that must not depend on is disjoint from the list of variables that depends on.)

Writing up

We stated earlier that the writing-up part of the program is quite crude. The main properties that the writing up has (neither of which we shall attempt to define precisely) are that it is local and faithful. The second property means that the sentences that appear in the write-up are basically translations of things that the program does in the same order, with none of the program’s working hidden, and the first means that the each sentence of the write-up basically depends just on one step, or perhaps a sequence of one or two adjacent steps, that the program takes.

So the first thing the write-up algorithm does is associate with each step a translation. For example, if the step is to go from

has property

to

where the statement “ has property ” expands to

then the write-up algorithm will translate that into something like, “Since and has property , .”

At this point one could get a horrible but just about comprehensible write-up by simply concatenating all these individual translations. However, it would have several defects. Most notably, it would be repetitive in a way that no human would. For instance, it would say things like this: “Let . Since , it follows that . Since and is open, it follows that there exists such that …”

We deal with that by removing hypotheses from sentences when they have only just been established and are therefore already “ringing in the ears” of the reader. So the above fragment might be reduced to “Let . Then . Since is open, it follows that there exists such that …” Actually, it is not reduced to exactly that, but we are already pretty close.

Other tweaks we performed were messing about with the rules for how to say that one thing follows from another (do we say “Since , ,” or “. Therefore .” or “We know that . It follows that .” or what?) until the write-ups that the program produced were not too repetitive and had the right sort of flow. By “flow” we mean that sometimes one has a chain of forward steps, and one wants to convey that one is chugging along through the proof, whereas at other times the forwards reasoning gets stuck and one has to turn to backwards reasoning and there should be more of a feeling of a new idea coming in or some almost forgotten hypothesis being resurrected. Words like “but” and “now” help to convey these kinds of things.

And that is more or less all there was to it. We have not (yet) done a proper job on the write-up algorithm, in the sense that we have not sat down and carefully analysed things like what it is that makes humans decide to write “but” or “it follows that” when they do. As a result, if you give the program new problems to solve, it tends to produce write-ups with slightly weird wording in places. So in that sense it is not fully robust: we have shown you the output that looks nice, but if you had a chance to choose your own input you would immediately produce output that looks slightly stilted in the ways just described, though it would be comprehensible and would still be English rather than some special-purpose formal language.

How the program chooses what to do next

The basic method we use is extremely simple. We give the program an ordered list of operations it can do, many of which were used in the worked example above. It then behaves greedily: that is, it repeatedly chooses the first operation it can do. The word “greedily” is a slight exaggeration though: for one thing we have chosen the ordering quite carefully, and for another some of the moves have constraints that must be satisfied if they are to be done.

We do not expect this very rudimentary architecture to survive for ever, but it works well for routine problems. (In a moment we shall say more precisely what we mean by “routine”.)

Constraints on the program

Going back to the proof-finding aspect of the program, there are three main constraints that we have imposed on ourselves. The first two are concerned with making it behave in a human way.

1. The program should not find a step hard if an experienced human finds it the obvious thing to do.

2. The program should not be willing to do a step if an experienced human would be very reluctant to do it.

The point of the first constraint is clear: if a human finds a step obvious but the program finds it only after some big search with a lot of backtracking, say, then we have not understood something about how humans do mathematics. (That is not quite true: it could be that we understand how humans do it, but find it hard to model — that may well be the case, for instance, if visualization plays a big role, but we are avoiding that problem for the time being.)

We (informally) define a problem to be routine if an experienced human can solve the problem easily without backtracking. Combining this definition with the first constraint above, we obtain the following consequence.

The program should be able to solve routine problems without backtracking.

A big advantage of this no-backtracking principle is that if you give the program a statement that it cannot prove (possibly because it is false), then it does a few sensible steps and gets stuck, rather than producing pages and pages of garbage.

As for the second constraint, the kind of thing we want to rule out is an undue willingness to use brute force, whether that means big messy calculations or large searches. This “undue willingness” often arises when one tries to produce methods that are too general. To give a very simple example, a pocket calculator when asked to multiply 30 by 70 will probably use the same method that it uses when asked to multiply 29 by 73. To give a slightly less simple example, if an algebra package was given the equation

to solve, it might well expand the brackets and collect terms before starting. If it did so, then even if it had no trouble with the resulting calculations it would fall foul of our second constraint, since a human would instantly see that the equation was easier to solve in the form in which it is already presented.

The third constraint has a rather different justification. It is intended to stop us “cheating” by hardwiring solutions of problems into the program itself.

3. The methods that the program uses should not be specific to any domain of mathematics. Any domain-specific behaviour should be the result of data that the program is given to work with.

Why impose any constraints at all? Surely if one is trying to produce a program that can prove as many theorems as possible, the fewer constraints there are the better?

The simple answer to that is that we are not trying to produce a program that can prove as many theorems as possible. Rather, we are trying with our program to build a foundation for more elaborate programs. We strongly believe that if we can create a program that solves easy problems the “right” way, then it will be much easier to extend it to programs that can solve harder problems.

A few things that the program cannot (yet) do

There are many very important things that the program cannot do. Here we list just a few of our immediate targets.

1. Second-order quantification

We said in the previous post that the program cannot solve non-routine problems and in fact cannot solve more than a small class of routine problems. However, during the last three years we have worked out rather more than is exhibited in this program. For example, we are close to being able to write a program that can handle second-order quantification. That will allow it to solve two major classes of problems: ones that involve producing sequences, and ones that involve producing open covers. A target problem that we essentially know how to do is the proof that a compact subset of a Hausdorff topological space is closed.

2. Algebra

At the moment, the level of granularity at which the program works is that of statements. For example, if it has a statement and a statement , then it notes the match between and and generates the statement . However, a great deal of mathematical reasoning, especially in algebra, takes place at the level of terms: you have a term that you want to make into another term by performing various allowable operations. Our program does not yet do this, and therefore there is a large class of routine problems that it is not yet capable of solving.

Dealing with equality is a well-known difficulty in the automatic theorem proving literature. In many approaches the problem is that without big restrictions on when you can substitute one equal quantity for another, a program is liable to make many irrelevant observations. It seems, however, that many of the ideas that have gone into our program so far can be modified to deal with term rewriting for sufficiently routine problems. We hope to make an attempt in this direction in the not too distant future.

One option that we do not wish to pursue is using a standard package such as Maple or Mathematica to do algebraic manipulations for us. This would be completely contrary to the spirit of our program, since it would not be modelling the way humans rewrite terms while solving routine problems. In the short term, it might be taking the easy way out, but in the longer term we would pay for not adequately understanding what human mathematicians do. (Of course, sometimes even humans have to do very complicated calculations. In such situations, a program could always say, just as a human might, “This calculation looks as though the answer would be useful, but it’s very complicated. I think I’ll use an algebra package.” But this would be regarded as an extremely costly option.)

3. Non-trivial existence problems

If the program needs to prove a statement that begins with an existential quantifier, then at the moment it has three basic methods, the first of which is a trivial case of the second. For the second, recall that suspension is the operation of putting a bullet on a variable to indicate that you are postponing the decision about what to substitute for it.

1. Direct substitution: it needs to prove a statement of the form and it has a hypothesis of the form , so it sets .
2. Direct substitution after suspension and simplification: it needs to prove a statement of the form , suspends the variable , carries out some steps, and then at a later stage finds that it can substitute directly for .
3. Use of the library: it has a few “standard” solutions in the library, such as the fact that if and are positive you want a positive number that is at most as big as and at most as big as , then you can take . (It used that at the end of its discovery of a proof that the intersection of two open sets is open.)

It is surprising how far one can get with such simple methods, but there are many other common techniques. For example, a method that works for some simple problems in group theory is looking through a list of a few very basic examples of groups to see whether any of them happens to have the desired property. Another method that works in many situations is what one might call “guess and adjust”. That is, you begin with an example that you don’t expect to work, but you hope is in the right ball park. You then try to identify what needs to be changed to make it work. And many examples are not built or even approximately built in one go but are “constructed” using basic building blocks and methods for combining or extending them. We would certainly like to introduce more advanced techniques like these, but for now there are more urgent priorities.

4. Backtracking

We have already mentioned that our program does not backtrack. This is a different kind of inability from the ones above, since it does not concern routine problems (by our definition of “routine”). Quite clearly, humans backtrack constantly when they are solving non-routine problems. We would not be interested in this project if we did not hope eventually to tackle such problems, but that will require a great deal of thought: we will need to find a way of allowing the program to backtrack without indulging in “unreasonable” searches. Probably we will start by allowing it, under very controlled circumstances, to try slightly risky steps that will either be confirmed as good almost immediately or be taken back. More or less equivalently, we might occasionally allow the program to look a couple of steps ahead. An example of the kind of problem where this could be useful is in proving the cancellation law in group theory. We are given that and want to prove that . Multiplying both sides on the left by makes the expressions more complicated, but if we look slightly further ahead we see that we get exactly what we want. (This is, however, a rather tricky example: it seems that what makes it easy for humans is the analogy with cancellation laws in arithmetic.)

The numbers in the polls

The numbers in brackets represent the numbers before the Hacker-News invasion. I have now made the results of the polls visible, so you can check that the main set of results are as we say they are.

Problem 1. (a=undergraduate, b=graduate, c=computer)

The computer-generated output is definitely (a): 123 votes (36)
I think the computer-generated output is (a) but am not certain: 156 votes (48)
The computer-generated output is definitely (b): 83 votes (12)
I think the computer-generated output is (b) but am not certain: 139 votes (27)
The computer-generated output is definitely (c): 266 votes (71)
I think the computer-generated output is (c) but am not certain: 362 votes (100)
I have no idea which write-up was computer generated: 131 votes (11)

Problem 2. (a=computer, b=undergraduate, c=graduate)

The computer-generated output is definitely (a): 140 votes (45)
I think the computer-generated output is (a) but am not certain: 133 votes (55)
The computer-generated output is definitely (b): 42 votes (13)
I think the computer-generated output is (b) but am not certain: 75 votes (32)
The computer-generated output is definitely (c): 10 votes (4)
I think the computer-generated output is (c) but am not certain: 34 votes (14)
I have no idea which write-up was computer generated: 42 votes (13)

Problem 3. (a=graduate, b=computer, c=undergraduate)

The computer-generated output is definitely (a): 24 votes (9)
I think the computer-generated output is (a) but am not certain: 39 votes (18)
The computer-generated output is definitely (b): 72 votes (17)
I think the computer-generated output is (b) but am not certain: 89 votes (32)
The computer-generated output is definitely (c): 42 votes (19)
I think the computer-generated output is (c) but am not certain: 63 votes (36)
I have no idea which write-up was computer generated: 48 votes (21)

Problem 4. (a=undergraduate, b=graduate, c=computer)

The computer-generated output is definitely (a): 27 votes (12)
I think the computer-generated output is (a) but am not certain: 52 votes (29)
The computer-generated output is definitely (b): 12 votes (4)
I think the computer-generated output is (b) but am not certain: 32 votes (16)
The computer-generated output is definitely (c): 82 votes (22)
I think the computer-generated output is (c) but am not certain: 79 votes (41)
I have no idea which write-up was computer generated: 38 votes (16)

Problem 5. (a=graduate, b=undergraduate, c=computer)

The computer-generated output is definitely (a): 12 votes (5)
I think the computer-generated output is (a) but am not certain: 17 votes (7)
The computer-generated output is definitely (b): 19 votes (6)
I think the computer-generated output is (b) but am not certain: 46 votes (23)
The computer-generated output is definitely (c) 110 votes (41)
I think the computer-generated output is (c) but am not certain: 115 votes (52)
I have no idea which write-up was computer generated: 49 votes (15)

Over the last three years, I have been collaborating with Mohan Ganesalingam, a computer scientist, linguist and mathematician (and amazingly good at all three) on a project to write programs that can solve mathematical problems. We have recently produced our first program. It is rather rudimentary: the only problems it can solve are ones that mathematicians would describe as very routine and not requiring any ideas, and even within that class of problems there are considerable restrictions on what it can do; we plan to write more sophisticated programs in the future. However, each of the problems in the previous post belongs to the class of problems that it can solve, and for each problem one write-up was by an undergraduate mathematician, one by a mathematics PhD student and one by our program. (To be clear, the program was given the problems and produced the proofs and the write-ups with no further help. I will have more to say about how it works in future posts.) We wanted to see whether anybody would suspect that not all the write-ups were human-generated. Nobody gave the slightest hint that they did.

Of course, there is a world of difference between not noticing a difference that you have not been told to look out for, and being unable to detect that difference at all. Our aim was merely to be able to back up a claim that our program produces passable human-style output, so we did not want to subject that output to full Turing-test-style scrutiny, but you may, if you were kind enough to participate in the experiment, feel slightly cheated. Indeed, in a certain sense you were cheated — that was the whole point. It seems only fair to give you the chance to judge the write-ups again now that you know how they were produced. For each problem I have created a poll, and each poll has seven possible answers. These are:

The computer-generated output is definitely (a).
I think the computer-generated output is (a) but am not certain.
The computer-generated output is definitely (b).
I think the computer-generated output is (b) but am not certain.
The computer-generated output is definitely (c).
I think the computer-generated output is (c) but am not certain.
I have no idea which write-up was computer generated.

I would also be interested in comments about how you came to your judgments. All comments on both experiments and all votes in the polls will be kept private until I decide that it is time to finish the second experiment. A small remark is that I transcribed by hand all the write-ups into a form suitable for WordPress, so the existence of a typo in a write-up is not a trivial proof that it was by a human.

If you did not participate in the first experiment but nevertheless want to try this one, that’s fine. [Update: I have now closed the polls. Very soon Mohan and I will post the results and a discussion of them. Further update: The results now appear below. They appear displayed in possibly a more convenient way in this post, which also contains a discussion of how the program works.]

Problem 1. Let and be open sets in a metric space. Then is open.

1(a) We want to show that for all in , there exists such that the open ball is contained in .

Let . is in , so is in and is in . Since and are open, there exist such that and . Let . Then is contained in , so it’s contained in . Similarly, is contained in , so it’s contained in . So . So is open.

1(b) For arbitrary , let . Consider an arbitrary . As are open there are such that and . Take . Then and . So . We’ve proved that for any there is an open ball ( in this case) that contains and is inside . So is open.

1(c) Let be an element of . Then and . Therefore, since is open, there exists such that whenever and since is open, there exists such that whenever . We would like to find s.t. whenever . But if and only if and . We know that whenever and that whenever . Assume now that . Then if and if . We may therefore take and we are done.

Problem 2. Let and be metric spaces, let be continuous, and let be an open subset of . Then is an open subset of .

2(a) Let be an element of . Then . Therefore, since is open, there exists such that whenever . We would like to find s.t. whenever . But if and only if . We know that whenever . Since is continuous, there exists such that whenever . Therefore, setting , we are done.

2(b) Let . We seek such that the open ball is contained in .

, so . is open, so we know that for some , . Since is continuous, there exists such that for all , ; i.e., . So if ; i.e., . So is open.

2(c) Take any . We have . As is open, there is an open ball in . Because is continuous, there is some such that for any , belongs to . Hence, for such , . So . So . We’ve proved that every point in has an open ball neighbourhood. So is open.

Problem 3. Let be a complete metric space and let be a closed subset of . Then is complete.

3(a) Consider an arbitrary Cauchy sequence in . As is complete, has a limit in . Suppose . Because is closed, belongs to . We’ve proved that every Cauchy sequence in has a limit point in . So is complete.

3(b) Let be a Cauchy sequence in . Then, since is complete, we have that converges. That is, there exists such that . Since is closed in , is a sequence in and , we have that . Thus converges in and we are done.

3(c) Let be a Cauchy sequence in . We want to show that tends to a limit in .

Since is a subset of , is a Cauchy sequence in . Since is complete, , for some . Since is a closed subset of , it must contain all its limit points, so . So in . So is complete.

Problem 4. Let and be metric spaces and let and be continuous. Then the composition is continuous.

4(a) Let , and let . We need to show that there exists such that for all , .

is continuous, so there exists such that for all , . is continuous, so there exists such that for all , . But then , as desired. So is continuous.

4(b) Take an arbitrary . Let and . Using continuity of , for any , there is some such that if (for ), then . As is continuous, there is some such that if (for ), then . So for any we’ve found such that if , then and therefore . Hence is continuous.

4(c) Take and . We would like to find s.t. whenever . Since is continuous, there exists such that whenever . Since is continuous, there exists such that whenever . Therefore, setting , we are done.

Problem 5. Let and be sets, let be an injection and let and be subsets of . Then .

5(a) Take . So there is some and such that . As is injective, and are equal. So . So .

5(b) Suppose . Then, for some , and . So . Since is injective, , so , so . So .

5(c) Let be an element of . Then and . That is, there exists such that and there exists such that . Since is an injection, and , we have that . We would like to find s.t. . But if and only if and . Therefore, setting , we are done.

Most of this post consists of write-ups of proofs of five simple propositions about metric spaces. There are three write-ups per proof, and I would be very grateful for any comments that you might have. If you would like to participate in the experiment, then please state your level of mathematical experience (the main thing I need to know is whether you yourself have studied the basic theory of metric spaces) and then make any comments/observations you wish on the write-ups. The more you say, the more useful it will be (within reason). I am particularly interested in comparisons and preferences. For each proof, the order of the three write-ups has been chosen randomly and independently.

It would also be useful if you could rate each of the 15 write-ups for clarity and style. So that everyone rates in the same way, I suggest the following rating systems.

Clarity.

-2 very hard to understand
-1 hard to understand
0 neither particularly hard nor particularly easy
1 easy to understand
2 very easy to understand

Style.

-2 very badly written
-1 badly written
0 neither badly written nor well written
1 well written
2 very well written

I stress that ratings should not be regarded as a substitute for comments and observations, or vice versa. What I really need is both comments and numerical ratings.

I do not want people to be influenced by the answers that other people give, so all comments on this post will go to my moderation queue. When I have enough data for the experiment, probably in a week or so, I will publish all the comments (unless for some reason you specifically request that your comment should not be published).

The more people who participate, the more reliable the results of the experiment will be. I realize that it may take a little time, so thank you very much in advance to everybody who agrees to help. (Update 26th March: I now have over 30 responses; they have been very helpful indeed, so I am extremely grateful for those. If they keep coming in at a similar rate over the next few days it will be wonderful.)

The write-ups.

Problem 1. Let and be open sets in a metric space. Then is open.

1(a) We want to show that for all in , there exists such that the open ball is contained in .

Let . is in , so is in and is in . Since and are open, there exist such that and . Let . Then is contained in , so it’s contained in . Similarly, is contained in , so it’s contained in . So . So is open.

1(b) For arbitrary , let . Consider an arbitrary . As are open there are such that and . Take . Then and . So . We’ve proved that for any there is an open ball ( in this case) that contains and is inside . So is open.

1(c) Let be an element of . Then and . Therefore, since is open, there exists such that whenever and since is open, there exists such that whenever . We would like to find s.t. whenever . But if and only if and . We know that whenever and that whenever . Assume now that . Then if and if . We may therefore take and we are done.

Problem 2. Let and be metric spaces, let be continuous, and let be an open subset of . Then is an open subset of .

2(a) Let be an element of . Then . Therefore, since is open, there exists such that whenever . We would like to find s.t. whenever . But if and only if . We know that whenever . Since is continuous, there exists such that whenever . Therefore, setting , we are done.

2(b) Let . We seek such that the open ball is contained in .

, so . is open, so we know that for some , . Since is continuous, there exists such that for all , ; i.e., . So if ; i.e., . So is open.

2(c) Take any . We have . As is open, there is an open ball in . Because is continuous, there is some such that for any , belongs to . Hence, for such , . So . So . We’ve proved that every point in has an open ball neighbourhood. So is open.

Problem 3. Let be a complete metric space and let be a closed subset of . Then is complete.

3(a) Consider an arbitrary Cauchy sequence in . As is complete, has a limit in . Suppose . Because is closed, belongs to . We’ve proved that every Cauchy sequence in has a limit point in . So is complete.

3(b) Let be a Cauchy sequence in . Then, since is complete, we have that converges. That is, there exists such that . Since is closed in , is a sequence in and , we have that . Thus converges in and we are done.

3(c) Let be a Cauchy sequence in . We want to show that tends to a limit in .

Since is a subset of , is a Cauchy sequence in . Since is complete, , for some . Since is a closed subset of , it must contain all its limit points, so . So in . So is complete.

Problem 4. Let and be metric spaces and let and be continuous. Then the composition is continuous.

4(a) Let , and let . We need to show that there exists such that for all , .

is continuous, so there exists such that for all , . is continuous, so there exists such that for all , . But then , as desired. So is continuous.

4(b) Take an arbitrary . Let and . Using continuity of , for any , there is some such that if (for ), then . As is continuous, there is some such that if (for ), then . So for any we’ve found such that if , then and therefore . Hence is continuous.

4(c) Take and . We would like to find s.t. whenever . Since is continuous, there exists such that whenever . Since is continuous, there exists such that whenever . Therefore, setting , we are done.

Problem 5. Let and be sets, let be an injection and let and be subsets of . Then .

5(a) Take . So there is some and such that . As is injective, and are equal. So . So .

5(b) Suppose . Then, for some , and . So . Since is injective, , so , so . So .

5(c) Let be an element of . Then and . That is, there exists such that and there exists such that . Since is an injection, and , we have that . We would like to find s.t. . But if and only if and . Therefore, setting , we are done.

In an ideal world, I would use the feature just for that post. However, as far as I can tell, my only options are allowing all comments, moderating all comments, or disabling comments completely on individual posts. Sending comments to the moderation queue on a post-by-post basis doesn’t seem to be possible, but if anyone knows a way, then I’d be very pleased to hear about it. Assuming there isn’t a way, then for a short while, all comments on this blog will be moderated, but I will try to approve comments on other posts regularly, so I hope this won’t be too annoying.

Update. Good job I did this test. I changed the relevant setting but didn’t click “Save settings”. Hence the three comments below.

Further update. OK, now it seems to be working just fine. Many thanks to those who sent test comments. I’ll put up the new post later this evening (British time).

I was told to write the article as I pleased — the articles in the first volume are very different in style from each other — so I went for a style that was not unlike what I might have written if I had wanted to present several of Szemerédi’s results in a series of blog posts. That is, I’ve tried to explain the ideas, and when the going gets tough I have skipped the details. So it seems appropriate to post the article on this blog.

If you look at it and happen to notice any typos, false statements, wrong emphases, etc., I think it isn’t too late to make changes, so I’d be grateful to hear about them. Here is the article.

Before I go any further, I should say that the topic in question is one about which I am not an expert, so it may well be that the answer to the question I’m about to ask is already known. I could I suppose try to find out on Mathoverflow, but I’m not sure I can formulate the question precisely enough to make a suitable Mathoverflow question, so instead I’m doing it here. This has the added advantage that if the question does seem suitable, then any discussion of it that there might be will take place where I would want any continuation of the discussion to take place.

Fast parallel sorting.

I am in the middle of writing an article about Szemerédi’s mathematical work (for a book about Abel Prize winners), and one of his results that I am writing about is a famous parallel sorting network, discovered with Ajtai and Komlós, that sorts objects in rounds. This is a gorgeous result, but unfortunately the constant is quite large, and remains so after subsequent improvements, which means that in practice their sorting network does not work as well as a simpler one that runs in time for a much smaller .

The thought I would like to explore is not a way of solving that problem — obtaining the AKS result with a reasonable constant — but it is closely related. I have a rough idea for a randomized method of sorting, and I’d like to know whether it can be made to work. If it gave a good constant that would be great, but I think even proving that it works with any constant would be nice, unless it has been done already — in which case congratulations to whoever did it and I very much like your result.

The rough idea is this. As with any fast parallel sorting method, the sorting should take place in rounds, where each round consists in partitioning all the pairs (or a constant fraction of them, but there isn’t much to be lost by doing extra comparisons) and swapping the objects round if the one that should come before the other in fact comes after it.

An obvious thing to try is a random partition. So you start with objects arranged in an arbitrary order. Your target is to rearrange them so that they are arranged according to some linear ordering that you don’t know, and all you are allowed to do is pairwise comparisons. To visualize it, I like to imagine that the objects are rocks that all look fairly similar, that you want to lay them out in a line going from lightest to heaviest, and that all you have to help you is a set of balances that can fit at most one rock on each side.

With this picture, the random method would be to partition the rocks randomly into pairs and do the corresponding comparisons. When you have compared rocks and , you put them back in the same two places in the line, but with the lighter one to the left of the heavier one (so they may have been switched round).

What happens if we do random comparisons of this kind? The answer is that it doesn’t work very well at all. To see why not, suppose that the ordering is approximately correct, in the sense that for every the th lightest rock is in roughly the th place. (To be more concrete, we could say that it is in the th place and that or something like that.) Then when we do a random comparison, the probability that we move any given rock is extremely small (in the concrete case around at most). Basically, we are wasting our time comparing rocks when we are pretty certain of the answer in advance.

This suggests an adjustment to the naive random strategy, which is to have a succession of random rounds, but to make them gradually more and more “localized”. (Something like this is quite similar to what Ajtai, Komlós and Szemerédi do, so this idea doesn’t come out of nowhere. But it is also very natural.) That is, at each stage, we would choose a distance scale and pair up the rocks randomly in a way that favours distances that are of order of magnitude , and we would make shrink exponentially quickly as we proceed.

The thing that makes it a bit complicated (and again this has strong echoes in the AKS proof) is that if you just do a constant number of rounds at one distance scale, then a few points will get “left behind” — by sheer bad luck they just don’t get compared with anything useful. So the challenge is to prove that as the distance scale shrinks, the points that are far from where they should be get moved with such high probability that they “catch up” with the other points that they should be close to.

It seems to me at least possible that a purely random strategy with shrinking distance scales could sort objects in rounds, and that if one devised the random partitions in a particularly nice way then the proof might even be rather nice — some kind of random walk with drift might be taking place for each rock.

So my questions are these.

(i) Has something like this been done already?

(ii) If not, is anyone interested in exploring the possibility?

A final remark is that something that contributed a great deal to the health of the discussion about the Erdős discrepancy problem was the possibility of doing computer experiments. That would apply to some extent here too: one could devise an algorithm along the above lines and just observe experimentally how it does, whether points catch up after getting left behind, etc.

I was shocked and saddened to hear about a week ago that Ted Odell, a mathematician to whom I owe a lot, died suddenly on January 9th of a heart attack while he was travelling to this year’s joint AMS/MAA meeting in San Diego. He was 65, but seemed a lot younger.

Ted was a world leader in Banach space theory, and in particular in the infinite-dimensional theory. The wry and slightly enigmatic smile you see in the photo was extremely characteristic: if I imagine Ted, I automatically imagine him with exactly that smile. Less clear from the photo, though perhaps it can be guessed from the camera angle, is that he was extremely tall: he belonged to a handful of mathematicians I know who make me feel short (Tom Sanders and Alex Scott being two others).

I first met Ted when I went to my first ever conference, in Strobl am Wolfgangsee in Austria in 1989. I can’t remember how it came about, but I ended up chatting to him, and he started explaining to me in a wonderfully clear way — the kind of explanation you just can’t get from a textbook — how Tsirelson’s space worked. I read in an obituary by András Zsak (which starts on page 30 of this issue of the LMS newsletter) that Ted had a reputation for being kind and encouraging to young mathematicians. He certainly was to me at this conference, taking the time to give this explanation to a graduate student about whom he knew nothing. Most of the next section describes an argument that he sketched out for me on one of those yellow pads of paper that seem to be standard in US maths departments. (I think I’ve still got the yellow sheets that he let me keep, but I’ve no idea where they are.)

Tsirelson’s space

At the time, I was very interested in the so-called distortion problem for Banach spaces. I shall have a lot more to say about that later, but let me briefly formulate the notion of distortion here. Two norms and on the same space are said to be -equivalent if there are constants and with such that for every . After rescaling, one can assume that they satisfy the inequalities , so for convenience I’ll do that. We call the norm a -distortion of if the above inequality holds, and if it cannot be improved by passing to an infinite-dimensional subspace. That is, for every , and for every subspace of and every you can find such that and .

It is far from obvious that there exists a constant and a norm that can be -distorted. If you want to get a sense of the difficulty of this problem, then you might like to spend half an hour thinking about it: if you haven’t already seen the answer, then you will get absolutely nowhere.

Of course, I’ve sort of given away that Tsirelson’s space provides an example, but even with that huge clue the problem isn’t trivial, since you have to come up with the -distortion. It was this that Ted explained to me.

I’m not going to give a full explanation with all its details, but I’ll try at least to convey the basic idea, starting with the definition of Tsirelson’s space (or rather, the dual of the space that Tsirelson originally defined, which, as was observed by Figiel and Johnson, is easier to work with). To define the norm, which is on a space of sequences, we need the following notation. If is a sequence and , then is the sequence that takes the value when and otherwise. In other words, we write for the coordinate projection associated with the set . If , we write to mean that : that is, ends before starts. Finally, we say that a sequence of subsets of is admissible if . (I write when formally I should write .) That is, a sequence of sets is admissible if each set in the sequence starts after the previous one has finished, and if the minimum of the first set is bigger than the number of sets in the sequence.

We now define the norm of a sequence using the following amazing formula. Let be the set of admissible sequences of sets.

In words, the idea is this. With each you want to associate a non-negative real number, and you want to make that number as large as possible. You have two options. Either you can settle for the norm of , or you can chop up into pieces (discarding everything up to the th coordinate), play the same game with each piece, add up the results, and divide by .

Note that if is finitely supported, then there is nothing to be gained by the chopping up if the support of is entirely contained in one of the pieces, since then you’ll simply have divided by unnecessarily. So the game terminates, and if you play it optimally, then you get the norm of .

How does one say anything about a norm that is defined by an implicit formula such as the one above? This is what Ted taught me. A first observation is that it is possible to find sequences of unit vectors that approximate the unit vector basis of as closely as you like. (The notation means that the support of ends before the support of starts.) The proof is simple. If the support of begins after , then the supports of themselves form an admissible sequence. It follows from the implicit definition that for any sequence of real numbers.

A nice inductive argument of R. C. James allows one to improve this -equivalence to a -equivalence. Let be any norm on such that for every . Let (for convenience I’ll assume that is a perfect square) and divide the standard unit vector basis into blocks of size . Then one of two things is guaranteed to happen.

(i) For every , the th block generates a vector such that .

(ii) For some , every vector generated by the unit vectors in the th block satisfies .

In the second case, we have obviously improved the equivalence to on a subspace of dimension . But in the first case we have as well, because is at least by hypothesis and at most by the triangle inequality.

By iterating this argument a few times, we can make the equivalence as good as we like.

Now comes Ted’s nice idea. If we want a -distortable space, then we can take Tsirelson’s space as defined above (actually, the normal thing to do was to take , but Ted wanted to take a general in order to show that -distortions existed for any ) and define an equivalent norm as follows:

Here, I’m being a bit sloppy about constants: and are basically the same, but for the chopping up in the above definition I want an integer. The idea is that you take a vector in Tsirelson’s space and you’re allowed to increase its norm by cutting it up into pieces and adding the norms of those pieces.

It is easy to show that this is an -equivalent norm on Tsirelson’s space. It is at least as big as the original norm. But also, since for every , it follows by the triangle inequality that . To show that we have a distortion, we need to show that sometimes chopping up a vector and adding the norms of its bits doesn’t help us, and sometimes it does. And we need to show that that is the case inside every subspace of Tsirelson’s space.

Here’s where I’ll get a bit sketchier. In any subspace we can find unit vectors that generate a subspace that’s -close to . If we take such vectors and define to be their average , then we obtain a vector that can’t be improved much by chopping, at least if is much bigger than . The reason is that, apart from a few edge effects, we’ve basically chopped up into sums of consecutive blocks of the , and since the generate something very close to a copy of , is very close to the sum of the norms of those sums. So the chopping didn’t achieve anything.

How about a vector where chopping does help? The technique here goes a long way towards explaining the point of the strange definition of Tsirelson’s space. Let’s call a vector of the kind I defined in the previous paragraph an -average. We now inductively construct (inside some arbitrary subspace) a sequence of -averages as follows. Once we’ve constructed , we pick to be much larger than the maximum of the support of and we let be an -average that starts after finishes.

It turns out that if we want to calculate the norm of the vector in Tsirelson’s space, we can’t do much better than the obvious decomposition into , which tells us that the norm is roughly .

Why can’t we do better? Well, suppose we chop into pieces that start after . Then for each such that we know that the maximum of the support of is smaller than , so we’ve completely missed out in the chopping up of . And for each we know that is much greater than , so there is nothing to be gained by chopping up (or any later ).

But if is approximately , then , since is at least .

What we have done with those calculations is show that every subspace contains vectors of two different kinds. One kind is averages, and the other is sums of averages for a sequence that grows very rapidly. For the first kind of vector , and are roughly equal, whereas for the second kind of vector , . So we have shown that Tsirelson’s space, defined with constant , is -distortable (or perhaps it’s safer to say -distortable just to provide some elbow room in the calculations).

This shows that for every there exists a -distortable Banach space. But that raises a very natural question: can we reverse the quantifiers and find a Banach space that is -distortable for every ? This question was answered by Thomas Schlumprecht, who had a very close collaboration with Ted over many years, which resulted in a large number of beautiful papers. Let me briefly give the definition of Schlumprecht’s space, the first known arbitrarily distortable space.

The philosophy behind Tsirelson’s space is that since the more pieces you chop a vector into, the bigger you can make the sum of the norms of the pieces, you need some kind of “penalty” associated with the number of pieces you chop into. The penalty is that if you want to chop into pieces, then you must sacrifice the first coordinates of your vector. Schlumprecht’s idea was to impose a different kind of penalty: instead of taking a fixed constant , you let grow with the number of pieces you chop into. Many functions would do for this purpose, but the function had some properties that made it convenient in calculations. Accordingly, Schlumprecht’s space is defined by the following formula.

Because grows slowly, every subspace of Schlumprecht’s space contains averages, by the James argument. But now if you take sums of averages with rapidly increasing , you can get the gain from chopping up into pieces to grow without limit as grows.

Incidentally, Tsirelson first constructed his space as an example of an infinite-dimensional space that does not have a subspace isomorphic to or for any . The fact that it contains copies of in every subspace and that these copies are generated by disjointly supported vectors rules out everything except . And is ruled out because if it contained a copy of then that subspace wouldn’t be distortable.

The distortion problem.

I now want to describe a problem that was for several years my favourite problem in mathematics. The problem is very simple indeed to state, given the definitions in the previous section.

Problem. If , is -distortable for some ?

We have seen above that there are distortable spaces. But what about the spaces we know best, the -spaces? An infinitary analogue of the argument of R. C. James that was used above to obtain -averages can be used to prove that is not distortable, and an argument along similar lines shows that is not distortable. But what about a Hilbert space, say?

The fact that Tsirelson’s space can be distorted offers little encouragement, because it relied on the general idea that there are two different kinds of vectors (in the strong sense that they can be found inside every subspace). A Hilbert space is so homogeneous that it looks hard, and perhaps impossible, to identify two different types of vectors.

I came to believe strongly that Hilbert spaces were not distortable. My reason was that there is a simple reformulation of the distortion problem that turns it into a problem in Ramsey theory that is very similar to known Ramsey theorems that have positive answers. Here is the reformulation.

Problem. Let and let the unit sphere of be coloured with finitely many colours. Must there exist an infinite-dimensional subspace and one of the colours such that every point in the unit sphere of is within of a point with that colour?

This is slightly different from a normal Ramsey theorem in that we don’t find an infinite-dimensional subspace that’s entirely of one colour. Instead, we expand the colours by and then try to find a subspace that lives inside one of the (expanded) colours. This is necessary, since otherwise one could colour a vector red if its first non-zero coordinate is positive and blue if it is negative. But the problem above is a natural “analytic” Ramsey problem.

To see the connection, observe that if you have a -distorted norm on the Hilbert space, then you can have one colour for vectors where the distorted norm is less than times the usual norm, and one for the rest. Since every subspace contains vectors that have norms roughly equal to the usual norm and other vectors that have norms roughly equal to times the usual norm, this gives us a counterexample to the Ramsey-type problem. In the other direction, if you have a two-colouring and no subspace is contained in the -expansion of either colour, then you can construct a distorted norm. I leave this as an exercise.

Why did I believe so strongly that this Ramsey problem had a positive solution? It is because of a beautiful theorem in Ramsey theory due to Neil Hindman. In its original formulation, it states the following.

Theorem. Let be coloured with finitely many colours. Then there exists an infinite sequence such that has the same colour for every non-empty finite subset of .

There is also a “finite-unions” version, which says the following.

Theorem. Let the finite subsets of be coloured with finitely many colours. Then there exist subsets such that has the same colour for every non-empty finite subset of .

Here the notation means that the maximum element of is less than the minimum element of .

To see that the sums version implies the unions version, think of each subset of as the binary expansion of a positive integer. That is, the subset corresponds to the positive integer . Given a colouring of the finite subsets of , pull it over to the integers using this correspondence, and apply the first version of Hindman’s theorem. This doesn’t quite finish things off, because it gives us some sets that may well not satisfy the condition . However, given any infinite set of integers and any , we can always find a sum of distinct elements of that is divisible by . This follows easily from the pigeonhole principle: you just find integers that are the same mod and add them up. Therefore, using the set that Hindman’s theorem has given us, we can find finite sums that correspond to sets with arbitrarily large minimum, from which it follows that we can find a sequence of (disjoint) finite sums that correspond to sets .

The finite-unions version of the theorem can in turn be rephrased in terms of 01-sequences. If and are two 01-sequences, write to mean that the last non-zero entry of comes before the first non-zero entry of . Then the finite-unions version of the theorem is trivially equivalent to the following statement.

Theorem. Let the finitely supported 01-sequences be coloured with finitely many colours. Then there exist sequences such that has the same colour for every non-empty finite subset of .

This is what got me excited. It seems very much like a combinatorial analogue of a positive solution to the distortion problem: a sequence space is replaced by a set of 01-sequences, but that shouldn’t be a huge deal.

I did in fact manage to pursue these thoughts to prove a theorem in the space that most resembles the space of finitely supported 01-sequences, namely (the space of real sequences that tend to zero, with the norm). It states the following.

Theorem. Let and let the unit sphere of be coloured with finitely many colours. Then there exists an infinite-dimensional subspace that lies entirely in the -expansion of one of the colours.

For what it’s worth, the theorem above is the one result of mine that appears in an Elsevier journal.

There are two things that make particularly combinatorial. One is that it can be discretized very naturally and easily: you just take all points with coordinates that are multiples of for some . The other is that if you add two disjointly supported vectors, then you basically just concatenate them. I won’t say much about the proof here — just that it used ultrafilters (as does one of the nicest proofs of Hindman’s theorem) and it involved a subtlety, in that the obvious discrete problem that you might be tempted to write down has a negative answer because of the sign-of-first-non-zero-coordinate example. So even for the discrete problem one has to prove an approximate result — the doesn’t just come from the approximation of the continuous problem by the discrete one.

Try as I might, I just couldn’t find a way of doing without the combinatorial crutch that I had in . And yet the homogeneity of and the fact that it was a very beautiful and natural sequence space made it impossible for me to give up the dream that there was some vast analytic generalization of Hindman’s theorem waiting to be proved.

What did finally cause me to give up that dream was the announcement in I would guess 1993 (since the paper was published in 1994) that Odell and Schlumprecht had constructed a counterexample to the distortion problem. Very surprisingly (to me at least), they deduced that every with is arbitrarily distortable from the fact that Schlumprecht’s space is arbitrarily distortable. To do this, they had to apply a very clever non-linear map to take the colourings that correspond to distortions of Schlumprecht’s space and convert them into colourings that correspond to distortions of . (I think I may be oversimplifying here — it was a long time ago and I don’t remember much more than I am writing here.)

Another way of stating the remarkable result that Odell and Schlumprecht proved is the following. I’ll state it for because there it is nicest.

Theorem. For every there exist subsets and of the unit sphere of with the following properties.

(ii) for every and .

In other words, the two sets and are almost orthogonal to each other and yet both intersect every single infinite-dimensional subspace. Even getting the inner product to be at most is formidably difficult, or at least seems so: it would be fascinating if somebody could come up with a fundamentally different construction.

For me, the whole episode had some important lessons for how mathematical research works. I became obsessed with the distortion problem, and ultimately failed to solve it. And yet, out of that obsession, and also from what I learned during that conversation with Ted Odell, came an understanding of distortability that ended up yielding me a number of other results. On the other hand, I was a little too convinced that the problem had a positive answer, and that meant that there was no chance of my solving it — though the solution of Odell and Schlumprecht was so ingenious that I don’t think I would have solved it even if I had gone all out to find a counterexample.

Conclusion.

I have tried in this post to give some idea of what Ted Odell contributed to mathematics, not by going through his numerous mathematical achievements one by one, but in a more indirect way by conveying the impact he had on me personally. Many other people will have different stories to tell, but the themes that will run through them are that he was a wonderful mathematician, generous with his ideas, and very supportive to younger mathematicians.

I’d like to end by mentioning a beautiful problem that arises very naturally from the above discussion and that is, as far as I know, still unsolved. It is all that remains of my hopes for proving an analytic Ramsey theorem for a non-combinatorial Banach space.

Problem. Does there exist a distortable Banach space that is not arbitrarily distortable? In particular, if Tsirelson’s space is defined with constant , can it be distorted by substantially more than ?

The Elsevier boycott: where do we now stand?

In the first few months after the boycott started, the number of signatories grew very rapidly. The growth is now much slower, but this was to be expected: given that, for understandable reasons, no editorial boards of Elsevier journals were ready to take the drastic step of leaving Elsevier, it was inevitable that further progress would depend on the creation of new publication models, which takes time and work, much of it not in the public eye. We are very pleasantly surprised by how much progress of this kind there has already been, with the setting up of Forum of Mathematics, a major new open-access journal, and the recent announcement of the Episciences Project, a new platform for overlay journals. We are also pleased by the rapid progress made by the wider Open Access movement over the last year.

In one respect the boycott has been an unqualified success: it has helped to raise awareness of the concerns we have about academic publishing. This, we believe, will make it easier for new publishing initiatives to succeed, and we strongly encourage further experimentation. We believe that commercial publishers could in principle play a valuable role in the future of mathematical publishing, but we would prefer to see publishers as “service providers”: that is, mathematicians would control journals, publishers would provide services that mathematicians deemed necessary, and prices would be kept competitive since mathematicians would have the option of obtaining these services elsewhere.

We welcome the moves that Elsevier made last year in the months that followed the start of the boycott: the dropping of support for the Research Works Act, the fact that back issues for many journals have now been made available, a clear statement that authors can post preprints on the arXiv that take into account comments by referees, and some small price reductions. However, the fundamental problems remain. Elsevier still has a stranglehold over many of our libraries as a result of Big Deals (a.k.a. bundling) and this continues to do real damage, such as forcing them to cancel subscriptions to more independent journals and to reduce their spending on books. There has also been no improvement in transparency: it as hard as ever to know what libraries are paying for Big Deals. We therefore plan to continue boycotting Elsevier and encourage others to do the same.

The problem of expensive subscriptions will not be solved until more libraries are prepared to cancel subscriptions and Big Deals. To be an effective negotiating tactic this requires support from the community: we must indicate that we would be willing to put up with cancelling overly expensive subscriptions. The more papers are made freely available online (e.g., through the arXiv), the easier that will be. Many already are, and we regard it as a moral duty for mathematicians to make their papers available when publishers allow it. Unfortunately, since mathematics papers are bundled together with papers in other subjects, real progress on costs will depend on coordinated action by mathematicians and scientists, many of whom have very different publication practices. However, a statement by mathematicians that they would not be unduly inconvenienced by the cancelling of expensive subscriptions would be a powerful one.

We are well aware that the problems mentioned above are not confined to Elsevier. We believe that the boycott has been more successful as a result of focusing attention on Elsevier, but the problem is a wider one, and many of us privately try to avoid the other big commercial publishers. We realize that this is not easy for all researchers. When there are more alternatives available, it will become easier: we encourage people to support new ventures if they are in a position do so without undue risk to their careers.

We acknowledge that there are differing opinions about what an ideal publishing system would be like. In particular, the issue of article processing charges is a divisive one: some mathematicians are strongly opposed to them, while others think that there is no realistic alternative. We do not take a collective position on this, but we would point out that the debate is by no means confined to mathematicians: it has been going on in the Open Access community for many years. We note also that the advantages and disadvantages of article processing charges depend very much on the policies that journals have towards fee waivers: we strongly believe that editorial decisions should be independent of an author’s access to appropriate funds, and that fee-waiver policies should be designed to ensure this.

To summarize, we believe that the boycott has been a success and should be continued. Further success will take time and effort, but there are simple steps that we can all take: making our papers freely available, and supporting new and better publication models when they are set up.

Doug Arnold, John Baez, Folkmar Bornemann, Danny Calegari, Henry Cohn, Ingrid Daubechies, Jordan Ellenberg, Marie Farge, David Gabai, Timothy Gowers, Michael Harris, Frédéric Hé lein, Rolf Jeltsch, Rob Kirby, Vincent Lafforgue, Randall J. LeVeque, Peter Olver, Olof Sisask, Terence Tao, Richard Taylor, Nick Trefethen, Marie-France Vigneras, Wendelin Werner, Günter M. Ziegler

What is an arXiv overlay journal? It is just like an electronic journal, except that instead of a website with lots of carefully formatted articles, all you get is a list of links to preprints on the arXiv. The idea is that the parts of the publication process that academics do voluntarily — editing and refereeing — are just as they are for traditional journals, and we do without the parts that cost money, such as copy-editing and typesetting.

The organization setting up this platform is called the Episciences Project, and they are referring to the journals as epijournals, which I’ll do here, though epijournals will probably not use the word “epijournal” in their titles (since they will want to make clear that the stamp of quality that they confer is every bit as legitimate as the stamp of quality conferred by a traditional journal). They aim to make the software good enough that the administrative burden on editorial boards is no greater than it is for a traditional journal. If they succeed in that aim, then it should be possible for epijournals to be “Diamond” open access — free to read and free to publish. Certainly the intention is that there should be no charges of any kind, with the costs of maintaining the site met, if I understand correctly, by an organization called Centre pour la Communication Scientifique Directe (CCSD) in collaboration with the Institut Fourier at Grenoble University.

One possibility being discussed, which I am very much in favour of, is each accepted article having not just a link to the arXiv but also a web page for (non-anonymous) comments and reviews. For example, the editor who accepts an article might wish to write a paragraph or two about why the article is interesting, a reader who spots a minor error might write explaining the error and how it can be fixed (if it can), and an expert in the area might write a review that could be very useful to hiring committees.

This may even go further, with comment pages being set up for other preprints and journal articles — not just the ones that have appeared in epijournals.

Apparently, the plan is for the whole thing to start this April. Because I have known about the project for some time, I have quietly sounded out a few people in additive combinatorics, and it seems that there is enough enthusiasm that we will be able to start an epijournal broadly in that area (with a title that is not yet decided, but that will definitely not be “The Epijournal of Additive Combinatorics”). I am also on a committee (actually, they call it an Epicommittee) that is discussing some of the details of what the platform should be like — any comments you might have will be read with interest. [Added later: now that he has said so on Google+, I feel I can add that Terence Tao is also on the Epicommittee, so he has joined the good guys too.]

One question that some people might have is why, when there are a number of initiatives out there, this one should be regarded as particularly promising and worth supporting. I don’t know enough to give a detailed answer to that, but my impression is that this initiative has significant institutional back-up, including funding, that makes it more likely to succeed. Also, it is being designed for mathematicians and with the needs of mathematicians very much in mind, though it may later expand into other subjects.

April is very soon, but I hope people reading this, especially people who are critical of FoM and would rather move straight to a more radically different publication model, will give serious thought to setting up epijournals or encouraging others to do so. Another possibility envisaged by the people running the project is that some existing journals might like to convert to epijournals, which would certainly be interesting if it happened. And finally, if and when people do start to set up epijournals, please support them: if an epijournal gets plenty of good papers, then it will be much easier for it to establish the kind of reputation that will impress hiring committees (though I hope that if post-publication comments and reviews take off, they will be seen to provide more useful information than what can be deduced from which journal a paper gets into).

The Episciences project will soon be releasing a statement about the project. When it has done so, I’ll provide the link here.

I’ve been slightly vague about who the people behind this project are, which is because I am not 100% sure. However, the initial approach came from Jean-Pierre Demailly, Ariane Rolland and Benoît Kloeckner and subsequent emails have come from Jean-Pierre Demailly, so I think it’s them — my uncertainty is over whether there are other people I should be mentioning too. If I discover that there are, then I’ll add their names.

Added later: Benoît Kloeckner makes the following comment below.

I can clarify a bit the “epi-team” composition. Jean-Pierre Demailly tried to launch a similar project some years ago, but it had much less institutional support and did not work out. More recently, Ariane Rolland heard about this tentative and, having contact at CCSD, made them meet with Jean-Pierre. That’s the real beginning of the episciences project, which I joined a bit later. The names you should add are the people involved in the CCSD: Christine Berthaud, head of CCSD, Laurent Capelli who is coding the software right now, and Agnès Magron who is working on the communication with Ariane.

If you are not already familiar with this debate, the aspect of Forum of Mathematics that many people dislike is that it will be funded by means of article processing charges (which I shall abbreviate to APCs) rather than subscriptions. For the next three years, these charges will be waived, but after that there will be a charge of £500 per article. Let me now consider a number of objections that people have to APCs.

It is just plain wrong to ask authors to pay to get their articles published.

There are many variants of this argument. For instance, an analogy is often drawn with vanity publishing: do we want vanity publishing for mathematical articles?

Let me begin with the “it is just plain wrong” part. A number of people have said that they find APCs morally repugnant. However, that on its own is not an argument. It reminds me of some objections to stem cell research. Many people feel that that is wrong, regardless of any benefits that it might bring. Usually their objections are on religious grounds, though I imagine that even some non-religious people just feel instinctively that stem-cell research is wrong. In the case of APCs, I very much doubt that there is a religious objection, so I think everybody can agree that merely saying, “I find the idea horrible,” is not an argument until one has given a reason for its being horrible. It is scarcely necessary to say this, but some people have advanced the “it is obviously immoral” argument, so I am briefly mentioning it.

To be fair to the people who have said that it is immoral, they have gone on to give further arguments, so all I’m saying is that instead of starting from the position that it is immoral (as some people have), we should start by discussing the benefits and harms and conclude that it is immoral only if we find that the harmful consequences are unacceptable.

Actually, I myself very strongly agree with the assertion that it is wrong for authors to pay to have their articles published. Why? The main one is that it gives an advantage to rich authors. When we judge the research output of other mathematicians, we pay some attention to the quality of the journals that they have published in. If it turned out that rich mathematicians could publish in better journals than poor mathematicians, we would be introducing a completely irrelevant criterion, wealth, when all that should matter is the quality of the mathematics. The fact that it would be giving an advantage to people who are already advantaged makes things even worse.

So what am I doing on the editorial board of a journal that will in due course have article processing charges? There is no inconsistency here, because authors will not pay to publish in Forum of Mathematics.

There is a misconception here, which I have unfortunately helped to perpetuate. In my previous post about Forum of Mathematics I made a bad mistake, which was to suggest that APC stood for “author publication charge” rather than “article processing charge”. Other people often refer to this method of financing a journal as the author-pays model. But it isn’t. An article processing charge is what it says — a charge for the processing of an article. So to call it an author-pays model is incorrect unless the APC is met by the author.

Forum of Mathematics will not under any circumstances expect authors to meet APCs out of their own pockets, and I would refuse to be an editor if it did. (I imagine the same holds for all the other editors.) Of course, it is one thing to say that authors are not expected to pay, and another to make sure that that never happens. Let me describe the safeguards that will be put in place.

First of all, when you submit an article, there will be no mention of APCs. The article will be processed in the normal way — sent out to referees, discussed by editors, etc. — and a decision will be made whether to accept or reject it. Since APCs have not been mentioned, the decision will be completely independent of financial considerations. (Of course, it will often be easy to guess whether an author can pay. But the editors will have absolutely no incentive to take this information into account. And if there were ever any pressure from CUP to be a bit more lenient to authors who were likely to be able to pay, I imagine that the entire editorial board would immediately resign.)

If your article is accepted, and if your institution is set up to meet APCs (as an increasing number of institutions already are) or they are covered by a grant that you are on, then you will arrange for your APC to be paid. Otherwise, CUP will ask for a letter from your institution stating that they are unwilling to pay the charge. No justification for this is required — just confirmation that it is the case. If you are not affiliated with an institution, then the charge will be automatically waived.

In short, you yourself won’t pay anything, and you won’t be expected to go to huge trouble applying for money to cover the APC. Either there will be a system in place for covering the charge, or you will need to organize for a simple letter to be sent. The worst that can be said for the effect this will have on you is that it will involve a bit more bureaucracy. But I don’t see that it will be any more time-consuming than correcting galley proofs, say. And even the bureaucracy should gradually become less necessary, since after a while CUP should be able to deal directly with the institutions that meet APCs and will know which ones don’t.

So if you regard APCs as immoral because you are imagining authors having to pay out of their own pockets, or authors from rich institutions having an advantage over authors from poor institutions, or authors having to go round with a begging bowl when they get an article accepted, or authors managing to get worthless articles published by paying money to unscrupulous publishers, then what you are objecting to does not apply to Forum of Mathematics. Maybe it will apply to other journals, and maybe that will be a problem: that is a question I’ll come back to.

What??!! How can it cost £500 to process an article?

There are two questions here. One is whether £500 will be a fair reflection of the costs that CUP will incur when processing Forum articles. The other is whether what they provide for those costs is worth paying for. The first question has a simple answer: it will. The answer to the second question is much less obvious, for which reason I want to postpone discussing it until the part of this post that will deal with the more serious objections to Forum of Mathematics.

So how can the costs reach anything like £500? I’ll talk in general terms here, and not specifically about Forum of Mathematics. There are many things that an academic journal does to a paper once it has gone through the refereeing process and been accepted. It does copy-editing, typesetting, addition of metadata, and making sure the article appears on various bibliographic databases. (I repeat that in this section I am not discussing whether we want all these things.) A typical cost for all this is around $20 per page. That’s just a fact: if you go round and ask people who work for conventional maths journals what it costs them per page to process an article, that is the kind of figure you will get.

At this point, you can do some calculations yourself. If an average article is 25 pages, that’s already $500, which is approximately the same order of magnitude as £500. Then you have to take into account a number of other factors, such as that it costs money to handle papers that are then rejected (not all that much, but even arXiv needs $7 per paper), and there will probably be several of those per accepted paper, that fees will be waived for some articles, that there will be staff costs and overheads (such as part of the cost of heating the building used by the staff — things like that), and so on.

For that kind of reason, it is a straightforward empirical fact that £500 is the right order of magnitude for the costs per article incurred by a journal that operates in roughly the same way as a current conventional print journal.

Forum of Mathematics is even worse than Elsevier.

Let’s think about what you are committing yourself to if you agree with this. First, the cost to the academic community of an article published in Forum of Mathematics is £500. What is the cost of an article published by Elsevier. This is harder to judge, for various reasons, but it seems to be at least an order of magnitude higher. Let me quote Mike Taylor writing in the Guardian a few months ago.

For Elsevier, the biggest of the barrier-based publishers, we can calculate the total cost per article as £1,605m subscription revenue divided by 240,000 articles per year = £6,689 per article. By contrast, the cost of publishing an article with a flagship open access journal such as PLoS ONE is $1,350 (£850), about one eighth as much. No one expects open access to eliminate costs. But we can expect it to dramatically reduce them, as well as making research universally and freely available.

I actually think that the “real” cost of the arrangement (which I won’t attempt to define here) is higher still, because Elsevier’s bundling arrangements mean that libraries are paying for a lot of articles that they don’t really want. Or perhaps what I should say is that while the average cost may be £6,689 per article, we should think of it as quite a lot more than that for the articles we want, and quite a lot less than that for the articles that we don’t want.

But even if we accept the figure of £6,689 as it stands, that’s a lot more than £500. So to show that Forum of Mathematics is worse than Elsevier, you need to establish that it is worth paying £6,189 per article to avoid the harm associated with submitting an article to Forum of Mathematics.

Let’s remind ourselves what that harm was: it was a little piece of extra bureaucracy that has to be gone through when you submit a paper. It isn’t any of those things that people like to imagine such as hard-up graduate students being shut out from the journal, faculty being unable to publish because their universities won’t cover the fee, authors paying substantial sums out of their own pockets, people using money to buy prestige, etc. Those would all be very bad things, worth fighting against. But you aren’t fighting against them by fighting against FoM.

If you say that FoM is worse than Elsevier, then you are saying that an hour of your time (to give a generous estimate for how long you would need to write and follow up on an email requesting either payment or a letter refusing payment) is worth £6,189 to the academic community at large.

Authors are doing a service to the world, so making them pay is ridiculous.

First, let me repeat that authors will not pay to publish in FoM. But let’s think about what the service is that authors do to the world. In some cases, they prove results that fascinate other mathematicians and stimulate a great deal of further research. That is undoubtedly doing a service. But that service is already done the moment they put their paper on the arXiv or their home page (assuming they do). So why do they bother to publish?

As I think everybody agrees, now that we have the internet, the main function left for journals is providing a stamp of quality. There is a big question about whether we actually need journals for that, but that question is independent of the question of who benefits from the service provided by journals. It is not the reader, since readers can quite happily look at preprints. The main person who benefits from the stamp of quality is the author, who boosts his or her CV and has a better chance when applying for jobs and so on. There is also some benefit to hiring committees, who can look at a publication list and get a quick sense of whether an author is publishing in good journals.

If you feel that APCs are wrong because if anything you as an author should be paid for the wonderful research you have done, I would counter that (i) it is not journals who should be paying you — they are helping you to promote yourself, and (ii) if your research is good, then you will be rewarded for it, by having a better career than you would have had without it.

Let me now turn to some arguments that I think have more merit to them.

Maybe a typical article costs around £500 to process under the current system, but do we need what we get for that money?

This is a much more serious question. While I’m discussing it, let me also highlight another misconception, which is that the editors of FoM regard it as a blueprint for the future of all of mathematical publishing. Maybe some of them do, but I don’t. There are two more modest ways in which it could be part of the picture: it might exist in its current form only as a temporary measure until newer and cheaper methods of assessment are developed and become accepted, or it might be that it and a few other journals would persist with traditional methods of processing articles but the bulk of mathematical publishing would be done much more cheaply, with minimal typesetting, copy-editing etc.

If traditional methods of processing articles do cost something like £500, whereas merely having an editorial and refereeing process should cost much much less (but not quite nothing, since there will be administrative costs), what is the argument for spending that much on the copy-editing and typesetting of articles that people find perfectly readable in their preprint form?

To my mind, the main argument is that moving from the current system to a radically new system is difficult unless there is a smooth path from one to the other. Imagine, for example, that somebody sets up an editorial board that does nothing except ask referees to report on papers on the arXiv, “accept” the papers it regards as good enough, and list those papers, with links, on a website. It seems to many people, including me, that such a board is doing pretty well all that we need of a mathematics journal. But suppose that a board of that kind were to be established, with the stated aim of competing directly with Journal of Functional Analysis, and that you were a postdoc trying to improve your publication list with a view to getting a good job somewhere. Wouldn’t you feel that it was safer to submit your paper to Journal of Functional Analysis than to the new “journal” that people reading your CV might not have heard of or might not trust?

I very much hope that ventures such as that will be set up, will be successful, will be trusted, and will look good on people’s CVs. But I think that that will take time. Meanwhile, FoM provides an option that is enough like a conventional journal that an article published there will look every bit as good as an article published in the journals it is competing with, and that is also open access and much cheaper to the academic community than a subscription journal.

In my ideal world, would every maths journal be run like FoM? Not at all. But to get to the ideal world, I think that it is going to be easier to persist with journals that are pretty conventional (but much cheaper) at least for now.

There is another argument in favour of what publishers currently do, which is that they help your paper appear on citation indexes, they give you journals with impact factors, and so on. I hate all that stuff: the measures are incredibly crude and far less useful than a well-written reference. I think most mathematicians share my distaste. But a lot of other scientists don’t seem to, and there is a danger that if mathematicians are perceived as “not really publishing” any more, then they will not be understood or taken seriously in situations where they are competing with people from other subjects.

I wish that argument would go away, and I hope that one day it will, but that’s an even bigger battle than the battle for reasonably priced journals.

I don’t want traditional-style journals with APCs. I want much more radical change.

I basically agree with this, but as I argued in the previous section, I think that there is a case for having APCs at least as a transitional arrangement. There is another, and to my mind stronger, argument for this, which is that APC-based journals are much more vulnerable if a better model comes along. The faults of the current subscription model have been obvious for years, but it has been very hard to do anything about it because of bundling, which means that you can’t easily cancel subscriptions. (For a great description of the problem, try this blog post of John Baez.) Suppose now that we lived in a world where all maths journals were open access and funded by article processing charges. And suppose that a lot of mathematicians decided that they were perfectly happy to publish in different ways — free electronic journals, arxiv overlay boards, or whatever. Then they could simply publish in those different ways. If you publish in a different way at the moment, your poor old library is still locked into all those expensive subscriptions, but if you publish in a different way in a world full of APC-based open access journals, then whoever would have had to pay the APC no longer has to.

I had a horrible fantasy the other day, when it occurred to me that publishers could try to reintroduce the bundling concept in connection with APCs. Suppose that Elsevier made an offer to a university that for a flat fee all academics at that university could publish free in Elsevier journals for the next five years. If the flat fee was set in such a way that the university expected to save money, then it would be a tempting offer. But what would happen then? The university would say to its academics, “If you have the choice between an Elsevier journal and a comparable journal published by someone else, please go for the Elsevier journal.” And once Elsevier (and other big publishers with similar arrangements) had driven the smaller journals out of business, it could start upping the fees, and it would be very difficult for new journals to compete. In other words, the major problem with subscription journals could be reborn in a new guise.

However, forewarned is forearmed. Now that we know that bundling arrangements, however tempting in the short term, are ruinous in the long term, we can tell our universities to have nothing to do with them. Any sign that a publisher is trying to introduce them can be met with widespread negative publicity. And I think that if this nightmare did eventually come to pass, mathematicians could have moved on to better publication systems before they were affected by it.

Maybe FoM’s waiver policy is OK, but by associating yourself with FoM, you are indirectly conferring legitimacy on many journals with much worse policies.

This is a danger, I’ll admit. However, I think that the right way to counter this danger is not to campaign against the principle of article processing charges itself, but to campaign for certain safeguards to apply to any journal that has such charges. Here is a possibly incomplete list of safeguards.

1. Editorial decisions should be completely independent of financial considerations. If the editors decide that a paper is good enough to be accepted, then it will be published. Ideally, editors should not know, when they handle a paper, whether the author has access to funds for article processing charges. [I say "ideally" simply because there will be situations where an author's institution's policy is known to an editor. For example, it seems that in the UK, as a result of government mandates, all universities will be obliged to have a pot of money for paying APCs, and an editor may well know that an author is British.]

2. Under no circumstances should there ever be any advantage to an author who is happy to pay an article processing charge out of his/her own pocket.

3. An author at an institution that is willing to pay article processing charges should not be at any advantage over an author at an institution that is not willing to pay article processing charges.

4. The article processing charges should be set at the level needed to cover reasonable costs of the publisher (including overheads and possibly a modest profit for the purposes of reinvestment).

As I have argued above, Forum of Mathematics has these four safeguards in place. The fourth one is perhaps less obviously essential than the others, for two reasons. One is that some institutions, such as learned societies, might want to make larger profits in order to support their activities (and perhaps replace lost subscription revenue). Another is that one might hope that market forces would operate more efficiently. If a subscription journal is outrageously expensive, bundling makes it hard to do anything about it, but if a journal charges outrageous APCs, it is easy (in many cases) to avoid publishing in that journal.

What I would like to see is (cautious) support for journals with safeguards like 1-4 in place, and strong criticism of journals that manifestly don’t — which in my case would probably include adding them to the list of journals that I am boycotting.

As always, there is much more that I could say, but I think I’ll end it there. Before I finish, I would like to mention that this post will be followed soon by a companion post entitled “Why I’ve also joined the good guys.” If the idea of APCs still sticks in your craw, then you will find that post more to your taste.

The purpose of the conversation was to help him catch up with some work that he had missed through illness. The particular topics he wanted me to cover were integrating , or as he called it, and integration by parts. (Actually, after I had explained integration by parts to him, he told me that that hadn’t been what he had meant, but I don’t think any harm was done.) But as we were starting, he asked me why the derivative of was , and what was special about .

That seemed like a good preliminary conversation to have, so I said, “OK, let’s try to differentiate from first principles and see what happens.” He didn’t know what I meant by “from first principles” so I tried to give him a nudge, by saying, “If you didn’t know the derivative of , then how would you go about working it out?”

At this point, he suggested . To be fair to him, he wasn’t saying that he knew that this was correct. Nevertheless, this was an interesting piece of cognitive dissonance, given that we were trying to understand why the derivative of was . But what bothered me more than the fact that he said it was the fact that he couldn’t see why it wasn’t right. And what bothered me perhaps even more than that was the fact that he should think of “from first principles” as mechanically applying the rule.

In an effort to get past that, I asked, “Yes, but what does the derivative actually mean?” He had no answer. So I drew a graph of an arbitraryish function, labelled it , drew a point on the curve, and asked him what the derivative meant. I think he did then say that it was the gradient of the curve at that point. (I don’t think he used the word “tangent”.) I asked how we could go about working that out. He suggested . I said, “So to work out the derivative you just divide by — is that it?” He laughed and said no.

It was time to go back to basics, so I asked him how you work out the gradient of a straight line. He said, “Rise over run”, terminology that I either never knew or had completely forgotten — but the meaning was obvious. I then asked what the difficulty was when the line wasn’t straight, to which he replied that the gradient was changing all the time. So what could we do about that? He suggested taking a point not too far away and working out the slope of the line joining that to the point in question.

Now we were getting somewhere. I had already drawn a line segment going up from a point on the x-axis marked until it hit the curve. I did the same for a line segment going up from a point marked and asked what the rise and run were. He correctly got the answers and . I then said that as got smaller and smaller, the curve got more and more straight, so

was a formula for the derivative .

“Have you really never seen that before?” I asked. He denied it, but when I pressed him, he eventually conceded that he probably had seen it, but that if so, then it would have been clearly flagged up as something that he didn’t need to know, because there wouldn’t be questions about it on the exam.

It’s that that bothered me enough to make me need to get it off my chest in the form of a blog post. It hardly needs spelling out what was wrong with the argument that his teacher gave (if his teacher did in fact give that argument — I cannot be certain about this, though the mere fact that that was the message that got across is bad enough) but I’m going to anyway. Let’s suppose that your aim is simply to do well at maths A-level and that there are no questions that test your familiarity with the formula for the derivative of an arbitrary (nice) function at an arbitrary point. Which is better?

1. Don’t make any effort to learn and understand the formula, but simply learn a few basic examples of derivatives (polynomials, exponentials, logs, trig functions) and rules for differentiating combinations (linearity, product rule, quotient rule, chain rule) and you should be able to differentiate anything that comes up in the exams.

2. Learn what the derivative means, derive the formula for the derivative of an arbitrary function at an arbitrary point, calculate a few derivatives from first principles, derive the product rule, quotient rule and chain rule, and then learn how to use them to differentiate combinations.

The answer is that if you are capable of doing 2, then 2 is far better. And the boy I was talking to was certainly capable of doing 2. Why is it better? Because (and this is something I plan to devote a blog post to at some point) memory works far better when you learn networks of facts rather than facts in isolation. If you don’t really understand what differentiation is all about, then the fact that the derivative of is is a completely different fact from the fact that the derivative of is . But if you’ve derived them both from first principles (I’ll come back to what I said about in a moment), then they are related: we have a process we do to the functions and and this is what comes out. Of course, another reason is that if you forget something, you have a chance of rederiving it, but that’s a slightly different point. Your knowledge of a piece of maths is far more grounded if you know how it is derived, or at least have some memory of the derivation, even if you have no problem remembering the fact in question. Even if you forget the details of the derivations, just having seen them has a major effect on binding together the facts you know.

After I had explained differentiation in the abstract, I suggested that we should try differentiating from first principles. Or as I put it, “Let’s apply this formula in the case . What do we get then?” To my further dismay, he didn’t immediately know what to do. “If , then what is ?” I asked. I can’t remember what his response was, but it wasn’t . He floundered and made wild guesses, not really understanding what I was asking. Again, something quite serious seemed not to have been done by his school, though I couldn’t give a precise diagnosis in this case — something along the lines of understanding the notion of a function well enough to talk about an abstract function and see that it could have many instantiations.

Anyhow, once that point was cleared up (not necessarily for good, but for then anyway) we got through the differentiation of without further trouble. Again he denied having seen that derivation — maybe it was at that point that he said the thing about not needing to know it for the exam.

The general point here is of course that A-levels have got easier and schools have a natural tendency to teach to the test. If just one of those were true, it would be far less of a problem. I would have nothing against an easy A-level if people who were clever enough were given a much deeper understanding than the exam strictly required (though as I’ve argued above, for many people teaching to the test is misguided even on its own terms, since they will do a lot better on the exam if they have not been confined to what’s on the test), and I would not be too against teaching to the test if the test was hard enough.

How about differentiating ? Well, after a couple of false starts we got to the expression

I asked what could be done with . Quite a bit of prompting was needed to get him to say . Then I asked what could be done with . A lot of prompting was needed to get the answer to that question. (I had to ask what he would do with .) Anyhow, we eventually got to

I decided just to point out that the last limit was the derivative of at 0. I also pointed out that the entire argument so far worked just as well for the function , whatever the (positive) value of . I then drew some pictures for different , pointing out that some of them crossed the y-axis with a slope less than 1 and others with a slope greater than 1 and that was the one where it actually equalled 1. He asked me why the slope was exactly 1 for , which was a good opportunity for me to try to explain that that was getting things the wrong way round, and that was chosen precisely to make that work. (Of course, the question would make very good sense the way he asked it if we had already defined in some other way, but I’m not sure he had. Certainly, I remember this as the definition of when I was at school, and I remember feeling slightly uncomfortable about it.)

There was plenty more of our conversation, though not much more to say about it. I accidentally fell into a derivation of the product rule, which again I think he had not seen. That was part of my preparation for deriving the formula for integration by parts. When I had done that, I went through an example or two. One of the examples I tried was from to . (He thought was , by the way, but was OK when we did it in degrees.) We got the answer . I then felt annoyed not to be able to see why the answer had to be 1. I still haven’t got round to thinking about that.

I also discussed integrating by the method I call guess-and-adjust. You guess because one part of what the product rule gives you is correct and you might be able to deal with the other part. Differentiating you get . “What can we do to get rid of that 1?” He suggested . We tried that, saw why it didn’t work, and then got to the right answer.

He managed to integrate between and with no help at all, so I think he got the basic idea, though whether he’ll hold on to it I don’t know.

Another thing I discovered was that he was very shaky on the chain rule. When I asked him what the chain rule said, he didn’t know what I was talking about. Eventually I got a glimmer of recognition out of him by writing down . But the idea that if you want to differentiate you first pretend that is a single variable with respect to which you are differentiating and then correct what you’ve just done by multiplying by the derivative of was completely foreign to him. We looked at a few examples but they’ll need reinforcing at some point. It was yet another illustration of the general principle that if you forget about understanding what’s going on and concentrate on mechanical manipulations, you’ll forget how to do even the mechanical manipulations.

One of the first things I found out when I arrived at the hospital was that I was having the ablation done under a general anaesthetic. I was quite surprised by this but the surgeon came and explained that he was expecting it to take quite a long time, and that he thought he would be able to do a better job if I was under a general anaesthetic. Since that was my top priority, I was happy to agree, though a little disappointed that I wouldn’t have an interesting and unusual experience that I was expecting to have. (That reasoning was conditional on my needing to have the ablation in the first place — obviously it’s not the sort of experience I would seek out otherwise.)

So I spent the morning being prepared in various ways. I had sticky pads attached to me so that I could be monitored, I had a canula inserted into my left wrist, and I was required to put on hilariously unsexy garments — long white socks with a hole at the end for the toes to stick out (which squeeze the leg hard and reduce the chance of an embolism), “paper pants”, which after the operation were half cut off, and a hospital gown that opened at the back. Another thing I had to do was discuss the risks and sign a consent form. The risks sounded worse than I thought — quite a long list, and quite a number of items being given as 1 to 2 percent. The most worrying was permanent damage to the phrenic nerve, a nerve that takes signals from the brain to the diaphragm. The registrar I spoke to wasn’t very specific, but I think that would have left me short of breath for the rest of my life. That was one of the 1 to 2 percent risks. I’ve just found a paper on the web that says that the risk is between 0.11% and 0.48% (whatever that means). Anyhow, I signed the form. I was later asked twice to confirm that it was my signature.

Eventually the time came and I was wheeled to the operating theatre. I was given a drug that made me feel pleasantly woozy, and after a couple of minutes of that I was out.

Next thing I knew, I was back in my room. My wife was there, and I thought to myself, “Great — survived that.” I can’t remember whether my surgeon was there too or whether he arrived soon afterwards. Anyhow, he told me it had gone well.

The next night was fairly awful, because they were worried that my blood pressure was low, so they came to measure it once an hour. I was fairly uncomfortable, so I had a succession of hours where I would be woken up, would get to sleep with some difficulty, and would be woken up again. So the next day I felt pretty awful, but assumed it was probably the night I’d had. The best part of the night was watching the moment the US election was called for Barack Obama — I was awake, and someone else was listening to it so loudly that I could hear what was going on, so there wasn’t much to lose by watching it myself.

The low point of the whole experience was when the registrar who had explained to me the risks (and also put in a canula with very shaky hands) said to me a rather matter-of-fact way that they had put a stent in one of my arteries. He explained to me that this meant I would be on a certain drug for the next year. I was quite surprised, to say the least. I asked why, and he explained that the artery in question was quite narrow. I said something like, “You mean it was dangerously narrow and they had to do it as an emergency procedure?” and he suddenly said, “Oh wait, hang on, I’ve got the wrong notes.” He came back with the right notes and helpfully explained to me that I had had an ablation. But for a couple of minutes I genuinely believed that I had had a stent fitted.

I was discharged in the early evening the next day, but then had another difficult night (mentioned above), a day in bed doing nothing, a better night, and now part of a morning sitting writing this. But I’m beginning to feel like going back to bed for a while.

The background is that over the last fifteen years or so I have had occasional bouts of atrial fibrillation, a condition that causes the heart to beat irregularly and not as strongly as it should. It is quite a common condition: I’ve just read that 2.3% of people over the age of 40 have it, and 5.9% of people over 65. Some people have no symptoms. I myself have mild symptoms — I can feel a slightly strange, and instantly recognisable, feeling in my chest, and I experience a few seconds of dizziness almost every time I stand up from a relaxed seated position — otherwise known as orthostatic hypotension, which I often used to get anyway (as do many people).

I would gladly live with those symptoms, but unfortunately that’s not all there is to it. When a heart is in atrial fibrillation, it is not beating as efficiently as it should, and a little pool of blood can form that doesn’t get pumped away. And if that happens, it can form a clot. And if your heart then goes back into sinus rhythm (that is, it starts to beat normally again), that clot can get pumped out into your bloodstream and wreak havoc: in particular, it can lead to a stroke. I know this all too well, because my father had a severe stroke for exactly that reason in 2001.

Until 2004, my bouts of atrial fibrillation, of which I think there were two, were several years apart and lasted a few hours each. But then in early 2004 I went into atrial fibrillation and didn’t come out anything like so quickly. To decrease the risk of stroke, I had injections of a blood-thinning drug called Heparin (or at least, that’s what it’s called in the UK) into my stomach, once a day. But then I was put on to Warfarin, a drug that is used in rat poison. When a rat eats Warfarin, it goes off and has a haemhorrage and dies, but if you take just the right amount, you can thin your blood to the point where it is not dangerously thin, but atrial fibrillation is less likely to cause clots. The appropriate dose varies widely from person to person, so they have to increase it very gradually, testing your blood several times, until you reach the right INR (international normalized ratio), which roughly speaking means the ratio of the time your blood takes to clot to the time it would take to clot if you didn’t thin it. The recommended INR for people with atrial fibrillation is between 2 and 3.

When my atrial fibrillation ended in 2004, I was at an early stage of the process of getting the dose right. I asked my doctor whether that meant that I had in fact been more or less unprotected at the critical moment, and the answer was yes. So a completely standard procedure — coming off the Heparin before the correct Warfarin dose was established — had a very obvious defect. Fortunately, I didn’t have a stroke. (The probability was quite small, but even so.) That’s nothing compared with my father’s experience: he had his stroke after being advised by his cardiologist to come off Warfarin, a recommendation that other doctors told him later made no sense at all.

Let me try to fast forward to the present day. I have had quite long periods without AF, sometimes as long as two or three years, but I have also had two periods where I went into AF and it became pretty clear that I wasn’t going to come out of it again spontaneously. This, by the way, is very normal — once it starts, it gets gradually worse. Twice I had an electrical cardioversion: you go under a general anaesthetic and are given an electric shock that stops your heart, and when it starts again, if you are lucky it starts in sinus rhythm. When you have an electrical cardioversion, they try three times before giving up. I was lucky both times and went into sinus rhythm after just one shock.

The first cardioversion lasted me three years, apart from one 24-hour bout that ended of its own accord. The second cardioversion was in June, but in early September I went back into atrial fibrillation.

So what is the decision I have recently made? Well, back in 2004, when I first went to visit the person who is now my regular cardiologist, he told me about an operation called a catheter ablation. This is a procedure where the surgeon puts a wire into your leg and up an artery all the way to your heart. The tip of the wire then burns a bit of the surface of your heart, which causes it to form scar tissue that doesn’t conduct some faulty electrical signals that are responsible for the atrial fibrillation. At the time, my cardiologist said that it offered the prospect of a permanent cure for atrial fibrillation, but that it was probably best to wait, since the operation was relatively new and improving all the time.

We occasionally discussed the operation in the intervening years, and then in June he arranged an appointment with somebody who specializes in catheter ablations and performs them at Papworth Hospital, which is near Cambridge and is famous for being where the UK’s first successful heart transplant was carried out. This second specialist told me the following things (some of which I had read on the internet already).

1. I was probably progressing from “paroxysmal AF” (occasional bouts) to “permanent AF” (what it sounds like).

2. Catheter ablation is more effective against paroxysmal AF.

3. Now that more data is available, it has become clear that catheter ablation is not after all a permanent cure, but it might delay the progression of AF by five to ten years or something like that.

4. It is more effective the younger you are when you have it, with the decline in effectiveness quite high at my age, so in his opinion I should have it done sooner rather than later.

5. It has only a 60-70% chance of working at all.

6. It carries risks.

Most of that was pretty negative news, so I left the consultation not entirely sure what to do, though thinking I probably ought to have the operation. But the risks seemed pretty serious — about a 1% risk of a major complication — so I wanted to think a bit harder about them.

The two that bothered me most (and still do) are stroke and death. There are other serious things that can go wrong, but if their effects are temporary, then for me that puts them in a different league from a stroke, which could end my productive life, and death, which would end my life altogether.

The risk of death is put at one in a thousand, and this is where things get interesting. How worried should I be about a 0.1% risk? How do I even think about that question? Perhaps if my life expectancy from now on is around 30 years, I should think of this as an expected loss of 30/1000 years, or about 10 days. That doesn’t sound too bad — about as bad as having a particularly nasty attack of flu. But is it right to think about it in terms of expectations? I feel that the distribution is important: I would rather have a guaranteed loss of ten days than a 1/1000 chance of losing 30 years.

In the end, what convinced me that I shouldn’t worry too much about this risk was looking up what the risk of death is anyway over, say, the next year. I found on this site that the average risk of death in the UK for a man between 45 and 54 is 1/279, much higher than 1/1000. So if I am worried about a 1/1000 mortality rate from an operation, I should be about as worried that I will die from some other cause over the next four months or so. And yet I don’t lose any sleep over that possibility.

But maybe the problem is that I am concentrating four months’ risk into a few hours. Doesn’t that change everything?

Yes it would if I was planning to have lots of catheter ablations, but this is much more of a one-off event (though quite a few people have to have it done two or three times before it works). That makes a significant difference. For example, if aeroplane flights carried a 1/1000 mortality risk, that would be completely unacceptable, since some people take enough flights that all those risks would combine to create a near certainty of dying. So what matters in addition to the risk of the operation is the fact that I will have it at most a very small number of times. Maybe a rough rule of thumb is that I shouldn’t be too concerned, since on average my frequency of having this operation will be significantly less than once every four months.

Another possible counterargument is that for various reasons I am probably less likely than average for my age to die over the next year — by most people’s standards I am well off, I am in generally good health, I don’t smoke, and so on. However, I think that many of those factors also reduce my chances of major complications from a catheter ablation, so I’m inclined to guess that the validity of the rough calculation above is not hugely affected.

What about the risk of stroke? That brings me to something I haven’t yet mentioned. Even if a 1/1000 risk of death isn’t something to get too worked up about, one doesn’t want to take that risk unless there is some benefit from doing so. And because the benefit can be measured in terms of reducing risks, I am in the useful position of being able to compare like with like. In other words, it’s not like being asked whether I want to play Russian roulette for a million pounds, where I would have to weigh up a lot of money against a one in six chance of dying. It’s more like being asked to play Russian roulette (but with much better odds) once in order to avoid having to play it once a year for the next five to ten years, since the additional risk per year of having a stroke if you are in atrial fibrillation is comparable to the risk of having a stroke as a result of catheter ablation, even if one is taking Warfarin. (AF increases your annual stroke risk by a factor of about 5, but Warfarin divides that by about 3, or so I’ve read.) I can’t now find the figures I used. Again, the calculations were complicated by the fact that relative to many AF patients my risks of stroke are quite small, but again I think that applies to the risks as a result of the operation as well. As a precaution, it is standard practice to have weekly blood tests to make sure that one’s INR stays within the right range for a good long time before the operation, which mine has, so I have done what I can to minimize the stroke risk.

One final complication is that Warfarin carries its own risk: because it thins your blood, it increases the chances that you will have a brain haemorrhage, which can have very serious consequences. I think the extra risk may be something like 1% a year. Unfortunately, it isn’t considered safe to stop taking Warfarin after a successful catheter ablation, so this risk isn’t going to go away. But from the point of view of balancing the risks and benefits of the operation, that means that this particular risk doesn’t have to be taken into account, as it will be there either way.

In summary then, I’ve looked online at various statistics, none of which tell me exactly what I want to know — since they refer to populations that are more general than me, and in particular usually somewhat older than me (which is good news, since my risks should therefore be lower than average) — and concluded that probably the risk of having the operation is comparable to the risk associated with not having it. The fact that the last time I went into AF, which I’m in now, was only three months after I had had an electrical cardioversion was what finally persuaded me of that. And if I do have it and it works, then my quality of life will be improved, though not hugely, by my not being in AF. And it seems that when the doctors say that the risks of the operation are low, they are (in this instance) talking sense, since the risks are comparable to the background risk that everyone faces.

The operation is tomorrow. It takes a few hours and is done under a mixture of a local anaesthetic (in your leg where the wires go in) and light sedation. So I’ll be conscious. Assuming all goes well, it should be quite an interesting experience — I’ll report back on that when it’s over.

Two more small things. One is that AF itself is mathematically interesting: it seems that something causes the heart to go from a nice periodic rhythm into a more chaotic one, and it is not well understood why. The other is that I have tried to look things up in the medical literature in order to be able to assess the risks as well as I can. Last night I decided I wanted to look at a paper in the Lancet, that renowned medical journal published by Elsevier. Cambridge subscribes to Science Direct, Elsevier’s huge electronic package of all its science journals, so I in theory I should have been able to read the paper. I won’t go into details, but suffice it to say that even though I was entitled to read it, the system, for some mysterious reason, wouldn’t let me and kept giving annoying error messages. It reminded me why I’m in favour of getting rid of the subscription model for academic journals, and also of why I don’t like being told by Elsevier how much they have invested in Science Direct.

I’ll end by saying that this post is very much not intended to be a plea for sympathy. I know many people who have had much worse decisions (of the same general kind, but with far less favourable probabilities) to face than this one. In fact, it’s supposed to be the opposite of a plea for sympathy: more like an explanation of why a risk of 1/1000, which initially seems quite scary, is in fact not that scary after all. If, very much contrary to expectations, something goes badly wrong tomorrow, that will be the moment for sympathy. But the chances of that are very much smaller even than the chances that Mitt Romney will win the presidential election, something that as an avid Nate Silver reader I find highly encouraging.

A very obvious candidate for a discrepancy theorem that we could try to modularize is Roth’s theorem, which asserts that for any -valued function on there exists an arithmetic progression such that . That gives rise to the following problem.

Problem. Let be a prime. What is the smallest such that for every function that never takes the value 0, every can be expressed as for some arithmetic progression ?

In this post I shall collect together a few simple observations about this question.

1. We can at least prove that such an exists. Indeed, by Szemerédi’s theorem, if is large enough then there is an arithmetic progression of length on which is constant. Since takes non-zero values and is prime, the sums along initial segments of run through all the numbers mod .

So the question is really asking whether the result can be proved with a reasonable bound for .

2. Roth’s discrepancy result tells us that if and takes only the values , then takes all values mod . That is because there is a progression such that in , and by what one might call the discrete intermediate value theorem, it therefore takes all values in between 0 and or between and . This is reasonably compelling evidence that the conjecture is true with a decent bound, but of course the intermediate-value-theorem argument breaks down completely when can take arbitrary non-zero values mod .

The known lower bounds for Roth’s discrepancy theorem for APs (which are equal to the upper bounds up to a constant) show that the best possible bound for the modular question is at least .

3. To make the problem more symmetrical, and therefore potentially easier to handle, it might be a good idea to begin by tackling the following variant.

Problem. Let be a prime. What is the smallest such that for every function that never takes the value 0, every can be expressed as for some arithmetic progression in ?

The big difference here is that arithmetic progressions are allowed to “wrap around”. That means that for every arithmetic progression and every coprime to (it might be convenient to insist that is prime), the sets and are also progressions. I think the bounds in the integer case with functions show that the best bound one could hope for for this modified problem are , but I need to check that.

3. One natural approach to proving that a function takes all values mod would be to attempt to show that it takes all values with approximately the same frequency. This would potentially allow us to bring in analytic tools. However, it is not in general true. For example, let be the function that is up to and from to . Then a small calculation shows that if is a mod- AP of common difference , then is never more than . I think can probably be taken to be 2, or maybe even 1. (By I mean the smallest modulus of any number congruent to mod .) Thus, for a positive proportion of common differences , the sums along APs of common difference never exceed , say, in modulus. If is large, this implies that the values of the sums are very far from uniformly distributed, since they are concentrated around 0. On the other hand, is obviously not a counterexample to the conjecture (by which I mean the assertion that the modular statement holds with a good bound).

4. That observation does not rule out a proof that goes via showing that the values of sums along APs are approximately uniformly distributed, but it does demonstrate that in order to prove that, we would have to put some conditions on . That is, we would need a two-step argument along the following lines (reminiscent of proofs of Szemerédi’s theorem, though I would hope that this problem is easier).

(i) If is quasirandom, or at least “contains substantial quasirandomness”, then the values of are approximately uniformly distributed mod .

(ii) If is not quasirandom, or better still “does not contain substantial quasirandomness”, then we can deduce in some other way (such as finding a long AP on which is constant, but that may be too much to ask) that the sums take all values.

5. The weaker the property we can get away with in (i), the better. However, in order to get started it would be good to find any quasirandomness property that implies that the values of are approximately uniformly distributed.

6. One thing that needs deciding here is what we mean by a random progression . The simplest would be to fix some and pick random mod- APs of length . These don’t work for the modular conjecture in general, since the constant function is a counterexample, but they might work for functions with a suitable quasirandomness property.

7. The fact that we can’t fix the length of the progressions shows that the modular conjecture differs in an important way from Roth’s discrepancy theorem, where fixing the length is not a problem (especially in the mod- version). This dampens any hopes one might start off with for adapting the proof of Roth’s discrepancy theorem to cope with the modular version. But that might in the end be a good thing, since the point of the modular conjecture is to introduce new techniques that can then be brought to bear on EDP.

8. When we are looking for a suitable quasirandomness property, it is tempting to turn to Fourier analysis. However, the following (modification of a standard) example is a bit troubling. Let be defined as follows. For each we randomly decide whether to set equal to or mod (where is the residue between 0 and ). If we now choose a random mod- arithmetic progression of length 4, there is an absolute constant such that with probability at least does not wrap around, and , and . When this happens . Therefore, the value 0 occurs much too often, but according to the natural Fourier definitions of quasirandomness (which we could get by looking at the function ) this function is highly quasirandom.

9. Of course, we are looking at much longer arithmetic progressions. However, it is difficult to think of proofs that work for long arithmetic progressions and fail for short ones. (It is mildly encouraging that at least the above source of examples seems to break down when the progressions become long — though that should be checked carefully.)

10. Hmm … just after writing that I thought of the following modified example. Instead of choosing randomly whether to set equal to or , let’s do it in the following highly deterministic way: if mod 4 we set equal to , if mod 4 we set it equal to , if mod 4 we set it equal to , and if mod 4 we set it equal to . In addition, let’s suppose that is a multiple of , since I seem to need that to make this example work. (Even if we can’t get rid of this restriction somehow, it will still show that a proof that worked for prime values of would have to make strong use of the fact that was not a multiple of — and there would be many other examples of a similar kind that would lead to similar requirements — which would be quite a big restriction on what it could look like.)

Then if is a multiple of 4 and is a random mod- AP of length , there is a probability 1/16 that and mod 4. (Note that because this makes sense independently of which residue we pick.) If that is the case, we can partition into APs of length 4 such that the sum of along each is zero. This shows that with probability at least 1/16 the sum is 0 mod .

Is this function quasirandom? According to many definitions, yes, since although we split into cases in a highly structured way, the behaviour of each case of is highly quasirandom.

I mentioned above that there are many similar examples. To give just one illustration, we could take as a multiple of and let take turns taking values , , , and . Essentially the same argument would work, but now the probability would be . (Actually, I now see that the probabilities can be doubled in both cases, since works just as well as .)

I’ll leave it there. I’ve mentioned some proof strategies, and also discussed some difficulties with them. Does anyone have other proof strategies to suggest? I haven’t mentioned using the polynomial method, but that is an obvious thing to try to do: that is, write down a polynomial that vanishes identically only if the modular conjecture for APs is true, and try to prove that it vanishes identically. It is certainly worth looking into that, though it is a little discouraging that the conjecture is false for APs of any fixed length, since that would necessarily make the polynomials less symmetrical. If the above strategy seems the most promising (or should that be least unpromising?) then does anyone have any thoughts about quasirandomness properties that might do the job? (Gil mentioned something in a recent comment — it would be good to have that thought in more detail.)

I’ll end by saying that even if proving the modular conjecture for mod- APs ended up having nothing much to do with EDP, it is a very nice problem on its own, and could make an excellent spin-off Polymath project.

In this post I want to mention three strengthenings of EDP. One of them I find interesting but not promising as a way of proving EDP, for reasons that I will explain. The other two look to me also very interesting and much more promising. All of them have been mentioned already, but the point of this post is to collect them in one convenient place.

Restricting the allowable common differences.

I know of no -sequence that has bounded discrepancy on all HAPs with common difference either a prime or a power of 2. The sequence has bounded discrepancy on all HAPs of prime common difference, but if you allow powers of 2 as well, then no periodic construction can work, since if the period is for some odd integer , then a HAP of common difference will have a bias of at least 1 in each interval of length , for parity reasons, and this bias will gradually accumulate.

One can defeat powers of 2 by using the Morse sequence (if you haven’t seen it, then it’s a nice exercise to see how it is generated), but that has unbounded discrepancy on HAPs of odd prime length. (I can’t remember exactly why.)

Why do I not think that this potential strengthening of EDP is likely to be useful for attacking the problem? Well, one promising feature of EDP is that it seems to generalize to sequences taking other kinds of values, such as complex numbers of modulus 1 or even unit vectors in an arbitrary Hilbert space. However, something that I’ve only just noticed (though others may have spotted it ages ago) is that if one restricts to, say, prime-power common differences, then even the complex version of the problem becomes false. A very simple counterexample is the sequence , where (that is, a primitive sixth root of 1). Then the sum along any HAP with common difference that isn’t a multiple of 6 will be a sum of a GP with common ratio , and will therefore be (uniformly) bounded. In particular, this is true of HAPs with prime-power common difference.

This simple example also shows that a certain real generalization of EDP is false when you restrict the common differences in this way: you cannot prove unbounded discrepancy for sequences that take values in the set . The counterexample is simply twice the real part of the example above: the sequence repeated over and over again. So if (and I think it’s a big if) EDP is true for HAPs with common differences that are either primes or powers of 2, then any proof must make pretty strong use of the fact that the sequence is a -valued sequence. This rules out a lot of promising techniques, so it appears to make the problem harder rather than easier.

A discrepancy question about matrices.

In my earlier post I mentioned what I called the non-symmetric vector-valued EDP. I have subsequently realized (though perhaps a better word is “remembered” since I must have been sort of aware of this at some point) that it is equivalent to another discrepancy statement that I now find more appealing. The statement is the following.

Conjecture. Let be a function from to and suppose that for every . Then for every real number there exist HAPs and such that .

If we take a function and define , then for every the above conjecture, if true, gives us HAPs and such that , which proves that the discrepancy of is at least . So the above conjecture implies EDP. But the class of functions of the form with is a very small subset of the class of all functions that take the value 1 along the diagonal, so this conjecture is very much stronger than EDP.

It is also equivalent to the statement that for every there exists a diagonal matrix of trace at least 1 that can be written as a linear combination , where for each and are characteristic functions of HAPs and and . (This is the statement that I discussed at length in my previous post.) In one direction this is easy: if a decomposition of that kind exists, then equals the trace of and is therefore greater than 1, but it also equals , which is at most . It follows that there is some such that .

In the other direction, if no such decomposition of a diagonal matrix exists, then the Hahn-Banach theorem gives us a linear functional that separates the class of diagonal matrices with trace at least 1 from the class of linear combinations of HAP products defined above. It is easy to check that this functional must be a diagonal matrix with a constant diagonal such that the value along the diagonal is at least 1 and such that for any two HAPs and .

The conjecture is so much stronger than EDP that I think it would be a mistake just to assume that it is true. My guess is that it is true, but I would be very interested if there was a counterexample (even if, like the complex sequence above, it is disappointingly simple — in fact, if there is a counterexample, then it seems quite likely that it will be fairly simple). And if there isn’t a counterexample, then the fact that it is so much stronger a conjecture than EDP does this time make me think that the result might be easier to prove than EDP itself.

Until very recently, I had been mainly interested in the dual version of the question (that is, the question about decomposing diagonal matrices), but now it seems to me that the matrix discrepancy question is worth thinking about directly. It is a clean question, and it has the big advantage over the original EDP question that it does not restrict values to the set , so a number of methods can be used that cannot be used directly for EDP. For instance, linear programming can be used to get experimental results: Sasha Nikolov may be going to look into this.

Given any matrix , one can find vectors and such that , so this matrix question is actually trivially equivalent to the non-symmetric vector-valued question I had formulated earlier. But expressing it in terms of vectors makes it harder to think about rather than easier, and that is what had previously put me off thinking about the question directly.

There is one class of matrices that is particularly worth mentioning I think. If EDP is true but this matrix question is false, then the best candidates for counterexamples to the matrix problem are probably matrices of high rank, and an obvious class of matrices that tend to have high rank is matrices that are constant on diagonals (that is, Toeplitz matrices).

Suppose, then, that our matrix is defined to be for some function such that . Then , where by I mean the number of ways of writing as with and . So we have a class of functions that I’ll call HAP convolutions, and we’d like to show that if is any function with and is any constant, then there exists a HAP convolution such that . This is true if and only if there is an efficient way of writing the function that is 1 at 0 and 0 everywhere else as a linear combination of HAP convolutions. This is another question that could be investigated using linear programming, and perhaps since it concerns functions of just one variable we could get more extensive results than we could for the general matrix problem.

The modular conjecture.

In his most recent guest post, Gil Kalai reformulates EDP as a question about sums mod . EDP is trivially equivalent to the following assertion.

Conjecture. For every there exists such that for every sequence of length and every there exists a HAP such that mod .

At first that doesn’t look like a very interesting reformulation since it is too obviously equivalent to EDP. But what makes it interesting is that it has a very natural generalization that doesn’t have any obvious counterexamples.

Stronger Conjecture. For every there exists such that for every sequence of length of non-zero numbers mod and every there exists a HAP such that mod .

In other words, we replace the condition that the sequence takes values by the much weaker condition that it is never zero. Gil calls this the modular conjecture. (He also presented it in a comment on a much earlier EDP post.)

As Gil points out in his post, one can write down a polynomial that is identically zero (mod ) if and only if the modular conjecture is true for . It is tempting to try to prove that it is zero by analysing its coefficients. More generally, this approach to the problem appears to open the door to a number of algebraic methods.

If you want to prove EDP this way, you have to solve the conjecture for some non-zero . (Since is prime, if you can show it for one then you’ve shown it for all.) However, the problem with is interesting in its own right, and in particular it seems to be interestingly different from the problem with non-zero . At the time of writing, I don’t see any way of modifying the EDP examples to obtain exponentially long sequences mod that avoid zero sums — it seems to me that the upper bound on the length could be significantly smaller. That would be interesting, as it would place constraints on what a proof could look like for non-zero .

A fourth strengthening.

This isn’t really meant as part of the body of the post, but more of an afterthought. A strengthening of EDP that has been considered since very early on in the project is where you replace a sequence by a sequence of unit vectors in a Hilbert space. To be precise, one looks at the following statement.

Conjecture. Let be a sequence of unit vectors in a Hilbert space and let be a real number. Then there exists a HAP such that .

There is something slightly curious about this conjecture, which is that it is very hard to see how having infinitely many dimensions to play with could help one find a counterexample. In some sense, if you use too many dimensions, then it ought to be the case that there will be a HAP such that the vectors with are pointing in lots of different directions and therefore not cancelling. That makes me wonder whether one can prove that if there is a counterexample to the above conjecture, then there must be a counterexample for some finite-dimensional Hilbert space. Or if that is too much to ask, perhaps there might have to be a counterexample where all the live in some compact set. I think it would be very interesting to think about whether the vague intuition I have just expressed can be made precise. As it stands, it is of course not even close to a proper argument. (I should stress one aspect of what I am saying. If there is any vector sequence of bounded discrepancy, then one can arbitrarily modify for each prime and the sequence will still have bounded discrepancy. So I’m not suggesting that every bounded-discrepancy sequence lives, or almost lives, in finite dimensions, but that it can be used to construct one that does.)

As I write that, another question occurs to me. For this one I don’t even have a vague intuitive argument. Let’s suppose that we can find a counterexample to the matrix discrepancy question earlier. Suppose also that takes the form for some (not necessarily unit) vectors and in a Hilbert space. Must there be an example where the and lie in a finite-dimensional Hilbert space, or at least in a compact subset of a Hilbert space?

A combined generalization.

A second afterthought is that the matrix question has a modular version that might be of some interest. Let be a prime and let be a function from to with for every . Must the sums of on products take all possible values mod ? What if we merely ask for to take non-zero values on the diagonal?

If the strong modular conjecture is false, then we can turn a counterexample into a diagonal matrix in the obvious way and we get a counterexample to the second question. Indeed, suppose that for every and when . Then , which avoids some value, since is a HAP. But with matrices there are many more ways of trying to avoid particular values, so the strong modular matrix conjecture looks like a much stronger statement. Again there are two cases — avoiding 0 and avoiding a non-zero value — of which only the second obviously implies EDP.

The polynomial method is another basic combinatorial technique that occasionally works. One way to describe the method is as a way to translate a combinatorial statement into the vanishing of a certain polynomial modulo .

A demonstration of the method

Theorem: Every graph (or hypergraph) G with n vertices and 2n+1 edges contains a nontrivial subgraph H with all vertex-degrees divisible by 3.

(This is a theorem of Noga Alon, Shmuel Friedland, and me from 1984.)

Before the proof: If we want to get a subgraph with all vertex degrees even then we need n edges (or n+1 edges for hypergraphs). This has a simple linear algebra proof which also gives an efficient algorithm.

From-scratch proof sketch: Associate with every edge e of the graph a variable . Consider the two polynomials

P= , and

Q=

If the theorem is false then P-Q=0, as polynomials over the field with three elements. This is impossible since P is a polynomial of degree 4n while Q is a polynomial which has a monomial of degree 4n+2 with nonzero coefficient.

The theorem follows more directly from a theorem of Chevalley-Warning and even more directly from a theorem of Olson, but the above proof serves best our purpose.

Remarks about the polynomial method:

1) The polynomial method has many applications but only in specific cases. It is not nearly as widely applicable as, say, the probabilistic method.

2) Good basic references: A. Blokhuis, Polynomials in finite geometries and combinatorics. In Keith Walker, editor, Surveys in Combinatorics, 1993, pages 35-52. Cambridge University Press, 1993.

Noga Alon, Combinatorial Nullstellensatz, Combinatorics, Probability and Computing 8 (1999), 7-29.

3) The polynomial method is related to the “linear algebra method” in combinatorics. Often, however, direct linear algebraic proofs lead to efficient algorithms while this is not known for applications of the polynomial method. For example, no polynomial algorithm to find the graph H in the above theorem is known, and there is a related complexity class introduced by Christos Papadimitrou . The polynomial method is closely related to arguments coming from the theory of error-correcting codes, and to arguments in TCS related to interactive proofs and PCP.

The modular EDP.

The following is an equivalent way to formulate the Erdős 1932 conjecture that the discrepancy for EDP is unbounded.

1) Consider the sequence as a sequence with modulo , where is a prime that we can choose as large as we want.

2) Then every number modulo can be expressed as a sum of the sequence along a HAP modulo .

Translating EDP (in this form) into a statement about polynomials modulo is cumbersome. But one thing we may have going for us is that it suggests a natural extension of EDP where the supposed-to-vanish polynomial is simpler.

Modular EDP Conjecture: Consider a sequence of non-zero numbers modulo p. Then if n is sufficiently large w.r.t. p, then every number can be expressed as a sum of the sequence along a HAP modulo p.

As in the original EDP we can consider general sequences or just multiplicative sequences.

The Polynomial identity required for the modular EDP

Here is the polynomial identity in n variables we need to prove over when grows to infinity with as slow as we wish. For every , ,

(*)

These polynomials are not familiar but they are related to generating functions which arise in permutation statistics. In particular, when we look at the product

and expand it to monomials, the coefficients have a combinatorial meaning in terms of permutations and inversions.

Given a permutation , and an integer we can ask how many inversions are there between and a smaller integer. This is a number between 1 and .

The coefficient of in the above product is the number of permutations where there are integers contributing j inversions. The proposed identity (*) may be expressed in terms of modular properties of such permutation statistics.

Challenge: Prove the modular EDP using the polynomial method.

What does the LDH tell us about the modular EDP?

It is especially easy to apply the large deviation heuristic to the modular version of EDP. Suppose we want to compute the probability that all HAP-sums miss the outcome .

Given , the probability that is not is . So we are interested in the value of with . (Restricting our attention to multiplicative sequences will divide the exponents on both sides by .) Solving this equation gives us . The LDH heuristic comes with a firm prediction and a weak prediction. In this case the LDH gives

a) (Firm prediction) There are sequences violating the modular EDP when .

b) (Weak prediction) There are no such sequences when .

The firm prediction is correct by the logn discrepancy constructions for EDP and as a matter of fact the LDH itself gives an even stronger prediction of for -sequences. By restricting our attention to sequences we see that the weak prediction is incorrect and LDH for the modular EDP is blind to the special substructure of sequences. Note that the firm conjecture is far from being known when we extend the modular EDP and replace all integers by a random subset of integers, or by square-free integers , or by SCJ-systems of integers etc.

The approach to the Erdős discrepancy problem that, rightly or wrongly, I found most promising when we were last working on it was to prove a certain statement about matrices that can be shown quite easily to imply a positive solution to the problem. In this post, I’m going to treat that matrix statement as the problem, and think about how one might go about trying to prove it. I’ll give the brief explanation of why it implies EDP, but not of what the possible advantages of the approach are (discussions of which can be found in some of the earlier material).

First I’ll need to recall a couple of definitions and some notation. A homogeneous arithmetic progression, or HAP, is a set of the form for some pair of positive integers and . Let me say that is a HAP-function if it is the characteristic function of a HAP. Finally, if and are functions defined on , let denote the function defined on that takes the value at , which can be thought of as an infinite matrix.

Here, then, is the statement that it would be great to prove. What is nice (but also quite daunting) about it is that it is a pure existence statement.

Problem. Prove that for every there exists a real matrix decomposition with the following properties.

(i) is a diagonal matrix and .

(ii) Each and is a HAP-function.

(iii) .

I have stated the problem in this form because (for a reason I will outline below) I am confident that such a decomposition exists. However, it seems to be quite hard to find.

Why would the existence of an efficient decomposition of a diagonal matrix into products of HAP-functions prove EDP? Well, let be a -sequence and consider the quantity . On the one hand it equals

,

and on the other hand it equals

.

Since , there must exist such that , from which it follows that there exists a HAP such that . This shows that must have HAP-discrepancy at least for every , and we are done.

Of course, one can deduce anything from a false premise, so one needs at least some reason to believe that the decomposition exists. Quite a good reason is that the existence of the decomposition can be shown (without too much difficulty) to be equivalent to the following generalization of EDP.

Problem. Show that if and are any two sequences of vectors in a Hilbert space such that for every , then for every constant there exist HAPs and such that .

This is a strengthening of EDP, since it clearly implies EDP when the Hilbert space is . However, it doesn’t seem like such a huge strengthening that it might be false even if EDP is true. At any rate, it seems just as hard to find a counterexample to this as it does to EDP itself. (You might ask why I believe that EDP is true. It’s partly blind faith, but the heuristics presented by Gil Kalai in the previous post offer some support, as does — very weakly — the experimental evidence.)

How does one ever solve existence problems?

Let’s begin by thinking very generally about the situation we’re in: we have to come up with a mathematical object that has certain properties . (Of course, we could combine those into one property, but it is often more natural to think of the object as having several different properties simultaneously.)

I’m going to list strategies until I can’t (or at least can’t immediately) think of any more that seem to have any chance of success. (An example of a strategy that I think has no chance of success for this problem is to search for a just-do-it proof, the problem being that the constraints appear to be quite delicate, whereas a just-do-it proof works better when one has a number of independent constraints that are reasonably easy to satisfy simultaneously.)

1. Use an object you already know, possibly with some small modifications.

2. Start with examples that work for a certain constant , and use them as building blocks for larger and more complicated examples that work with a larger constant .

3. Use some kind of duality to prove that if an example does not exist, then some other object does exist, and show that in fact the second object does not exist.

4. Use a computer to search for small examples, try to discern a pattern in those examples, use that pattern to guess how to construct larger examples, and check that the guess works.

5. Strengthen the conditions on the object until there is a unique (or almost unique) object that satisfies those conditions.

6. Weaken the conditions on the object (e.g. by forgetting about some of the properties ), try to describe all objects that satisfy the weaker conditions, and then reformulate the problem as that of finding an object of that description that satisfies the stronger conditions. [As a very simple example, if you were required to find a positive solution of some quadratic equation, you would solve the equation without the positivity condition and then pick out a positive root.]

7. Assume that you have an object with the required properties, deduce some interesting further properties, find a natural object with the further properties, and hope that it has the original properties as well. [This is a variant of 6.]

How do the general strategies fare when it comes to EDP?

I will discuss them briefly at first, before concentrating in more detail on a few of them.

1. This doesn’t appear to be the kind of problem where there is already some matrix decomposition (or related mathematical object) waiting in the wings to be modified so that it becomes the example we want. Or if there is, the only way we are likely to discover it is by pinning down more precisely the properties we are interested in so that someone somewhere says, “Hang on, that sounds like X.” But I think it is very unlikely that we will discover an example that way.

2. I am definitely interested in the idea of starting with small examples and using them to build bigger examples. However, an approach like this needs plenty of thought: non-trivial small examples seem hard to come by, and the only obvious method for building bigger ones is taking linear combinations of smaller ones. (A trivial small example is to take a linear combination of the form where each is the characteristic function of a singleton. This gives us . To get anything interesting, we need the to be HAP-functions for longer HAPs, but then it is very difficult to get the off-diagonal terms to cancel.)

3. We are trying to write as an efficient linear combination of objects of a certain form. That is a classic situation for applying the Hahn-Banach theorem: if there is no such linear combination, then there must be a separating functional. What can we say about that functional?

This approach seems promising at first, but is in fact of no use at all, since if you pursue it, the conclusion you come to is that if there is no such linear combination, then there are sequences of unit vectors and in a Hilbert space and a constant such that for every pair of HAPs and . The reason duality doesn’t help is that we used duality to reformulate EDP as a problem about decomposing diagonal matrices. If we use duality again, we get back to (a generalization of) EDP.

4. We have tried this. Moses Charikar and others have written programs to search for the most efficient decompositions (for this and for a related problem). The results are useful in that they give one some feel for what the diagonal matrix needs to be like, but it does not seem to be possible to obtain a large enough matrix to use it to go as far as guessing a formula. For that, more theoretical methods will be necessary (but knowing roughly what the answer needs to look like should make finding those methods quite a bit quicker).

5. One difficulty is that we are not looking for a unique object: it would be nice if the properties we were trying to obtain determined the matrix and its decomposition uniquely, since then we could hope to calculate them. An obvious property to add is that the object is in some sense extremal (though what that sense is is not completely obvious). Another thing we can do is insist on various symmetries — something that has already been tried. Yet another is to restrict the class of HAPs that may be used: as far as we can tell at the moment, we can get away with HAPs with common differences that are all either prime or a power of 2. (I say that simply because it seems to be hard to find a sequence that has bounded discrepancy on all such HAPs. Gil’s heuristics suggest that the discrepancy should grow like for sequences of length .)

6. What weaker properties might we go for? One idea is simply to try to find a linear combination of products of HAP-functions that leads to a considerable amount of cancellation off the diagonal, even if there are still off-diagonal terms left. This would be interesting because if one naively tries to find a decomposition that works, it seems to be very hard to get a substantial amount of cancellation (except if very small HAPs are used, in which case the resulting decomposition is not very informative). Another (somewhat related) idea is to look for interesting decompositions but with not particularly large values of . Yet another might be to allow a wider class of functions that HAP-functions.

7. I don’t have much idea of how to apply this strategy to the entire decomposition, but there is plenty one can say about the diagonal matrix that ones wishes to decompose, and that information, if it could be expressed neatly, would surely be very useful: it seems much easier to try to decompose a specific diagonal matrix than to try to decompose a diagonal matrix that you have to choose first, when the vast majority of diagonal matrices cannot be efficiently decomposed.

The proof that the diagonal decomposition implies EDP actually proves a much stronger result. It gives us a function such that such that if is any sequence of real numbers (not necessarily -valued) with , then has unbounded HAP-discrepancy. To put that loosely, the sequence cannot correlate with the square of any sequence of bounded discrepancy. That puts very strong conditions on , but so far we have not fully understood what those conditions are. It seems that we should be able to make progress on this, perhaps even adding reasonable conditions that would allow us determine the function uniquely.

Creating new examples out of old ones.

Suppose that and . Clearly if we take a linear combination of and , we get another decomposition of a diagonal matrix, and if the original decompositions were into products of HAPs then so is the new one.

Can we do anything more interesting? Another possibility is to look for some kind of product. Now Dirichlet convolutions are natural objects to look at in the context of EDP, since the expression can be thought of as follows. You start with a function defined on , then you define the dilate to be the function that takes the value at and 0 at non-multiples of , and finally you take a linear combination of those dilates. If is identically 1, then this is taking a linear combination of HAPs.

We can’t take a Dirichlet product of two matrices, but we can do something similar: given functions and of two integer variables we can take the function . If and are diagonal (meaning that they are zero if ), then the terms of the sum are zero unless and , so in this case , considered as a function of only, is the Dirichlet convolution of and , also considered as functions of one variable.

This operation also respects products in the following sense. If and , then

,

which equals , where is the Dirichlet convolution of and and is the Dirichlet convolution of and .

One further obvious fact is that the operation is bilinear in the two matrices that it is applied to.

So far everything is working swimmingly: out of two decompositions of diagonal matrices into linear combinations of products we build another one that has some number-theoretic significance. However, there is a problem: a Dirichlet convolution of two HAP-functions is not an HAP-function except in trivial cases.

How much does that matter? Without doing some detailed calculations I don’t know. Here are a couple of simple observations. Let and be two HAP-functions. Then their Dirichlet convolution is a sum of dilates of , one for each point in the HAP of which is the characteristic function. So we can at least decompose the Dirichlet convolution of two HAP-functions as a sum of HAP-functions. The sum of the coefficients is the number of points in the HAP corresponding to — though since the situation is symmetrical, we can add dilates of instead, so it is better to say that the sum of the coefficients can be taken to be the length of the smaller of the two HAPs.

That doesn’t sound very efficient, but we must also remember that when we take the Dirichlet convolution of two diagonal matrices, we obtain a diagonal matrix that sums to the product of the sums of the original two matrices. So there may be some gain there to compensate for the loss of efficiency. That is why I say that more detailed calculations are necessary, a point I will return to at some stage.

What can we say about the diagonal matrix ?

As I have already mentioned, if a diagonal decomposition of the required kind exists, then we can place strong constraints on the diagonal matrix itself. Writing for , it must have the following property: if is any real-valued function defined on the integers such that , then the HAP-discrepancy of must be at least . (This follows from the proof that the existence of the decomposition for a diagonal matrix with implies that every -sequence has HAP-discrepancy at least .)

Turning that round, if we have any real-valued sequence of discrepancy at most , then cannot be more than .

A simple example of the kind of conclusion one can draw from that is that the sum of over all s that are not multiples of 3 is at most 1. This follows from the fact that the sequence has HAP-discrepancy 1. More generally, if is large, then for all small most of the largeness must occur on multiples of .

In qualitative terms, this suggests that ought to be concentrated at numbers with many factors, and the experimental evidence backs this up, though not quite as cleanly as one might ideally like. (However, “edge effects” could well account for some of the peculiar features that are observed experimentally, so I still think it is worth trying to find a very aesthetically satisfying diagonal matrix .)

Here is another reason to expect that should be larger for numbers with many factors: they are contained in lots of HAPs. We are trying to find a measure on (at least if we insist that takes non-negative values, which seems a reasonable extra condition to try to impose) such that any function with a large -norm with respect to that measure must have a large HAP-discrepancy. What might make it difficult to find a function with large -norm and small HAP-discrepancy? It would be that we had lots of constraints on the values taken by the functions. And the natural way that we can place lots of constraints on one value is if has many factors and is therefore contained in many HAPs.

As an example of the opposite phenomenon, suppose we take an arbitrary sequence and alter its values however we like at all primes. What will happen to its HAP-discrepancy? Well, each HAP contains at most one prime, so we cannot have changed the HAP-discrepancy by more than 2. This illustrates fairly dramatically how little it matters what a sequence does at numbers with few factors.

A first wild guess.

It is tempting to jump straight from this kind of observation to a guess of a function that might work. For example, what if we took some primes , a collection of subsets of and let be the characteristic measure of the set of all numbers that can be written as for some . That is, takes the value on all such numbers and zero on all other numbers.

Does a function like that have any chance of working? Because we have ensured that it sums to 1, we would now be looking for a decomposition with absolute values of coefficients summing to at most for a small constant . But before we even think about doing that, we should try to find a sequence that is large on many of the products of primes but that has bounded discrepancy. Only if we find it hard to do that is it worth looking for a decomposition.

Let me be more precise about what we are looking for. If we can find a decomposition of (the diagonal matrix with entries given by ) as where the and are HAP-functions and , then for every sequence , we have two ways of calculating . One of them gives us , while the other gives us . So if , then there must be some HAP on which has discrepancy at least .

Therefore, in order to show that this particular will not do, we need to find a sequence such that averages at least 1 on the numbers but has bounded discrepancy.

Let us make our task easier to think about by looking for a sequence that takes values on the numbers . Can we choose the values elsewhere in such a way that we cancel out any discrepancy that we might accidentally pick up with those values?

To answer this question, let us think about how HAPs intersect the set, which, since I’m mentioning it quite a bit, I’d better give a name to: I’ll write for the set of numbers with . Let me also write as shorthand for . Any HAP that intersects must have a common difference of the form for some set . The resulting HAP will contain every for which , which is rather pleasantly combinatorial.

Let’s suppose that we’ve chosen the values everywhere on . What we are trying to do now is choose values off (not necessarily ) in such a way that any HAP-discrepancy contributed by values in is cancelled out by a roughly opposite discrepancy in the values outside .

An obvious way to do that is to be fairly greedy about it. That is, we look at HAPs that have a substantial intersection with and simply choose a few further values on those HAPs, hoping that our choices won’t mess up too many discrepancies elsewhere. Let’s see if we can get something like that to work.

Consider, then, the (infinite) HAP with common difference . As already mentioned, this intersects in every such that . So as we trundle along the multiples , we find that the partial sums of the s that we encounter and have already chosen go up and down, and we would like to choose some further values to ensure that they remain bounded.

At which values of are we still at liberty to choose the value ? It must be a multiple of , and two ways of ensuring that that multiple does not lie in are (i) to make sure that is also a multiple of some with and (ii) to make sure that is a multiple of for some .

The question now is whether if we do that we will find that the value we choose has an effect on lots of other HAPs at the same time. And it looks rather as though it will. Suppose is quite a large set. Then any multiple that does not belong to is also a multiple of for every , so there is indeed a danger that by choosing a value of we are messing up quite a lot of HAPs and not just the one we were interested in. We could of course try to sort out the HAPs with large first and later correct any damage we had done to smaller sets — but a large set has a lot of subsets.

This is another situation where the devil is in the detail. Maybe there is enough flexibility that some kind of careful greedy approach would work — perhaps we could even choose to be 1 everywhere on — but so far it seems at least possible that there exists an efficient decomposition of a diagonal matrix with entries that are concentrated on square-free numbers with many prime factors.

It is sometimes convenient to describe arithmetic progressions of the form as HAPs. Let me do that now, and let me define two HAPs to be adjacent if they are disjoint and their union is also a HAP (in this generalized sense). In other words, two HAPs are adjacent if one is simply a continuation of the other.

If and are HAP-functions coming from adjacent HAPs and (in which case I’ll say that they are adjacent HAP-functions), then is 1 on the diagonal at all points in the HAP , while off the diagonal it is 1 on and latex P\times Q\cup Q\times P$. If we take lots of products of this form, we can hope to get lots of cancellation off the diagonal, while getting none at all on the diagonal.

Let’s try to pursue this idea in more detail. Suppose we have some kind of probability distribution on functions of this form. That is, we pick, according to some distribution, a random set and then a random pair of adjacent HAPs and . Now let and be their characteristic functions and let and be two positive integers. What is the expected value of ?

A necessary condition for this quantity to be non-zero is that should divide both and . So let us condition on this event. If we are clever about the way we choose our probability distribution, we should be able to organize it so that if you choose a random , then typically plenty of its subsets have a similar probability of being chosen. The reason that is potentially important is that if we fix a common difference and pick random pairs of adjacent HAPs and of some given length, then pairs that are close to the diagonal will tend to get positive values (because normally if one of them is in then so is the other, and similarly for ) while pairs that are a little further away tend to get negative values. But if we have many different common differences in operation then we can hope that some of these biases will cancel out: a difference that tends to result in a positive value at one scale might tend to result in a negative value at another scale.

While writing this I have just remembered a useful trick that we were well aware of during the previous attempt at EDP. It is straightforward to show that EDP is equivalent to the same question for the rationals. That is, one wishes to show that for every function defined on the positive rationals and every constant there is a HAP (the definition is obvious) on which the sum of the function has absolute value at least . The nice thing about this formulation is a much greater symmetry: for every rational , the map is an isomorphism from (considered as a group under addition) to itself. This makes it unnecessary to think about numbers with lots of factors. (It is not hard to get into this nice situation using just integers — for example, you can just multiply everything by for a very large and you can divide your numbers by whatever you like, within reason — but it is more natural to use rationals.)

The hope was that if we thought about functions defined on the rationals, then the rather peculiar properties of the diagonal matrix one wishes to decompose might become less peculiar. However, one would still need to think carefully about ratios of numbers.

I’m tempted to continue thinking aloud about these possibilities, but I’ll save that up for the comments.

A second wild guess.

I’m now going to ignore what I’ve just written about working with the rationals and go back to the question of trying to find a natural function that is biased towards positive integers with many factors. What I’d like to do is build a sequence of non-negative functions , each better than the last, in the hope of eventually finding one that has (or at least doesn’t obviously not have) the property that every sequence such that has large HAP-discrepancy. (To make this non-trivial, I also need .)

I’ll start with a function I know doesn’t work: let for every and 0 for every . That fails because the sequence has HAP-discrepancy 1 but (at least if is a multiple of 3). That doesn’t satisfy the condition I said, but it does if we multiply it by .

The problem there was that has a large sum on non-multiples of 3. To correct that and similar problems, it feels natural to give greater weight to numbers with more factors. But how? Well, let’s do it in a rather naive way and simply weight each number by how many factors it has (and therefore how many HAPs it belongs to). This at least has the advantage of being a natural number-theoretic function: it is (up to a constant) the Dirichlet convolution of the constant function 1 with itself, at least until you get beyond . That is, .

Roughly how big is ? This sum is the number of pairs of positive integers such that (since is the number of pairs of positive integers such that ). Counting the number of pairs with we can crudely estimate this as , which is roughly . On the other hand, the Dirichlet convolution with itself of the function that is up to and 0 thereafter sums to (since each pair of positive integers with contributes 1 to the sum). Since the two functions are equal up to , we find that the vast majority of the second function lies beyond the point where it equals .

But let’s not worry about that. On average, grows like , which is not too fast a rate of growth, so let’s simply take the function up to . We now want to prove that every sequence such that has HAP-discrepancy at least (for some that tends to infinity with ). Alternatively, we want to find a counterexample, which will induce us to try to improve the function.

An obvious example to try is the usual one, namely the sequence . To see whether this works, we need to estimate the sum of over all non-multiples of up to . That equals the number of pairs such that neither nor is a multiple of 3 and . And that is roughly times what you get without the divisibility condition. In other words, it is roughly . So to get to equal we need to multiply the sequence by , which gives it a HAP-discrepancy of , a slight improvement on the that we obtained with a constant function.

This doesn’t prove much — maybe there are better examples — but it suggests in a weak way that taking Dirichlet convolutions results in improved functions. To be clear what I mean by “improved”, I am now looking for a non-negative function defined on such that for large , or perhaps even all , if is any sequence such that , then has HAP-discrepancy at least . The bigger I can get to be, the better I consider the function to be.

If you take repeated Dirichlet convolutions of the constant function 1, then you get functions , where is the number of -tuples of positive integers such that . (The function is equal to .) Suppose is large, is very large, and is a sequence such that . Does it follow that has large HAP-discrepancy? More ambitiously, does it have HAP-discrepancy that grows exponentially with , provided that is large enough?

Back to the rationals.

The following idea is one that feels familiar — I think something similar has come up already. But there’s nothing like a break from a problem to make an old stale idea seem new and fresh, and sometimes the apparent staleness was an illusion. So here it is again.

I’ll begin by creating a function on the diagonal that seems to me to have a sporting chance of being efficiently decomposable into HAPs. Let (for generating set) be the set and let be the -fold multiplicative convolution of the characteristic function of with itself. That is, is the number of ways of writing as with and all the and being integers between 1 and .

Before I think about how one might go about decomposing the diagonal “matrix” (that is, a function defined on that is zero at unless ) into products of HAP-functions, I want to see whether it looks hard to find a function such that but is of bounded HAP-discrepancy.

To have any chance of that, we need to understand a bit about . We can get that understanding by representing every rational as a product of (possibly negative) powers of distinct primes. For the rationals where is non-zero, the primes are all at most , so we can think of as a function defined on a lattice of dimension (the number of primes less than or equal to ). Then is the -fold convolution of the characteristic function of regarded as a subset of this lattice. If is large enough, then this convolution will look like a large and spread-out Gaussian, which will have the potentially useful property that it is approximately invariant if you multiply (or divide) by an integer between 1 and .

If has that property, does it rule out some kind of variant of the example? A key feature of that example is that the function in question is zero on multiples of 3, but in there is no such thing as a multiple of 3 (or rather, everything is a multiple of 3 so the concept ceases to make sense).

But that argument is far from conclusive. How about defining a function as follows: given a rational in its lowest terms, let be 0 if either or is divisible by 3, and otherwise 1 if mod 3 and if mod 3?

Let us think what the values of are at , , etc. If is a multiple of 3, then all those values are 0 (since is not a multiple of 3). If is a multiple of but not , then except if is a multiple of . If and are congruent mod 3, then the values of go . If they are not congruent mod 3, then they go . That last argument worked even if , so we have covered all cases.

What is the density of rationals such that neither numerator nor denominator is a multiple of 3? I haven’t made the question precise, and that allows me to give two different answers. The first answer is that if we pick the numerator and denominator randomly, then the probability that neither is a multiple of 3 is . Also, the probability that they are coprime is , and the probability that they are both coprime and not multiples of 3 can easily be shown by similar methods to be at least an absolute constant. So it seems that our function is supported on a set of rationals of positive density.

But another way of thinking about it suggests that the support of the function has zero density: every rational can be written as a product of powers of primes, and the probability that the power of 3 that we choose is 0 tends to 0 as we choose more and more rationals.

The second answer is, I think, the correct one for us, though the question is a bit misleading. The -density of rationals for which the “3-coordinate” is zero does indeed tend to zero, so if we regard as our measure, rather than something more additive, then we do appear to have ruled out examples that are similar to the example.

Can we use information about the original problem to help us with the dual problem?

We know that for the integers there is not an efficient HAP-decomposition of a diagonal matrix that is 1 for the first terms and zero afterwards. The proof we have is that if such a decomposition existed, then the sequence would have to have large HAP-discrepancy, which it doesn’t. But can we give a “direct” proof? I’m not quite sure what I mean by that, except that it should not involve the use of a separating functional. Ideally what I’d like to do is use that example and others like it to work out some rather general constraint that would end up saying a precise version of, “If an efficiently decomposable matrix equals 1 on the diagonal, then it must have quite a lot of stuff off the diagonal,” and ideally say a fair amount about that “stuff”.

The hope would be to find a clear obstacle on the dual side to a direct approach to decomposing the identity matrix, and then to show that when we move to the measure on the rationals defined in the previous section, that obstacle goes away and allows us to do what we wished we could do over the integers. The thing that would make life easier would be the ability to divide anything by small integers.

Actually, maybe it isn’t so important to go for the dual side. Let’s just understand what the existing proof is telling us goes wrong when we try to find an efficient HAP-decomposition. For the decomposition to be efficient, the HAPs we use have to be quite long. And for that to be the case, they must contain roughly as many numbers congruent to 1 mod 3 as they do numbers congruent to 2 mod 3. (They might contain none of either.) Therefore, any HAP product has roughly as many points congruent to or mod 3 as points congruent to or mod 3. Let us call these white points and black points, respectively. By roughly I mean that these two numbers differ by at most a constant. In fact, I think they differ by at most 1. But no points on the diagonal are congruent to or to , so if a product of HAPs includes points on the diagonal that are congruent to or , then it must include at least more black points than white points off the diagonal.

So, roughly speaking, we can’t build up the non-multiples of 3 on the diagonal without building up a bias towards black points off the diagonal.

Does that matter? Can’t we correct that unwanted bias by adding or subtracting some other HAP products? No we can’t, unless we also cancel out much of the sum on the diagonal.

What rescues us if we deal with rationals instead (or more precisely a portion of the rationals weighted by the function )? Also, how easy is it to find an efficient decomposition so that the weight off the diagonal is comparable to the weight on it? I suspect the answer to the second question is that it is not easy, since we have used just the fact that the sequence has bounded discrepancy, but there are many other examples.

A variant of the HAP problem.

I stopped the previous section prematurely because I was feeling a bit stuck. Instead I would like to think about a modified problem that may well be easier. EDP asks for a high discrepancy inside some HAP. The set of HAPs has the nice symmetry property that if you dilate a HAP then you get another one. This is particularly good over the rationals, when you can also contract.

Now let me ask a slightly vague question and then attempt to make it precise. Can we prove EDP for a different “base collection” of sets? For EDP I’m regarding the base collection as the collection of all sets . If you look at all dilations of those, you get all HAPs. Since I’m having trouble proving EDP for HAPs, I’d like to try to prove it for a different base collection. Of course, the smaller the collection, and the more closely related it is to the set of HAPs, the better, but for now my priority is to be able to prove something.

I actually have a collection of sets in mind. Let us think of the rationals multiplicatively as an infinite-dimensional lattice, or rather the portion of that lattice that consists of points with only finitely many non-zero coordinates. That is, the point corresponds to the rational , the being integers. I’m not sure exactly what I want to do next, so let me be slightly less precise. If we look at the rationals multiplicatively, as we are doing here, then dilation becomes simply translation. What I’d really like is something like an orthonormal basis, but I’d like it to consist largely of functions that are translates of one another. So far that won’t work, but it looks a lot more hopeful if we allow some kind of “spreading out” (which would be the analogue of taking longer and longer intervals). The most natural type of spreading out is, I would suggest, to take repeated convolutions. So a possible question — not necessarily the exact one I’ll end up asking — is this. Suppose we start with the set that we considered earlier. Is it true that every function that has large norm relative to the measure defined by (multiplicatively) convolving times has a large inner product with some (multiplicative) translate of some iterated convolution of ?

This problem should be not that hard since it is purely multiplicative, so we can take logs and make it purely additive. If the answer turned out to be yes, then for any -function we would get a large inner product with some iterated convolution of . If we could nicely decompose that iterated convolution into HAPs, we would then be done, though that seems like quite a lot to ask.

Hmm … I’ve almost instantly run into trouble. It seems that my “multiplicative version” of EDP is just plain trivially false. Consider the function that is 1 for products of an even number of primes and -1 for products of odd numbers of primes. (This is the completely multiplicative function , with which we are already very familiar.) From a multiplicative point of view, where we think of as an infinite-dimensional lattice, this is putting 1 at “white points” and -1 at “black points”. If we now take any “cuboid”, by which I mean a set of the form for every , then the sum of on that cuboid is either 0, 1 or -1. I was about to say that I couldn’t see an elegant proof of this, but here’s a fairly quick one. If we remove two adjacent slices from the cuboid, we remove an equal number of white and black squares. “Removing two adjacent slices” means reducing by 2, provided . If some is even, then by repeating this operation for that we can remove all points, which shows that the numbers of white and black points were originally the same. If all are odd, then we end up reducing the cuboid to a set with just one point, in which case we get .

If we multiplicatively convolve any two cuboids, we obtain a function that can be expressed as a linear combination of translates of a cuboid, so its inner product with will also be small, as will those of all its translates. (What’s more, those small inner products will themselves alternate in sign as you multiply or divide by primes.) So will have small inner product with everything you build out of products of GPs using multiplicative convolutions.

It could be fruitful to think about why HAPs have a better chance of giving rise to discrepancy than the kinds of multiplicative sets I’ve just considered. In preparation for that, here is a question to which I know the answer. Let be a nested collection of subsets of such that for every and . Is it possible for there to be a function and a constant such that for every and every ? In the case that for every this is EDP. But for a general nested collection, it is easy to see that bounded discrepancy is possible for that collection and all its dilates. You just take a completely multiplicative function such as and choose in such a way that for every (and such that the are distinct and every integer is equal to for some ). Then the sum is always either -1 or 0, and multiplicativity ensures that the sum on dilates of the is always either 1, -1 or 0.

The partial sums of grow (it is believed) like , so if we choose our sets greedily — that is, by taking to be the smallest positive integer we have not yet chosen such that takes the value — then will contain the first integers, where is around (at most), and look a bit like a random selection of integers within around of for the rest of the set. In other words, will be pretty close to the set , so the system of sets will be pretty close to the set of all HAPs. This suggests that proving EDP is going to be really quite delicate.

In fact, it will be more delicate still, since we can take a better multiplicative function than . If we take the unique completely multiplicative function that takes to 1 if is congruent to 1 mod 3, to if is congruent to 2 mod 3, and 3 to -1, its partial sums grow logarithmically, so we can say something similar to the above but with around rather than around .

We can go slightly further with this function. Because it is 1 on the infinite arithmetic progression and on the infinite arithmetic progression , we can choose our sets to be intervals up to roughly plus logarithmically short (non-homogeneous) arithmetic progressions of common difference 3.

Since these sets are cooked up so as to ensure that the function has bounded discrepancy, they might seem not that interesting. But I think they are important, because if we want to find an efficient decomposition of a diagonal matrix into HAP products, we probably need to understand what it is that makes HAPs so much better than HAPs with highly structured tiny logarithmic perturbations at the end. Otherwise we are in danger of taking seriously arguments that would work just as well for statements that we know to be false.

As I write, it occurs to me that we might be able to prove a result that would be quite a bit easier than EDP but nevertheless interesting (and in particular interesting enough to be publishable, unless the answer turns out to be disappointingly easy): that there exists a permutation of such that the discrepancy of every sequence on the sets of the form is unbounded. We know it is not true for all permutations — even ones that in some sense don’t permute by very much — and we suspect that it is true for the identity permutation but find that very hard to prove. What about if is in some sense “fairly random”? For example, what if we chose a large and randomly permuted the first integers. That would give us sets and all their dilates. Could we prove that the discrepancy on that set system is at least ? What if we drop the condition that the union of the should be all of ? What if instead we ask that for some strictly increasing sequence ?

I rather like those questions, which makes this a good place to stop the post — for the length of which I apologize.

Here is a very general probabilistic-based heuristic that seems to give good predictions for questions related to EDP. I will refer to this heuristic as “LDH”. (In my polymath5 comments I referred to it as PH – probabilistic heuristic)). I am thankful to Noga Alon and to Yuval Peres for some helpful help.
Here is an example: Suppose we want to study the following extremal problem.

What is the largest number of edges in a graph on n vertices with no triangle.

If we use the probabilistic method we can ask what is the probability that a random graph in contains no triangle. As long as this probability is positive we know that a triangle-free graph with n vertices and m edges exists. (Being a little careful we can consider instead of where . Looking at random graphs gives us a perfectly correct proof of the assertion that there are triangle-free graphs with vertices and edges for every .

LDH:

1) Estimate naively the probability that a random graph in G(n,m) contains no triangle.

2) Choose m so that this estimated probability behaves like 1 over the number of graphs with n vertices and m edges.

So let’s implement this plan. The probability that a random graph in does not contain a specific triangle is . Naively assuming that these probabilities are independent we estimate the probability of not having any triangle as . We want to find so that this probability is roughly . This is the case when roughly .

LDH prediction for Mantel-Turán problem: The maximum number of edges in a triangle-free graph behaves like .

In other words, the LDH gives two predictions that we will refer to as the “firm” prediction and the “weak” prediction.

A) (The firm prediction.) There exist triangle-free graphs with vertices and edges

and

B) (The weak prediction.) There are no (substantially) larger triangle-free graphs.

Prediction A is correct. There are indeed triangle-free graphs with vertices and edges. (But the LDH does not prove their existence.) Prediction B is miserably false: Actually there are graphs with edges without a triangle. The LDH heuristic ignores the fact that not containing one triangle is not independent of not containing another, and is completely blind to large bipartite graphs.

Excercise: What is the LDH prediction for the question: How large can a subset of the integers {1,2,…,n} be if it contains no 3-term arithmetic progression?

We would like to propose the following points regarding the large deviation heuristic:

• LDH predictions about the existence of some combinatorial objects are quite often true. (We refer to such predictions as the firm predictions.)
• LDH weak predictions are blind to various structured examples. Sometimes, if we understand the relevant structures we can update the large deviation predictions. (I will come back to this at the end of the post.)
• LDH predictions appear to be quite good for the Erdős Discrepancy Problem (EDP) and for variations of EDP. This is what we are going to discuss now.

Since LDH is based on a heuristic method to compute probabilities it is quite possible that different heuristics will give different answers but, overall, we did not encounter this.

Problem 1: Find natural examples where the LDH firm prediction, namely the prediction for the existence of certain combinatorial objects, fails.

Problem 2: Find ways to improve the LDH when it fails. (Especially when the answer is known by other methods).

What does LDH predict for EDP?

Direct Computation

We consider the multiplicative version of Erdős Discrepancy Problem . The number of multiplicative -sequences of length is close to . (These sequences are determined by their values on prime indices and the Prime Number Theorem tells us that the number of primes smaller than behaves like .) We expect that the LDH computations for the general question will give the same answer.

What we need to compute is:

What is the value of so that the probability that all partial sums of a random sequence of length belong to the interval satisfies ?

This question can be formulated in terms of a simple random walk of length on the line. We start at the origin and at each step we go a unit length to the left or to the right with probability 1/2 for each direction. We want to know the probability that the random walk will be confined to the interval and the value of for which this probability is .

Problem 3 (the answer is known): What is this value ?

Towards answering problem 2, I asked over Mathoverflow “What is the probability that a random walk of length n will be confined to the interval ” and Douglas Zare provided a very detailed answer. Yet, I did not complete the work needed to answer Problem 2. So a few weeks ago I asked Yuval Peres

What is the probability that the simple random walk of n steps will be confined to the interval , and what is the value of for which this probability is ?

And a few hours later I received the following reply from Yuval:

Gil, the confinement probability in decays up to a constant like where is known: it is . This is classical and you can find it e.g. in Feller volume 2 or in Spitzer’s book. This holds for all . So the answer to your query is that for a suitable .

So, we get:

The LDH prediction for the EDP is that the maximum discrepancy of a multiplicative sequence of behaves like . (The same prediction applies to general sequences.)

General sequences and positive correlations.

If we consider general sequences we get the same answer. We need to compute the probability that for a random sequence of length , all HAP are confined to . These HAP are random sequences of lengths , , . The probability that the initial sums of a random sequence of length is confined to is . If we assume that these probabilities are independent we get an estimate of which behaves like when . We get the same outcome. Of course the confinement of different HAPs to are not independent. In fact they seem positively correlated and this strengthens the case for the firm prediction. But I don’t know how such positive correlation can be used to prove that the firm prediction is correct. (Of course, the feeling that small discrepancy on different HAP are positively correlated is rather tentative. We know that for every individual HAP we have with positive probability discrepancy bounded by 1. Yet the probability that this happens for all HAPs is zero.)

Problem 4: Fix two positive integers . For a function consider the two events

1) For a maximal HAP of gap the initial sums of the function are confined to .

2) For a maximal HAP of gap the initial sums of the function are confined to .

Are these two events positively correlated when is large enough?

Indirect computations

At an earlier time, I had a version of the LDH based on gaps between vanishing partial sums. Let me discuss it here. We start with the following question: What is a probability that a sequence of of length will not have a vanishing partial sum where ? Another way to ask this question is:

What is the probability that a simple random walk of length t will not reach zero in the interval ?

For our purposes all we need to know is that this probability tends to some constant strictly between 0 and 1. The precise value is related to a classic question and let me cite another email by Yuval Peres about it:

The probability that a simple random walk will not meet 0 in the time interval [s,t], where s=xt, tends as to . This is one of the two classical arcsine laws for random walks that you can find in many sources, including e.g. Durrett’s book or proposition 5.7 page 137 in My Brownian book . There you will see this law applies to all random walks with increments of mean zero and finite variance. More combinatorial arguments for the special case of SRW can be found in Feller vol I, as well as in these slides.

Given a random walk on the line we will try to estimate the probability that we can find sequence of indices for which the random walk reaches the origin so that the differences between consecutive indices is between and and compute the value of for which this probability is . Since the probability for a random walk of steps to reach the origin between the th and th steps is a certain constant we have the following LDH predictions:

(Firm prediction) There exists a multiplicative sequence of length such that the gap between two consecutive vanishing partial sums is (up to a multiplicative constant) at most .

(Weak prediction) This is best possible.

Some additional gymnastics allow us to move from the prediction regarding multiplicative sequences of length where all the gaps between consecutive zeros behave like to sequences where, in addition, the maximum discrepancy behaves like . The probability that between every two consecutive zeros the maximum discrepancy will be smaller than is indeed small but if is large this too behaves like so the indirect applications of the LDH based on gaps between consecutive zeroes gives the same prediction as the direct prediction above.

Here too the computation for general sequences gives the same outcome and indicates positive correlation between events which in the heuristic are pretended to be independent.

(Let me remark that over the polymath5 threads there were several remarks in the direction of trying to show that there are no sequences of length where the differences between consecutive vanishing partial sums are bounded for all HAP. This is weaker than what is required to show that the discrepancy is unbounded.)

Variations of EDP and related discrepancy problems:

Variations

Let me mention briefly several variations of EDP and what LDH says about them. The LDH is responsible for the guesses we propose in the previous post for variations E1-E8 of EDP.

The weak LDH predictions will not change if we consider sequences where the zero entries occupy the indices divisible by 3. But the prediction fails in this case (the discrepancy can be bounded).

If we consider only HAPs with prime power differences the LDH prediction is the square root of . I would guess that this is the true behavior. Note that if we consider HAPs with prime differences, we have the same LDH prediction, but since we can have bounded discrepancy the weak LDH prediction is false.

The firm prediction of the LDH predicts a polylog(n) discrepancy (in fact even discrepancy) when we restrict our attention to square free integers and to random subsets of integers.

Probabilistic analogs for EDP

Problem 5: Does the LDH give good predictions for discrepancy problems for random hypergraphs and $on {1,2,…,n}?

Let be the hypergraph in which we consider precisely one edge which contains every element with probability . Let be the hypergraph obtianed fron by adding as edges to all initial segments of edges in . The LDH predicts that the discrapancy of $\cal H$ is bounded. When we move from to then we come back to the prediction. Such probabilistic versions take away the number-theoretic aspect from EDP. Still the probabilities of low discrepancy for different edges are not independent and the problem still looks hard.

LDH and the six standard deviation theorem

Of course, it is of interest to understand the LDH predictions (both for and ) for the general case when we have probabilities and the edges are random subsets based on these probabilities. The most famous case is when we consider all s to be 1/2 and . Joel Spencer’s Six Standard Deviation Theorem asserts that the discrepancy for random subsets of is at most . Nore generally it was proved that the discrepancy of a random hypergraph with edges behaves like when and like for .

Roth’s theorem and how the LDH predicts the answer

Recall that Roth’s theorem is about the discrepancy of the hypergraph whose edges correspond to all arithmetic progressions in {1,2,…,n}.

The LDH predicts the correct answer, at least roughly. The probability that a maximal AP of gap r will be confined to [-K,K] is . When we have to multiply the th power of , for , where is small and this will give us for . The contribution coming from APs of larger gaps will be of a smaller order of magnitide.

Problem 6: Are the methods of proving upper bounds for the disrepancy problem for APs relevant for proving better upper bounds for EDP, its extensions, and its variations?

Large deviations for extremal graph properties revisited

Graph limits and the work of Chatterjee and Varadhan

We started this post by trying to answer a classical problem in extremal graph theory using a probabilistic heuristic which is based on large deviation estimates. It would not be irresponsible to say that the heuristic estimates proposed a very poor prediction to the extremal problem we considered.

Let be a fixed graph and let be the maximum number of edges for a graph on vertices that does not contain as a subgraph. Turán’s 1941 theorem determined the value of when is a complete graph with vertices. Turán’s theorem is the starting point of the wide and deep area of extremal graph theory. The case of triangle-free graphs goes back to Mantel in 1907.

One of the important discoveries in graph theory which is related both to additive number theory and to probability theory is the notion of limits of graphs. This notion is connected to the famous Szemeredi lemma. The theory of limits of graphs sheds new light on extremal graph theory; in a sense it tells us what the relevant structures for are when is not bipartite.

A recent paper entitled The large deviation principle for the Erdős-Rényi random graph by Sourav Chatterjee and S. R. S. Varadhan revealed a connection between large deviation for properties of Erdős-Renyi graphs and graphons – limits of graphs. (It complements a large body of related results obtained by various other methods.) Here is the abstract.

Abstract: What does an Erdős-Renyi graph look like when a rare event happens? This paper answers this question when p is fixed and n tends to infinity by establishing a large deviation principle under an appropriate topology. The formulation and proof of the main result uses the recent development of the theory of graph limits by Lovasz and coauthors and Szemeredi’s regularity lemma from graph theory. As a basic application of the general principle, we work out large deviations for the number of triangles in G(n,p). Surprisingly, even this simple example yields an interesting double phase transition.

So we can “morally” understand why the weak prediction of LDH fails for the property of “including a triangle”. To obtain a better prediction we also need to condition on various relevant limit structures of the random graph.

LDH for the triangle-free process

Consider the following graph process. We start with the empty graph on vertices and add random edges one after the other conditioned on not forming a triangle. Bohman proved that such a process for will lead with substantial probability to a triangle-free graph not containing an independent set of size . This proved a conjecture of Joel Spencer and gave a new proof to a famous result by Jeong Han Kim (with a remarkable history that I won’t describe here). Can we apply the LDH to the triangle-free processes to give a heuristic argument why triangle-free graphs on vertices with edges exist?

Extremal problems for bipartite graphs

Problem 7: Let be a fixed bipartite graph. Does the LDH give good predictions for ?

It turns out that for extremal problems on graphs the LDH gives quite similar predictions to those obtained by the rigorous well known edge-deletion method based on the following proposition:

Proposition: If, for a random graph with n vertices and 2m edges, the expected number of copies of is smaller than half the number of edges, then .

For Turán-type problems, the LDH’s predictions are quite similar to those obtained by the edge-deletion method. (I am not aware of a similar trick for discrepancy problems.)The LDH predicts the existence of -free graphs with roughly times more edges than what the edge-deletion method gives. Achieving an improvement of similar kind is a difficult and well-known problem in extremal combinatorics. See the paper: T. Bohman and P. Keevash. The early evolution of the H-free process. Inventiones Mathematicae, 181, 291–336, 2010.

The LDH prediction are still far from the correct answer. Erdős conjectured that for any bipartite with degeneracy the Turán number is at most , and that the Turan number is iff the graph is bipartite and 2-degenerate. For more on that see, e.g., N. Alon, M. Krivelevich and B. Sudakov, Turan numbers of bipartite graphs and related Ramsey-type questions, Combinatorics, Probability and Computing 12 (2003), 477-494.

The general form of a discrepancy problem

Let be a hypergraph, i.e., a collection of subsets of a ground set . The discrepancy of , denoted by is the minimum over all functions of the maximum over all of

.

We will mention one additional definition, that of hereditary discrepancy. When is a hypergraph and , the restriction of to is the hypergraph with vertex set whose edges are all sets of the form for edges of . The hereditary discrepancy of is the maximum over all of the discrepancy of restricted to .
Here is a link for a recent post discussing discrepancy and the famous Beck-Fiala theorem. The Beck-Fiala theorem assert that if every element in is included in at most sets in then . (Of course, the theorem applies also to the hereditary discrepancy.)

The Erdős’ Discrepancy Problem reviewed

The problem and links to polymath5 material

Erdős Discrepancy Problem (EDP). Is it possible to find a -valued sequence and a constant such that for every and every ?
A HAP (Homogeneous Arithmetic Progression) is an arithmetic prograssion of the form {k,2k,…,rk}. EDP asks about the sum of a sequence on HAPs.
Given a sequence define to be the maximum of sums of subsequences over HAPs which are subsets of {1,2,3,…,n}. EDP asks if we can find a sequence for which is uniformly bounded and we will be interested in finding sequences where grows slowly.
EDP was extensively discussed and studied in polymath5. Here is the link of the first post. Here are links to all polymath5 posts. Here is the link to polymath5 wiki.
A sequence is called completely multiplicative if for every and . EDP is of great interest even if we restrict our attention to completely multiplicative sequences. For those we have only to consider partial sums .

An example where the discrepancy grows like log n.

The function that takes n to 1 if the last non-zero digit of n in its ternary representation is 1 and -1 if the last non-zero digit is 2 is completely multiplicative and the partial sum up to n is easily shown to be at most . Therefore, the rate at which the worst discrepancy grows, as a function of the length of the homogeneous progression, can be as slow as logarithmic.

What a random assignment gives us.

A random sequence (or a random completely multiplicative sequence) gives us discrepancy close to . (There is apparently an additional factor but I am not sure of the precise asymptotic behavior.)

Two greedy algorithms

From now on we write instead of .
Greedy algorithm 1 (for multiplicative functions): Assign the value as to minimize the maximum discrepancy in every partial sum whose terms are now determined.
Greedy algorithm 1 (for general functions): Assign the value so as to minimize the maximum discrepancy in every HAP whose terms are now determined.
Problem 1: How does Greedy algorithm 1 perform?
Empirical observation: (This was claimed in some polymath 5 remarks but I am not sure if there was definite evidence. I would appreciate clarifications.) The discrepancy for sequences based on the greedy algorithm 1 (for multiplicative functions and for general functions) is .
Interpretation: Greedy algorithm 1 optimizes an “irrelevant task”.
We would like to suggest here
Greedy algorithm 2 (for multiplicative sequences): Assign the value so as to minimize the maximum discrepancy in every partial sum where unassigned entries get the value zero.
Greedy algorithm 2 (for general sequences ): Assign the value f(n) as to minimize the maximum discrepancy in every partial sum in every HAP where unassigned entries get the value zero.
Problem 2: How does Greedy algorithm 2 perform?
Omri Floman ran Greedy 2 on inputs such as N=10000 and got a discrepancy of around 20 (there is a bit of randomness involved in cases of ties). For N=100000 he got about 45. It is unclear what the behavior is.
Problem 3: Can we find an optimal compromise between Greedy 1 and Greedy 2?
Problem 4: Can randomization help?
Of course, since we know that a sequence of length n and discrepancy log n exists if we draw a random sequence there is a probability larger than of reaching such a low discrepancy sequence. What we want to ask is if randomization can lead to a method of getting a low-discrepancy sequence with larger probability, or even better with provable larger probability. (Or even better yet, to a sequence of provable low discrepancy via an effective algorithm.)
Problem 5: How do Greedy algorithms 1 and 2 perform if we apply them for a random ordering of {1,2,…,n}.

Extensions of Erdős’ Discrepancy problems

Here are some variations of the Erdős’ Discrepancy Problem along with some guesses for the answers. I will explain where these guesses came from next time.
E0: Erdős’ Discrepancy Problem

Guess:

E1: Allow f(n) to attain values which are complex numbers of norm 1.

Guess, same answer

(Perhaps we can even consider norm-1 vectors in some Euclidean space or Banach space. If this is too premissive () we may go down to a constant.)

Square-free integers

E2: The EDP for square-free integers
Here we simply consider sequences where if is not square-free. For this variation the multiplicative version of the problem is not a special case of the general question. (Here multiplicative means that if are coprime.)

Guess: for both versions, same answer .

Random subsets of integers

E3: Instead of square-free integers consider a random dense subset of integers and assume that the sequence vanishes for indices not in the subset.
Guess: same answer .

Random sets of primes

Here we consider multiplicative functions which are non zero only on a random dense sert of primes.
E4: The EDP for a random dense subset of primes.

Guess: same answer

Problem 6: Find a sequence for problems E2, E3, and E4 with discrepancy , or better or better . Update: see below. It can be shown that sequences with discrepancy exist for all these variations.

Prime power differences

E5: The EDP for HAP with prime power differences.

Guess: .

Beurling primes and integers

Beurling primes were defined by Arne Beurling in 1937 and he also proved a prime number theorem for them. The most general definition is very simple: Consider a sequence of real numbers regarded as “primes” and consider the (ordered) sequence of their products (multiplicities allowed) as the “integers”. (We will assume that all products are distinct although for the original purpose of defining a zeta function multiplicities may play a role.) Beurling primes played a role in the polymath4 discussions.
E6: The EDP for Beurling primes and integers

Careless Guess: at most .

Sune Kristian Jakobsen’s systems of pseudointegers

One way to think about Beurling primes is to identify with and to reorder the integers according to the ordering of the s. Actually, given the ordering we can recover uniquely the Beurling primes. A much more general notion of “pseudointegers” was suggested over polymath5 by Sune Kristian Jakobsen. See also the overview over Polymath5′s wiki.
An ordering of the natural numbers is a SKJ-ordering if it fulfills the following two conditions:
1) If and then .
and
2) for any the set is finite.
Remark: 1) Sune Kristian considered orderings on sequences of integers (the exponents in the prime factorization). This is equivalent to (but perhaps less provocative than) the formulation here. 2) We can expect that Beurling-orderings are a tiny tiny subset of SKJ-orderings.
Given an SKJ-ordering of the natural numbers we can ask the EDP for that ordering.
E7: The EDP for Sune Kristian Jakobsen systems of integers.

Careless Guess: at most polylog n

Problem 7: How does Greedy algorithm 2 perform for variations E2 -E7.
The answers for E2, E3 and E4 is especially interesting because these are examples where the best upper bounds I know behave like and are obtained from a random assignment. (See problem 5.)
Problem 8: Prove the assertion of EDP (namely that the discrepancy is unbounded) for an SKJ-pseudointegers of your choice.

Related Discrepancy Problems

Discrepancy and hereditary discrepancy

Recall that for a hypergraph defined on a ser the discrepancy of , denoted by is the minimum over all functions of the maximum over all of

.

When is a hypergraph and , the restriction of to is the hypergraph with vertex set whose edges are all sets of the form for edges of . The hereditary discrepancy of is the maximum over all of the discrepancy of restricted to .

An operation of hypergraphs

Let be a hypergraph on a vertex set and assume that is ordered, e.g., . Consider the hypergraph obtained from by adding for every set all its initial subsets w.r.t. the ordering. We will consider the operation of moving from to . Note that for EDP all our variations E1-E8 the underlying hypergraph is obtained by this construction . (From a certain natural hypergraphs ). I further guess that for all the variations E1-E8 if we consider the hypergraph (before taking initial segments) then the discrepancy is bounded.

Probabilistic versions

Let be a ground set and let be a set of reals. First consider the discrepancy problem for a random hypergraph with edges, whose ith edge is a random set so that every element of belongs to with probability (and all these events are statistically independent). Next we can move from to as described above.
The case of taking for the probabilities can be seen as a probabilistic analog of EDP. My guess for the discrepancy of this case is also . I also guess that the discrepancy of itself is bounded.
Problem 9: How do Greedy algorithms 1 and 2 perform for the random hypergraph .

Roth’s theorem about the discrepancy of arithmetic progressions

For EDP the ground set is all natural numbers, or just the set {1,2,…,n} and the hypergraph is the collection of all HAPs(homogeneous arithmetic progressions). Roth considered the hypergraphs of all arithmetic progressions. Roth proved that the discrepancy in this case is at . The existence of with discrepancy of was proved by Beck and the factor was removed by Matousek and Spencer. These works by Beck, Matousek and Spencer may be very relevant to prove the existence of low-discrepancy sequences for EDP and some of its extensions.

Some results with Noga Alon

Here are some results proved in collaboration with Noga Alon which are based on the Beck-Fiala theorem and a general argument on moving from to . (It is likely that some of them are known and we will be happy to know.)
Proposition 1: For the discrepancy of described above is at most .
This follows from the following general proposaition:
Proposition 2: Let be a hypergraph on an element ordered set and let be the maximum degree of a point of , then .
Proof: Let be a partition of into two nearly equal intervals, a partition of into 2 nearly equal intervals and similarly , etc ( levels). Now define a new hypergraph obtained from by replacing each set in by the following sets (possibly some empty):

, , , (for all ), .

Note that the maximum degree of a point in is , hence the discrepancy of is at most by Beck-Fiala’s theorem. Also, each initial segment of is a union of at most pairwise disjoint members of . This gives Proposition 1, and since for the case described in Proposition 2 the degree of vertices for behaves like we obtain Proposition 2 as well.
Proposition 3: for versions E1-E8 there are examples where the discrepancy is .
Proof: In all these examples we consider a hypergraph on the ground set A={1,2,…,n}, or on a subest of A with positive density (or for E7 and E8 on another non conventionally ordered set of integers). Then move to the hypergraph of initial subsets of edges of . To apply Proposition 2 and Beck-Fiala’s theorem we need to find an upper bound on the degree of a vertex in the hypergraph $\cal H$. Consider the case of EDP. (The other cases are similar.) The maximum degree is obtained by an integer that is the product of the first k distinct primes. In this case the degree is smaller than . Note that the proof applies even to hereditary discrepancy.
Problem 10: Use similar ideas to prove better (even polylog (n)) constructions for EDP and its variations E1-E8.
I extend all my earlier guesses when we move from discrepancy to hereditary discrepancy.
Proposition 4: The hereditary discrepancy for the hypergraph of HAP on {1,2,…,n} is .
The proof is obtained as follows: Consider the first primes, and a hypergraph on {1,2,…,m} of discrepancy . (For example, a Hadamard matrix of order describes such an hypergraph. ) Next, for every edge consider the integer that correspond to products of primes whose indices are in . Then restrict your attention only to these integers.
Propositions 3 and 4 show that the hereditary discrepancy of the hypergraphs of HAPs in {1,2,…,n} is between and .

A Quick Outlook at Polymath 5: EDP 1-21

Before our quick review of polymath5, let me mention a major difficulty which seems relevant to all approaches:

If we allow zero entries in our sequence, even a small density of zero entries, then the discrepancy can be bounded.

For example consider the sequence 1, -1, 0, 1, -1, 0, 1, -1, 0, …
Experimentations: Computer experiments played a large role in polymath5. One of the most striking discoveries was a sequence of length 1124 of discrepancy 2. Later a second sequence of length 1124 and discrepancy 2 was found but not a larger sequence. Several people made great contributions with computer experiments.
Additive Fourier analysis ideas: Using Fourier analysis on our sequence and somehow reaching a contradiction when we assume that the discrepancy is bounded was a suggestion that was dominant in the first few threads and we came back to from time to time.
Terry Tao’s reduction: Terry Tao found a reduction from the general question to the variation of multiplicative functions with complex norm-1 values (E1). The beautiful proof relies on “multiplicative” Fourier analysis and it is striking how “little” the proof uses.
Problem 11 (Sune Kristian Jakobsen): Does Tao’s reduction apply to arbitrary SKJ-pseudointegers.
Semi-definite programming: A major turning point was a suggestion by Moses Charikar to use a natural semi-definite relaxation for the problem. This promising avenue was explored over several threads and here too computer experimentation was done. One reason to regard Charikar’s approach (and the related linear programming approach described next) as hopeful is precisely because it offers a convincing way to get around the difficulty demonstrated by the sequence 1, -1, 0, 1, -1, 0, …
A generalization and linear programming: The last few threads were centered around a different related relaxation proposed by Tim Gowers. The problem was generalized from a single-sequence problem to a pair-of-sequences problem. (This is a common motif in extremal combinatorics although here the motivation came from functional analysis.) Then relaxation of the problem led to a very appealing linear programming question.
Mathoverflow questions: Mathoverflow was used to ask several questions related to the project.
Participation: Polymath 5 attracted much interest and wide participation. Overall, it did not attract researchers with major prior interest in discrepancy theory.

Discrepancy

Discrepancy theory is a huge an exciting area. Let me just give references to some books.
Jozsef Beck, William W. L. Chen: Irregularities of Distribution, Cambridge University Press, 1987. And, Jozsef Beck: Irregularities of Distribution (Cambridge Tracts in Mathematics Cambridge Tracts in Mathemat Volume 89)(Paperback)
Bernard Chazele: The Discrepancy Method: Randomness and Complexity
Jiri Matousek: Geometric Discrepancy: An Illustrated Guide (Algorithms and Combinatorics)
J. Beck: Combinatorial Games: Tic-Tac-Toe Theory, Cambridge University Press, 2008.
J. Beck: A forthcoming book.
Reading these books will prepare you better to deal with EDP and will enrich your life tremendously.

If you want to look at some of the earlier posts, they are collected together in the polymath5 category on this blog.

Now for the longer answer. A subset of is called a Sidon set if the only solutions of the equation with are the trivial ones with and or and . Since the number of pairs with is and whenever , it is trivial that if is a Sidon set, then , which gives an upper bound for of around .

There is also a matching lower bound, up to a constant. I think it is due to ErdŎs. One notes first that the subset of defined by is a Sidon set inside . Indeed, if , then and . Assuming the quadruple is not degenerate, , so we can divide the first equation by the second to deduce that , and that implies that and , so the quadruple is degenerate for a different reason.

Thus, we can find a Sidon subset of of size , which is the square root of the size of . We can now regard this set as a subset of , and then map each point to the number . It is easy to check that the resulting set is a Sidon subset of .

Much more is known about this: the correct bound is now known up to a constant. However, that is not the theme of this post. What I want to discuss is whether any kind of converse of this result is true. That is, can one say that a Sidon set of size close to must be constructed in some kind of quadratic way?

Some very weak evidence in favour of a conclusion like that is that random methods fail miserably to produce Sidon sets of anything like size . Indeed, let’s pick a subset of randomly by choosing each element with probability . The total number of quadruples with taken from is roughly (since apart from a few edge effects and can be chosen more or less freely and they determine ), and each nondegenerate one has a probability . So the expected number of nondegenerate quadruples with is roughly . For this to be smaller than (so we can remove a point from each one and still have plenty of left), we need to be significantly smaller than , which gives a bound of , which gives us .

There are several other ways to see that random sets of size are very different indeed from Sidon sets, which suggests that Sidon sets of size close to must have interesting structure. But what is that structure?

It isn’t easy even to state a conjecture here, since before we map the subset of to the integers, we can apply an invertible linear transformation to it. It is hard to think of a way of characterizing the kinds of sets that can arise like this, let alone thinking about how to make the structure looser when the set has size only . (Additive combinatorialists will immediately recognise that I am inspired by Freiman’s theorem here.)

But today … make that yesterday as a night has passed since I wrote those two words … I was at a talk given by Javier Cilleruelo, thanks to which I learnt of an example that places a much more serious constraint on any conjecture one might hope to make. The example is one I sort of knew already, since it is based on a brilliant argument of Imre Ruzsa, who created an infinite Sidon set such that the asymptotic size of is around . (As with the finite case, is very easy; the true answer is conjectured to be ; Ruzsa was the first person to beat and his exponent has not been improved.) What Cilleruelo mentioned in passing was that you can use some of Ruzsa’s ideas in an easy way to get a pretty decent example in the finite case. This is something I hadn’t realized. I presume Ruzsa himself was well aware of it, but I don’t remember it being mentioned in his paper (though I haven’t looked at the paper for many years, so he might have done). What I find particularly interesting is that the example is completely different from the graph-of- type examples.

Since the construction is simple (to understand, that is) and rather gorgeous, I thought it merited a blog post. And maybe it can also serve as an advertisement for Ruzsa’s amazing paper.

It begins with the observation that the logarithms of the primes form a Sidon set of reals: if are primes with , then either and or and ; therefore, the same is true if .

“So what?” it is tempting to say. After all, the logs of the primes are not integers. Indeed, there are far more of them than there are integers.

However, at this point a principle comes in that I often feel doesn’t deserve to be as incredibly fundamentally useful as it in fact is: that two distinct integers must differ by at least 1. It’s not too much of an exaggeration to say, for example, that to prove that a concrete number is irrational or transcendental you have to find some way of reducing the problem to this principle. (That is, you say that if your number is rational/algebraic then there must be two distinct integers that have difference less than 1.) For this problem we make a much simpler use of the principle. If , then , since are integers. Since the derivative of is , that tells us that and differ by at least . (One could be a bit more careful here: I think Cilleruelo stated a bound of .)

Note what we now have that’s better: instead of merely knowing that in nondegenerate cases, we have a lower bound on the difference. This allows us to approximate and discretize.

So let’s start with all the logs of the primes up to , of which there are about by the prime number theorem. If we multiply those by say , then all the differences between the distinct sums, which were previously at least , are now at least . Therefore, if we move each number to the nearest integer, the differences will be altered by at most 2, so will be at least 1. So we have a Sidon set! It is a subset of the integers up to and it has size around , so this construction gives a Sidon subset of of size around .

The moral of this is that to prove a structural result about Sidon sets of density close to , then one would probably have to insist on close meaning “within a constant of” (which one would then relax to some slow-growing function later). Or alternatively, one would look for a much more general notion of structure, but I don’t see an obvious generalization of the two examples I now know. Another moral is that it’s worth looking for more examples of dense Sidon sets before even trying to make a conjecture.

I began with a warm-up: I asked how many times a ball was kicked during the recent European Football Championship. Of course, I made it clear that I wasn’t after an exact answer. My first surprise of the day was that pretty well the moment I had asked the question the room went from almost silent to very far from silent: after a while I got used to this. The conversation was, it seemed, entirely about the question at hand. One person asked whether I was interested in the number of passes or the number of actual kicks. I said the latter.

After a while, and when I had managed to quieten everyone down, I drew a logarithmic scale on the whiteboard, going up the powers of 10 from 10 to 1,000,000. I then asked people for their answers and marked them roughly on the scale. There was a significant cluster around 100,000 (by which I mean in an interval from about 70,000 to about 120,000). I then asked people how they had arrived at their answers.

Everybody seemed to know that there had been 31 matches. (Proof: there were four groups of four, so 24 matches at the group stage, and then eight teams left in a knockout competition.) So I wrote that down on the whiteboard. From that point there were two approaches. Unfortunately, I can remember only one of them, which is the one I would have used: to estimate the average amount of time between kicks. The person who proposed this approach thought that was 1-2 seconds, so I asked what we should do with that information. That quickly led to the calculation , which is about . I don’t know whether they were used to doing things like saying “90 is basically 100″ but they quickly got into the spirit of it.

Next up, I said that sometimes one reads of prophetic dreams, and that in order to get a feel for how seriously to take them I wanted to know how significant it would be if somebody dreamt of the death of someone close to them and that person died the next day. So I asked a rather vague question: what is the probability of that happening? I followed that up by asking what information it would be useful to know.

One suggestion that came in quite quickly was the probability that someone dies on any particular day. That led to a discussion of whether we were talking about expected deaths (e.g. if someone is very ill or very old) or unexpected deaths. There was general agreement that if someone is expected to die soon, then an apparently prophetic dream would not be too surprising, since both the dream and the death are more likely. So we decided to restrict attention to unexpected deaths. In the end we calculated the probability by simply estimating the number of days in 70 years — we went for 25,000 — and taking the reciprocal.

What else did we need to know? At this point an interesting discussion arose. One person suggested that a relevant parameter was the number of people that a typical person knows. That led to a subdiscussion about how close an acquaintance needed to be to be counted. I suggested that pretty well any identifiable acquaintance would be surprising enough to be worth counting. One other clarification asked for was whether we were talking about reports of prophetic death dreams or true reports of prophetic death dreams. I said the latter.

I haven’t yet said what the interesting discussion was. It was that some people thought it was relevant how many people people typically know, while other people thought it was not relevant at all. The arguments were roughly as follows. Those who thought it was relevant said that if somebody died, then you needed to know how many people they knew if you wanted to assess how likely it was that one of those people had dreamt of their death the night before. Those who thought it was not relevant argued that if someone dreams of a death, then all you care about is the chances that the person dreamt about will die the next day.

Did we have a paradox? Somebody gave a reasonable explanation of why we didn’t, which I made a bit more precise by saying that we were conditioning on two different events (the occurrence of a dream and the occurrence of a death), so it wasn’t too surprising that we had to take into account different things.

But the problem still wasn’t very clear, which itself became clear when people wondered whether we needed to know how many people there were in the whole world. In the end I stated the problem more precisely: roughly how many times per year would we expect somebody in the UK to have a death dream that comes true unexpectedly the next day? We settled on the following method: estimate how many times a death dream occurs per person per year, multiply by the population of the UK, and divide that by 25,000. I asked people how often they had death dreams per year and there were very different answers: some people said they thought 10 was about average and seemed surprised when I said that I thought that for me it was more like one every two years. We settled on 3, which I still found a bit high, but not outrageously so.

So that led to the calculation . I think we had already multiplied 3 by 50,000,000 to get the number of death dreams per year, so what I actually wrote down was 150,000,000/25,000. So I naughtily changed that to 250,000,000/25,000 which gave me 1,000 and then compensated for the naughtiness by dividing that by 2 to get 500. That confused one or two people, so I explained it again more slowly.

While I can remember which questions I discussed during the session, I can no longer remember the order in which I covered the next three. However, I think it was either at this point or after the next question I discussed that there was a short break — not that it is too important. Anyhow, at some point round now I described Terence Tao’s airport-inspired puzzle, which is the following question. You want to get from one end of an airport to the other and your shoelace is undone. If you want to get to your destination as quickly as you can, is it better to tie your shoelace when you are on a moving walkway or when you are on stationary ground?

The moment I had finished asking the question, the noise switch was in its “on” position again, which was fine. After I had managed to quieten the room down again (which was hardish but not impossible), I took a vote. The options I gave them were “faster on walkway”, “faster off walkway”, and “it makes no difference”. “Faster on walkway” got a few votes, “faster off walkway” got one vote, and “it makes no difference” came top, though not by miles. There were also a number of abstentions.

The next thing I did was invite people to argue for the answers they had voted for. The one person who had voted for “faster off walkway” did not look at all keen to do this, but after a little while someone else said, “You would need to know the speeds of the walkways, your walking speed, and the time it takes to tie your shoelace,” which prompted me to add a “not enough information” option, which was popular, and to which the “faster off walkway” person transferred her vote.

Here are some arguments I collected. In favour of the “on walkway” answer was that when you tie your shoelace you are moving along rather than wasting time being still. In favour of the “makes no difference” option was that wherever you tie your shoelace it takes the same amount of time. And when I tried to elicit an argument for the “not enough information” option, I failed: I asked under what circumstances it would be better to stop on the walkway and under what circumstances it would be better stop off the walkway, but nobody had much to suggest. I also explained that while knowing all the various speeds and times is sufficient to work out the answer, it isn’t obviously necessary: for example, if you are deciding whether it is better to use the walkway or not use it, you don’t need to know how fast it is.

A notable aspect of the discussion that permeated the whole thing was that people were keen to introduce non-mathematical considerations. For example, they (rightly) pointed out that it would make a difference if the airport was crowded: if the people just ahead of you on the walkway are stationary and you can’t get past, then obviously that’s a good moment to tie your shoelace, and if the people just behind you are in a hurry, then you will annoy them by stopping. So I asked them to assume that the airport was deserted. Other assumptions that had to be spelt out were that your walking speed is unaffected by having a shoelace undone and that there isn’t a robot sensor on the walkway that detects that you are tying your shoelace and stops it. Later on I found out who the mathematicians were and who the non-mathematicians were. It turned out that many of these objections came from the non-mathematicians. They were very helpful to the discussion, since one of the points I wanted to get across was that mathematical modelling involves choosing appropriate simplifications. One of the non-mathematicians begged me to say what the answer was (well before I wanted to do so) so the question had clearly worked its magic.

If I had been discussing this question as part of a genuine course, I would have waited much longer for someone to come up with a good solution. But as I wanted to get through several questions, I hurried things along a bit. I started by offering the advice to consider extreme cases. To illustrate what I meant by an extreme case, I suggested that maybe the walkways could be very very fast. (Actually, I think that would have helped people guess the answer, but it wasn’t the extreme case I had in mind.) Pretty soon after that, someone suggested considering the case where someone starts tying their shoelace just after getting on a walkway. Since that was exactly the case I was interested in, I basically gave away the answer at that point by suggesting also considering the case where someone starts tying their shoelace just before getting on the walkway. Most people could see that that was a proof that doing it on the walkway was better. One or two had a bit of trouble still, so I drew some pictures of two people and what would happen if one stopped just before and one just after getting on.

If I’d had more time I would have tried to get everyone to understand why the “you spend the same amount of time wherever you do it” argument gives the wrong answer.

Another thing I did at around this stage was what I call the dividing-up-the-pot game. I first asked question 41 from my earlier post, which is this. You and a friend are out for a walk, when you are approached by a stranger, who offers the two of you £1000 on one condition: that you agree how to split it between you. After establishing to your satisfaction that you are not about to be kidnapped, you propose a 50-50 split to your friend. To your astonishment, your friend insists on receiving £900 with only £100 going to you, and appears to be prepared to lose all the money rather than accept anything less than this deal. What should you do?

The main person to make a suggestion suggested that you should call your friend’s bluff, since when the stranger starts to walk away, your friend would be bound to say, “OK OK stop! I’ll settle for £500!” I wasn’t sure what more to say at this point, so I told them that they would now have a chance to see for themselves. We would have five rounds, in each of which people would be randomly paired and would have the chance to share ten points, on condition that they could agree how to do so. At the end of the five rounds we would see who had the most points. To do the random pairings I had brought along (part of) a pack of playing cards, which I shuffled and dealt out once per round, and each person was paired with the person who had a card of the same colour and same denomination.

It took a while to get through the rounds, but once it was finished, I asked whether anyone had 30 or more points. Several did. 35 or more? Nobody. 34? Yes, one person.

I then asked people what their strategies had been. To my surprise, several people had used randomized strategies, the main one being to decide by the toss of a coin (or in one case stone, scissors and paper) who would get all the points. I asked one person why he had adopted that strategy. His response was that the main aim of the game was to come top, which one couldn’t do by sharing points out equally and couldn’t easily do by insisting on lots of points each round. So he judged that the all-or-nothing strategy gave him his best chance of coming top. He turned out to be one of the people doing maths and further maths. I think his strategy got him 30 points. The person who got 34 points got them by, as he put it, “threatening his opponents.”

I then started a discussion of what possible other strategies there might be. I had planned to think about a population of strategies and how they would perform against each other, making a link with things like the evolution of morality, but after a few seconds I aborted it because I thought it would be too long and complicated. But this was something else that I would definitely have tried to do if I had devoted more time to this one question.

Similarly, I had a brief discussion of what real-life situations might be reasonably well modelled by this game. There were some suggestions that I didn’t like and now can’t remember. (The problem with one of them was that there was no analogue of getting zero if you can’t agree.) There was one about haggling, which seemed fine. I gave the example of the Conservatives and Liberal Democrats drawing up their coalition agreement two years ago, and said that any negotiation where there is a range of outcomes that would benefit both parties could be thought of as an instance of this game. If I had had more time I would have held back for much longer before giving these examples.

Next (or possibly previously), I asked question 37(i), which is the following. In several parts of the UK the police gathered statistics on where road accidents took place, identified accident blackspots, put speed cameras there, and gathered more statistics. There was a definite tendency for the number of accidents at these blackspots to go down after the speed cameras had been installed. Does this show conclusively that speed cameras improve road safety?

The same person who argued for the randomized strategy in the negotiation game basically knew the answer to this question already. He said no, since if you pick out the extreme cases then you would expect them to be less extreme if you run the experiment again. I decided to move on quickly from this question since there wasn’t a lot more to say. But I told people about a plan I had had, which was to do a bogus telepathy experiment. I would get them to guess the outcomes of 20 coin tosses, which I would attempt to beam to them telepathically. I would then pick the three best performers and the three worst, and would toss the coins again, this time asking the best ones to help me beam the answers to the worst ones. People could see easily that the performances would be expected to improve and that it would have nothing to do with telepathy. One person said, “Oh I want to do that,” of the experiment.

Penultimately, I discussed the following table, slightly modified from an example of Joseph Malkevitch. There are five options, and the orders of preference amongst 55 people are as follows.

14532 — 18 people

25431 — 12 people

32541 — 10 people

43251 — 9 people

52431 — 4 people

53421 — 2 people

Which option should the group go for? (I gave examples of different film genres for the options in a key below, but we didn’t use those.)

I asked for relevant observations. The group seemed pretty clued up on voting methods, so I was soon writing up that option 1 would win under first past the post, and someone calculated that option 3 (if I remember correctly) would win under the alternative vote. I then suggested doing pairwise comparisons, and was very gratified when everyone thought that transitivity was obvious (though one person did say that we should consider not just the preferences but the strengths of the preferences), so I could surprise them by showing that it failed in this case. I then gave the simple 123/231/312 example. Some people were keen on the system where you award 5 points for a first preference, 4 for a second and so on. We could probably have discussed this for quite a bit longer. I briefly mentioned Arrow’s theorem, without stating it precisely, and moved on to my final question.

I told them that for the last couple of years I have been reading Proust. (I asked how many of them had heard of Proust. One had.) More precisely, I have been reading Remembrance of Things Past, Scott Moncrieff’s translation of Proust. The edition I have is in twelve volumes, and my practice has been to read one volume of Proust for every two or three other books I read (my reading taking place before I go to sleep at night). I’ve now read eight of the volumes, and on page 25 of the eighth, I came up against this sentence. (I didn’t tell them that the eighth volume was the second half of “Sodom and Gomorrah”, which Scott Moncrieff coyly translates as “Cities of the Plain”.)

But if the course of life, by making Cottard assume, if not at the Verdurins’, where he had, because of the influence that past associations exert over us when we find ourselves in familiar surroundings, remained more or less the same, at least in his practice, in his hospital ward, at the Academy of Medicine, a shell of coldness, disdain, gravity, that became more accentuated while he rewarded his appreciative students with puns, had made a clean cut between the old Cottard and the new, the same defects had on the contrary become exaggerated in Saniette, the more he sought to correct them.

I displayed it for them and asked how we might go about checking that it was syntactically correct, and, even better, actually understanding it. It was projected on to a whiteboard, so I was in a position to annotate it. Fairly quickly the suggestion came in to use brackets. Somebody also said that we should work from the inside outwards. So I asked where people would like the brackets to go. The first suggestion was to have a bracket opening before “because of the influence” and ending after “Academy of Medicine”. That looked right, but turned out to be wrong: the opening bracket needed to be before “if not at the Verdurins’”. However, eventually we sorted it all out, and I got them, with a bit of prodding, to say that the general principle we were using was to put brackets round bits of text if the sentence still makes sense with those bits removed. (It was because it initially looked as though one could jump from “where he had” to “a shell of coldness” that we made the initial mistake.)

If it had not been inconvenient to do so, I would have rewritten the sentence with indentations, as follows.

But if the course of life,
….by making Cottard assume,
……..if not at the Verdurins’,
…………where he had,
…………….because of the influence that past associations exert over
…………….us when we find ourselves in familiar surroundings,
…………remained more or less the same,
……..at least in his practice, in his hospital ward, at the Academy of
……..Medicine,
….a shell of coldness, disdain, gravity,
……..that became more accentuated while he rewarded his
……..appreciative students with puns,
had made a clean cut between the old Cottard and the new,
the same defects had on the contrary become exaggerated in Saniette,
the more he sought to correct them.

That would have shown more clearly just how many levels of subordinate clauses there were in the sentence. I would also have talked about trees and perhaps broadened the discussion into one about parsing trees for other sentences (which would have required them to decide and try to justify what the natural tree structure of a sentence is).

I didn’t do this at the time, but it is quite interesting to try to rewrite the sentence to make it more comprehensible. Here is what happens if instead of chopping clauses in two, you try to put subordinate clauses at the end. For example, instead of, “He decided, because he was in a good mood, to smile at people he would usually have ignored,” you write, “He decided to smile at people he would usually have ignored, because he was in a good mood.” The sentence above can be reordered as follows.

But if the course of life had made a clean cut between the old Cottard and the new, by making him assume a shell of coldness, disdain, gravity that became more accentuated while he rewarded his appreciative students with puns, if not at the Verdurins’, where he had remained more or less the same because of the influence that past associations exert over us when we find ourselves in familiar surroundings, at least in his practice, in his hospital ward, at the Academy of Medicine, the same defects had on the contrary become exaggerated in Saniette, the more he sought to correct them.

It still isn’t easy, but it’s easier. Does it lose something important? I think it loses some of its distinctively Proustian character — decoding this kind of sentence is part of the peculiar pleasure of reading Proust — but why it should be pleasurable rather than just a nuisance is hard to say.

I might add that the sentence is fairly typical of the whole book, though not many sentences go quite that far. And there are similar phenomena at other scales: paragraphs sometimes go on for several pages, and chapters can go on for hundreds of pages. So not only does one have to read twelve volumes of 350-400 pages each, but the reading is slow going. Out of curiosity, I looked up the sentence in the original French. Interestingly, it uses one pair of brackets, which makes it a lot easier to understand. (What I mean by “interestingly” is that I wonder why Scott Moncrieff decided to do without the brackets.) Here it is. I got it from an online version.

Mais si la vie, en faisant revêtir à Cottard (sinon chez les Verdurin, où il était, par la suggestion que les minutes anciennes exercent sur nous quand nous nous retrouvons dans un milieu accoutumé, resté quelque peu le même, du moins dans sa clientèle, dans son service d’hôpital, à l’Académie de Médecine) des dehors de froideur, de dédain, de gravité qui s’accentuaient pendant qu’il débitait devant ses élèves complaisants ses calembours, avait creusé une véritable coupure entre le Cottard actuel et l’ancien, les mêmes défauts s’étaient au contraire exagérés chez Saniette, au fur et à mesure qu’il cherchait à s’en corriger.

What is the Forum of Mathematics?

Terminological questions aside, how will this new journal-like object work? I think the easiest way of explaining it is to describe the process for submitting an article, which is similar to the process for submitting an article to a conventional maths journal, but with one or two unusual aspects.

First of all, as with any journal you’ll need to decide whether your article is likely to be of the required standard. Except that here there are two standards to worry about, which brings me to the main unusual feature: the Forum of Mathematics, as the journal(s) is called, has two “levels”, called Pi and Sigma. A paper suitable for Sigma should be of the kind of standard you would expect in a leading journal in the area of that paper. For example, if your paper is in combinatorics, then it will be suitable for Forum of Mathematics: Sigma if it is of roughly the standard expected by Combinatorica or JCT A/B. As for Pi, that is for papers that are sufficiently interesting or important that their appeal goes beyond their immediate area of mathematics. Thus, Pi papers will be at the level of leading general mathematics journals and will be an open-access alternative to them. Discussion is still going on about what precisely this means, but it looks as though the aim will probably be for Pi to be a serious competitor for Annals, Inventiones, the Journal of the AMS and the like. Of course, we can only really know what the level and characters of the two journals will be when they have been going for a while. The CUP website says this:

Pi is the open access alternative to the leading generalist mathematics journals. Papers published will be of a high quality and of real interest to a broad cross-section of all mathematicians.

It will also be possible for papers submitted to Pi to be reconsidered for Sigma, if the author is willing for this to happen. However, to encourage authors to think hard before submitting to Pi, rather than merely trying their luck with it, there will be a requirement that papers submitted to Pi are accompanied by a justification of a side or two, which should explain why the paper is of more than merely specialist interest. I hope that these justifications will be very useful documents for the editors, and perhaps even for readers of the papers at a later stage, but that kind of detail has not yet been decided.

The editorial board of the Forum of Mathematics will be divided into “clusters” of people, with each cluster representing a different area of mathematics. When you submit a paper, you will decide which is the appropriate cluster to submit it to, and the editors in that cluster will handle the paper. However, not all papers can be easily classified, and that is where the not-quite-singular-but-not-quite-plural aspect of the Forum of Mathematics becomes apparent. It’s not just papers that are hard to classify, but also editors, and some editors will belong to more than one cluster. Also, there will be plenty of communication between editors of different clusters in an effort to achieve a reasonably uniform standard across the whole of the Forum of Mathematics.

Otherwise, the journal will be pretty conventional. Once you’ve submitted a paper, it will be processed in basically the same way as papers are processed for any other journal. A small but important difference is that the Forum will not have “issues”. As with many other electronic journals, once a paper is accepted, it goes straight up on the website. It will have a number for reference purposes, and if it is a Sigma paper then it will in some sense “belong” to the relevant cluster, but it won’t belong to an issue (though there will be “volumes” for those who want print copies). One of the advantages of this is that the Forum of Mathematics will be able to aim for an absolute standard rather than taking space into account when deciding whether or not to accept papers.

Speaking as one of the editors, I’d like to say that this is something I feel very strongly about. It is very unclear at this stage how popular the Forum of Mathematics will be. I would hate to be under pressure to lower standards in order to attract more papers, or to turn away good papers because there wasn’t room for them. An electronic journal makes it easy to avoid that kind of pressure, and for the Forum of Mathematics it will be avoided.

How will the journal be paid for?

So far, what I’ve been discussing is only rather minor modifications of the normal practice of journals. But I said at the beginning of this post that the Forum of Mathematics will be open access. What does that mean, and how will the finances work? Before I answer this, let me briefly introduce some terminology.

It is unfortunate that the phrase “open access” has come to mean different things to different people. The result is that if you say that you are in favour of open access, you have to clarify what you mean, since there may be business models that you are not in favour of but that are nevertheless called open access by their proponents. However, there are two types of open access, known as gold and green, that have well-established meanings. Gold open access is where a paper is published in a journal and made freely available online, and the publisher is paid by the author. So a more descriptive name would be author-pays open access. Green open access is where papers are published in the usual way, but also made available on repositories such as the arXiv, so that even if the formatted journal versions are behind a paywall, freely downloadable versions of the papers will still appear when you Google them.

The advocates of green open access argue that if everyone makes a habit of posting their papers on places like the arXiv, then in time the problems we have with expensive journals will melt away: there will simply no longer be any point in subscribing to them at ridiculous prices. Opponents of green open access, and especially of mandates from funding bodies that people should make their papers available online, use exactly the same argument but with one extra line: if everybody did it, then there would be no point in subscribing to a journal, and therefore the journals as we know them would no longer be financially viable.

I will resist the temptation to discuss these arguments in more detail, because gold open access is more relevant to the Forum of Mathematics (though papers submitted to the Forum of Mathematics can also be posted on the arXiv, so it is completely sympathetic to green open access). Here there is bad news and good news. The bad news is that it will cost money to have a paper published in the Forum of Mathematics. In other words, it is following a gold open-access model. The good news is hidden in the word “will”. For the first three years of the journal, Cambridge University Press will waive the publication charges. So for three years the journal will be what Marie Farge (who has worked very hard for a more rational publication system) likes to call diamond open access, a quasi-miraculous model where neither author nor reader pays anything.

A second piece of good news will seem like good news only if you know what typical author publication charges, or APCs as they are commonly called, are. (Actually, APC officially stands for “article processing charge”.) Here are a couple of examples. One of the most famous open-access journals is the Public Library of Science, or PLoS. PLoS has several high-quality journals and a big all-encompassing journal called PLoS-ONE that has a high acceptance rate and is basically somewhere where you put a paper if all you want to signal is “this was a publishable paper”. The APCs are between $2000 and $3000 for the high-quality journals and are $1350 for PLoS-ONE. The details are on this page from their webiste. Another example is the London Mathematical Society’s open-access option. If you want your LMS-journal paper to be freely available after it is published, then you can have that for a fee of $3050 (or £1925) from outside the UK or £2310 from within the UK. The UK figure includes VAT of 20%.

The proposed charge for the Forum of Mathematics after the initial three years is £500 or $750. These figures do not include VAT, and are in absolute terms quite a lot of money, but they are nevertheless much cheaper than what appears to be the industry standard at the moment. They will also be waived for people from developing countries (CUP has a list of the countries for which fees will be waived) and for people who can demonstrate a genuine inability to pay. In addition, CUP promises to be transparent about its costs, so one will be able to understand the justification for the fees. I do not know the exact form that this transparency will take: what I would like to see is a web page somewhere that discusses in detail the cost of processing a typical paper, but maybe they won’t go that far.

A further point is that CUP would like to keep the fees as low as possible, even after the three years are over, and will seek funding to do so. The dream scenario would be a rich donor agreeing to underwrite APCs for several years after the initial three-year period. But there are also smaller things that can be done. For example, some people are at institutions that routinely agree to cover author charges. Such people will be encouraged to pay the author charges if they get a paper into the Forum of Mathematics, since it will not be a problem for them to pay, and the money received will be used to mitigate in one way or another the introduction of APCs in three years’ time. I am being slightly vague here because there are different forms that this mitigation might take: for example, there might be a choice between waiving APCs completely for a further period and half waiving them for longer.

Some pros and cons of gold open access.

I have talked to a number of mathematicians about author publication charges, and it is clear that the idea is regarded with deep suspicion by many people. It is also clear that many of the arguments that people have against it are good ones. Before I go into some of those arguments, I would urge you to bear in mind that there are also some very strong arguments against the current subscription system. It seems to me that every system of publication has its drawbacks, so pointing out those drawbacks is not enough: one must argue that they outweigh the drawbacks of the existing system.

The main drawback of gold open access is that it makes life more difficult for people from less wealthy institutions. More generally, it introduces financial considerations into certain decisions that one would prefer to be made entirely on academic grounds. An example of the kind of consequence this might have if gold open access became the norm is a less well-off UK university telling its academics that it would pay for just the papers needed for the next Research Excellence Framework. In fact, I have even heard a rumour of this kind, and apparently the decision about which papers would be supported was to be taken by the university administration and not by the academics. I very much hope that won’t be typical.

One of the best ways to avoid unpleasant consequences of that kind is to make the charges small enough that they are relatively painless for universities and funding bodies to pay. Maybe the charges for the Forum of Mathematics could be lower still — I don’t know — but the fact that they are much lower than the ones that are prevalent at the moment is at the very least a significant piece of progress. (I would add here that Rob Kirby, who will be the managing editor and who has been campaigning against expensive journals for many years, regards the £500 price as an absolute bargain.)

An important point to make is that while libraries continue with their subscriptions to existing journals, APCs are simply an additional cost to universities and funding bodies. However, if we were to switch tomorrow to an author-pays model (that is, stop all subscriptions and do all publishing through gold open access), then the total cost of the publication system would be much less than it is now. Or at least so I have read many times, and I find it plausible. For instance, I have been told that University College London pays over a million pounds a year to Elsevier for its subscriptions. Even if APCs were £2000, a million pounds would be enough for 500 papers. Elsevier has 2000 or so journals, so I would guess that the total number of articles is something like 30,000. If 1% of these articles come from University College London (surely an overestimate — many of those 2000 journals are not all that high quality and there are many good universities round the world), then that’s 300 papers, so already a significant saving. Improvements to this back-of-envelope calculation are welcome. (Of course, APCs of £500 alter the calculation dramatically.) Thus, one needs to take a long-term view when thinking about APCs. I see the pain of paying APCs as to some extent a good thing: if academics themselves feel that pain (not quite directly but through their departments or other funders) then they will be less likely to accept outrageous prices than they are now, when the pain of journal prices is felt by librarians.

Since switching to an author-pays model would save universities a lot of money, one option that looks very attractive on the face of it would be for a large number of universities to club together and agree to fund APCs for journals that were sufficiently cheap and of sufficiently high quality. If that could be organized, then the consortium of universities would have much more bargaining power than libraries do at present: a journal would much rather be able to say, “Don’t worry about APCs — they are paid for by consortium X,” than have explicit APCs. (Why? Well, which kind of journal would you prefer to submit to?) So with luck there would be significant downward pressure on prices. The difficulty, of course, is of a prisoner’s-dilemma type: while it may be in the collective interest of universities to set up a fund to cover APCs, any individual university would do better by not contributing to the fund. However, if enough universities could be persuaded to start such a fund, then moral pressure could be put on other universities to join in — especially wealthy universities whose members frequently benefit from the fund. I hope that the Forum of Mathematics will quickly become such a valued part of the mathematical landscape that many of us will make efforts to bring a fund like this into existence (perhaps initially just for FoM, but ideally the fund would grow and support other open-access journals).

If you are interested in arguments for and against various kinds of open access, I would recommend the writings of Peter Suber, who also has a book coming out soon.

My plans as an editor.

Here I am speaking for myself only. The idea of taking on a significant editorial job was not one that would ordinarily have appealed to me, but when David Tranah (whose brainchild all this was) summoned me for a chat in early February, shortly after I had published my first Elsevier post, I realized that it was basically impossible for me to say no.

Having got myself into that position, I now want to think of ways of doing the work efficiently. Here are a few ideas I have had so far.

1. I will follow the practice of many editors these days and ask for quick opinions first, unless I myself already have a quick opinion. I will proceed to a more detailed reference only if the initial opinion suggests that it is worth doing so.

2. One reason I have been rather inefficient at this kind of job in the past is that I always feel guilty asking potential referees for a favour. Maybe I shouldn’t, but I do. To counteract that, I would like to compile a list of names of people who volunteer in advance to do a certain amount of work for the discrete maths cluster of the journal (which is the cluster I’ll be part of). What I’d like is for people to tell me roughly how often they are prepared to handle a paper and roughly what topics they are ready to cover. That way, if I have a paper that matches a referee who has not yet reached his/her quota, I’ll be able to ask that person without the slightest embarrassment. If you are reading this and feel like showing your support for the journal by dropping me an email and making a commitment of that kind, I will be very grateful. If you don’t, and if you are a discrete mathematician, you may find that I email you at some point. I won’t hold anyone to the commitments that they make. The main purpose of this is to make it easier to ask for help, but if you’ve said you can handle three papers a year and a long and difficult paper comes along at a bad time, I will understand.

3. I plan at some point to write some guidelines for what I personally consider makes a combinatorics paper good enough to publish in Sigma and what gives it that extra Pi-like quality. This probably won’t be easy, but I hope that authors and referees will find it useful to have some explicit criteria (which they will be free to ignore) rather than relying on some vague instinct that “this paper is good enough for the Forum of Mathematics: Sigma”.

How does all this relate to the Elsevier boycott?

Here again I am speaking for myself rather than for CUP, though I hope my opinions will be shared by many others. As it happens, the idea of the Forum of Mathematics predates the Elsevier boycott, so in a sense the answer is that it has nothing to do with the boycott at all. However, I myself see it as potentially a very important development in the campaign for a better system of academic publishing. In particular, it greatly weakens what was previously quite a strong argument for some people against participating in the Elsevier boycott: that the best specialist journal in their area is an Elsevier journal so joining the boycott would harm their career.

Well, maybe that has been true up to now, but it is about to become less true. If you have a paper that is suitable for the best specialist journal in your area and that journal happens to be very expensive, then you now have the option of submitting your paper to Forum of Mathematics: Sigma instead. Similarly, if you were thinking of submitting a top-notch paper to a top-notch but expensive journal (Inventiones comes to mind here — a Springer journal, but this discussion is not just about Elsevier), then you can get just as much of a career boost from Forum of Mathematics: Pi but also enjoy the warm glow that comes from knowing that your paper is freely available to all mathematicians.

At some point I hope to compile an informal list of expensive journals of roughly the same standard as Forum of Mathematics: Sigma, though for that I’ll need help, as I have very little idea of the standards of specialist journals outside my areas of mathematics.

It is of course unlikely that the Forum of Mathematics will change the face of mathematical publishing in three years. To be a serious direct threat to Elsevier’s mathematics journals, for example, it would need to cause a reduction of their quality by enough to make libraries consider cancelling their subscriptions, which is very difficult when mathematics journals are bundled together with journals from other subjects. However, the Forum of Mathematics can still have a big influence. For one thing, it will demonstrate that a major publishing house can produce a high-quality journal with high-quality formatting and editing with APCs of around £500. I hope that people in charge of funding bodies who are considering open-access mandates will ask some tough questions of publishers who continue to charge four times that. Secondly, if the Forum of Mathematics is successful, it has the potential to reduce the quality of a number of other journals, which will at least strengthen the hand of librarians who are bargaining with publishers like Elsevier (because cancelling their subscriptions won’t cause quite the inconvenience that it would at the moment). Thirdly, the Forum of Mathematics may encourage other publishers to set up cheaper open-access journals — one huge gap in the market would still be a place where people could submit papers that were perfectly worthy but not good enough for the Forum of Mathematics. I myself think that for such papers, there is a strong case for not putting in too much effort into formatting and typesetting, since the papers will usually not be read all that much, and since many people can produce a decent typescript themselves. So it ought to be possible to produce such a journal (or journal-type object, if it is like the Forum of Mathematics) with smaller APCs.

And even if we forget all about price, we still have the huge bonus that the papers published in the Forum of Mathematics will be freely accessible. I hope that once people start to get used to a high-end journal being freely accessible, they will feel all the more keenly the inconvenience of paywalls.

So I urge you to support the Forum of Mathematics. If you’ve got a good paper, then why not hold on to it until 1st October and submit it to us? If the Forum of Mathematics takes off, then you will be able to look back with pride and say that your paper was one of the very first — who knows, perhaps even the first — to appear in it. If your department has not yet thought about author charges, then another thing you can do is initiate a discussion about those — something I plan to do in Cambridge — since, as I said above, any author fees that are paid voluntarily during the first three years will help keep the fees lower later on and increase the chances that the venture will be a success, which in turn will raise the prospect of saving a lot of money in the longer term.

As regular readers of this blog will know, I am keen on the idea of much more radical changes to the way we evaluate and disseminate our work. However, I am also in favour of evolutionary change rather than a sudden collapse of the existing system, and that is what the Forum of Mathematics offers. It is not a solution to all our problems, but neither is anything else. What it is is another way of doing things, and the more of those we have, the greater the chance that some of them will work and become successful and help us move to a cheaper and more open system of publishing.