The file is “data-file-rectangular-area-4-length20″ in http://abel.math.umu.se/~klasm/Data/EDP/DECOMP/

The best solution in this class for N=66 is better than the best symmetric solution for N=840, and has only coefficient 1/12, with positive 2×2-squares and a single negative 1xK strip. However unless I am missing something a solution with that structure can never take us below C=1/2

What you say in the last paragraph is what I was saying, but more vaguely, in my last comment but one. I think it might be interesting to look at some of the experimental evidence coming out of the LP problem in order to get a feel for what the dual solutions are like — what I would be hoping is that it would be easier to spot patterns in the dual solutions.

I suppose another idea, though it might be rather too ambitious, would be to try to think of a strengthening of the discrepancy statement, insisting on lower discrepancy for some squares than for others, so that all the constraints become tight. Equivalently, one would attach different weights to the squares in a decomposition, so that some of them were more expensive than others, in the hope that the best decomposition then used the squares much more uniformly.

It may be worthwhile to look at LP solutions for inspiration in constructing functions that have low discrepancy on squares. In fact, if the squares and fly-traps construction does arise as an optimal solution to the LP, then the dual solution is a low discrepancy function with maximum discrepancy equal to the bound established by squares and fly-traps. One would hope that there should be considerable structure in this dual solution. One problem with looking for structure in these LP solutions is that you need large values of since . A trick to handle this issue is to start at instead of , so plays the role of the highly divisible number. But I’m not sure if starting at makes sense for the current discussion. In any case, it may be an easier variant to think about because there was a very definite trend in the optimal values of the LP for the problem over – in fact the optimal value seemed to be exactly .

I’m a little rusty, but I recall from looking at these LP solutions previously that it appeared that the optimal dual solution was probably not unique and there were some extra degrees of freedom. Even so, in principle, one should be able to determine the form of these dual solutions if indeed the squares and fly-traps construction arises as an optimal solution to the LP. What this tells us is that the squares used in the linear combination for the squares and fly-traps are the squares where the maximum discrepancy is achieved – all other discrepancy constraints are not tight. The tight constraints (provided we also know the sign for each of them) should determine a family of dual solutions and there must exist some choice of free variables such that the dual solution obtained satisfies all the discrepancy constraints.

An important difference between this question and most discrepancy questions is that we are not assuming that the function takes values but instead are assuming that there is one vertex that belongs multiply to many hyperedges. It might be reasonable to assume that the ray is the only ray that intersects any square in our collection more than once. In that case, we are trying to show that the effect is to give a positive kick to the sets in our collection that we cannot manage to cancel out.

What I’d like to investigate is the number-theoretic question of how to find squares that create a hypergraph that has even the remotest chance of achieving this. For instance, if any square contains a point that is the only point in its ray that intersects a square in our collection, then we can give that point any value we like and cancel out the discrepancy on that square. So we may as well get rid of that square.

I think I’ll write a new post soon and try to formulate various questions as precisely as I can.

If that works, then what I would hope to do is generalize it as much as possible, the aim being to narrow down considerably what a decomposition could look like. That could have two possible benefits: making the computational problem of finding a decomposition much easier, and making the theoretical problem somewhat easier.

I’m interested in revisiting some of the experimental evidence from a few months ago from this point of view and seeing whether we can “dualize” it somehow. By that I mean that we could look at where various searches backtrack and try to build decompositions based on those points and relationships. For instance, perhaps the program could tell us that certain HAP-constraints were important ones and others were not. That would then give us a huge clue about how to find a decomposition.

One snag is that the decompositions tend to be equivalent to vector-valued problems, so it may be that the evidence we have collected so far is not in fact helpful for finding decompositions. And it’s not completely obvious to me how one should search for best possible bounds in the vector-valued case.

A depth-first search (favouring zero values) running in Python has so far reached 49:

http://www.obtext.com/erdos/psol-49.png

It seems to have got rather stuck at 50 (not surprisingly, since it seemed to go a bit mad at 25).

1. Each is a 2-by-2 or 3-by-3 square and each is a fly trap.

2. sums to 1 along the line and to 0 along all lines that pass through a point of the form or

3. For all that do not belong to one of the above rays,

4.

Let me check that that does follow from the existence of a function with the discrepancy property mentioned in the previous comment. I’ll do that by thinking about the sum Let us split this sum up according to rays. The sum along the ray is 1, since is constantly 1 and sums to 1. Along any ray that goes through a point of the form or the sum is zero, since is constant and sums to 0. Along any other ray, the sum is zero, since is identically zero. So the sum is 1.

Now let’s get a contradiction by summing instead over the squares and fly traps used to decompose We have that the sum is at most

But by hypothesis for every and So the sum is at most

which, by hypothesis 4, is less than the desired contradiction. As I say, I haven’t carefully checked, but I basically know that Hahn-Banach will show that this is necessary.

I think the existence of therefore implies that there is no way of obtaining a bound better than if we just use 2-by-2 squares, 3-by-3 squares and fly traps. (I don’t claim that I’ve definitively proved that — there are some important details that need checking, such as whether I can reduce and what happens with intervals that do not start at 1. In fact, that second point may turn out to be particularly important.)

I think I may already have established that this is not the case. If then What can we say about mod if it is an integer, and otherwise? Answer: we know that it is a multiple of or Now if we have two distinct such numbers, they must differ by a lot. Either they will be two distinct multiples of or or they will use different s but will in any case be distinct multiples of for some

So now all one has to do is define as follows. If then If either or is divisible by then If or and we have not already chosen the value of some to be 0, then we set the value of to be -1/2.

This guarantees immediately that the discrepancy on any 2-by-2 square is either 2 or 1 (depending on whether we put in 0 or -1/2 off the diagonal). As for a 3-by-3 square I think the argument above shows that all its off-diagonal entries will be unless one of or is or for some such that is a multiple of Now the only thing that can stop the discrepancy on a 3-by-3 square being at most 2 is if all the off-diagonal entries are 0. Since the numbers and are well-separated, the only way even the entries can both be zero is if or But even then, for the discrepancy to be more than 2, we need to be zero as well. So we need or to be of the form And that can’t happen.

If that argument is correct, then what it shows is this. We can’t find a proof that the discrepancy has to exceed 2 (that is, improve on ) by taking a bunch of 3-by-3 squares and one fly trap and arguing that if the discrepancy is at most 2 on those squares then it forces us to choose values on some fly trap that add up to something large.

That wasn’t very well put, but I think I’ve now thought of a nice way of putting it. I think it is possible to find a function with the following properties.

1. whenever

2. whenever

3. The discrepancy of is at most 2 on all 2-by-2 or 3-by-3 squares.

4. The discrepancy of is bounded on all fly traps.

In other words, any proof that the discrepancy is unbounded on fly traps if it is bounded on small squares would have to use the fact that is constant not just on rays that go through points of the form or but on more general rays, even if we are talking only about 2-by-2 and 3-by-3 squares. I think the bound in 4 could be made pretty small too, but that needs checking.

The question is this. It is easy to create functions that are 1 on the main diagonal and that have discrepancy at most 2.999 on all squares when or However, the obvious ones have the property that they give rise directly to unbounded discrepancy on fly traps. What I want to do is say precisely what I mean by “give rise directly to” and then ask whether what I have observed is necessary, or whether there exist cleverer examples that do not give rise directly to unbounded discrepancy on fly traps.

Here is the definition of “give rise directly to”. I just mean that once you’ve decided on the values of when is 1 or 2, you then put in all other values that follow from the condition You then see if for every there exists a fly trap such that it is impossible to fill in the remaining values on that fly trap so as to keep the discrepancy below To be more precise, for each you fill in all the values that are forced by the constant-on-rays condition. Then you see whether there is an interval of s such that the values are all fixed and add up to more than in modulus.

Let’s suppose that we want to keep the discrepancy on fly traps below something like 1000. Then we have a problem at only if there exists such that the number of factors of that are less than outnumbers the number of non-factors of that are less than by at least 1000. Let’s suppose that and are two such numbers, and that and are factors of and that are less than Then and have at least 1000 factors in common, which … well, ordinarily it might suggest that and differ by the lowest common multiple of some pretty huge number, so that and are not close to each other.

But as I write that, I see that it’s not obviously true, which is good news. So let’s try to think in more detail about how it could be false.

I’ll take an extreme example. First we insist that both and are divisible by 1000!. Sorry, this example isn’t working. Back to my previous line of thought.

Let’s just look at for a bit. Let be some primes that do not divide Then no number that is divisible by any of the can be a factor of It follows (provided is not too big) that the probability that a number less than is a factor of is at most or so. From that it follows that the sum of the reciprocals of the cannot be too large. Assuming that is not too small, that tells us that almost all primes up to are prime factors of

Now I want to know whether it is possible to find some such that there exist and less than such that is one number like that and is another. The difficulty is that and are coprime, but maybe we can deal with that by multiplying them by and to get the extra factors we need. Except that that doesn’t seem very easy: and are much much smaller than so they don’t seem to have enough smoothness to create all the extra divisibility that we need.

Let me try to say that more clearly. Here’s a number-theoretic question I don’t know the answer to. Fix a large positive integer For how big a can we find sets and of integers between 1 and such that and every number in is coprime to every number in ? I suspect that has to be quite a lot smaller than One possibility is to partition the primes and take …

OK, that problem is easy. The best you can do is partition the primes into two and let be all numbers you can make out of one set of primes and be all numbers you can make out of the other. I feel as though that should make the product of the sizes of and be less than but I don’t yet see why I’m saying that. Yes I do. If we make and maximal like that, then every number up to can be written uniquely as a product of something in and something in (Just take the prime factorization and split it up in the way you have to.) But that just gives a lower bound on the product of the sizes of and I’m not sure how to get an upper bound. Actually, it’s false, since just by taking primes we can get and to have size

OK, I can at least prove, but won’t bother with the details, that one or other of and must have size

I need to stop this rambling comment for a bit. But it’s looking quite hard to demonstrate that any major problems would arise if one decided to adjust the values on the very smooth fly traps to be zero, and made any other changes that were implied by that. Of course, there would still be many more changes that had to be made.

I think this gives another perspective on why finding a decomposition using 3-by-3 squares is much harder than finding one using 2-by-2 squares: there is much more flexibility when it comes to devising functions with low discrepancy. For instance, we might try to do it as follows. First we define to be for every Then we do a bit of adjustment: we look at numbers with lots of small factors, where there will now be fly traps with large discrepancy, and we make some adjustments to the values at when either or is of the form for some small factor of That will involve creating some 2-by-2 squares where the discrepancy is now slightly bigger than 2, but we could keep it down to perhaps.

Actually, I’ll stop there because I’ve got confused about what my aim is. Is it to keep the discrepancy down to almost 2, or is it merely to keep it below ? I’m interested in both problems, but they seem fairly different.

Instead, I’ll just make the general point that if the only numbers where we make adjustments are of the form where has many small factors and is one of those small factors, then it’s not clear to me whether we have to make adjustments at big clusters of consecutive numbers. I think this may be at the heart of the problem — perhaps even at the whole of EDP — though I don’t yet have a precise formulation of what it is I’m asking.

I set out on this hoping to find a clever function that would have discrepancy at most some constant less than 3 on all 2-by-2 squares, all 3-by-3 squares and all fly traps. (Of course, I’m always insisting that the function should be 1 on the diagonal and constant on rays.) I thought it was going to be easy because to deal with the squares I cared only about what happens for pairs (r,s) with |r-s| at most 2. And indeed, it is easy to get a non-trivial bound for the squares: for instance, if we define to be 1 if and if or 2, then the sum on all 2-by-2 squares is 3/4 and the sum on all 3-by-3 squares is -3/4. This starts to suggest a bound of 4/3, but we know that can’t be achieved, since the inclusion of 2-by-2 squares means that 2 is the best bound we can hope for. The reason this isn’t a contradiction is that we haven’t dealt with the fly traps.

And that is where things start to get difficult. Once you start putting in lots of values off the diagonal, as we have now done, you commit yourself to many more. From this point of view, the choice of all the way down the diagonals is disastrous for us, since it will force values of everywhere on the fly traps of width at (that is, the ones Alec used). So it will in fact give rise to unbounded discrepancy.

Thus, there is quite a nice problem I don’t yet know how to answer. First and foremost, can one improve on the trivial bound of 3 for this problem? That is, can one find and a function that’s 1 on the main diagonal, constant on rays, and that has discrepancy at most on all squares with and on all fly traps? So far, I can’t even answer the following weaker question: can one get the discrepancy to be at most on the squares and bounded on the fly traps?

http://www.obtext.com/erdos/sol42.png

I suggest saving the image file to your computer and zooming in. Light pixels correspond to large positive values, dark ones to large negative values.

I have added two solutions files data-file-rectangular-area-9 and data-file-rectangular-area-25

The solutions reached at the end of the area 9 run stand out in that, up to signs, they really have only two different coefficients, 1/36 and 1/18, and are still made up of 2×2 squares and rectangles of width 1.

Of course, we know that rectangles of width 1 are not good enough to make efficient decompositions, so this doesn’t come as a huge surprise. It seems to indicate that the assumption is quite a strong one, though I’m not quite sure about that. It would be nice if the version had some one-dimensional consequence: I think I’ll think about that next.

Its a text file with 3 columns, the first two giving the coprime pair and the third the value of the function at that pair. I haven’t included rational approximations to these as some of them seem very spurious. My code doesn’t assume that the function is symmetric in interchanging the two coordinates, hence we have values for both and . It looks like they’re all the same though, which isn’t surprising, but I haven’t checked it.

I’ll modify the code to assume reflective symmetry tomorrow and thus be able to produce some bigger solutions.

In our case, is the function that is, the function defined on positive rationals that’s 1 at 0 and 0 everywhere else. consists of functions obtained by taking a square and counting for each rational how many pairs in the square have ratio equal to that rational. The property is telling us that and the property that for every is telling us that the discrepancy of the function is at most on every square.

My solutions are not sparse at all. For example for , the solution is non-zero at all of the 91 coprime pairs involved. Some of the values are nice rationals, but others don’t appear to be, (although that’s maybe just a feature of how I’m converting to rationals). I can make the code, (which is for Matlab), or the solutions available if people want them but I don’t know how useful they’d be.

Tim

Could you possibly clarify, or reference a comment clarifying, why the problems are identical. I understand your explanation in a previous comment that the existance of a function with discrepancy forces the norm of the coefficients in a decomposition to be , but I can’t see why the bound is atained. Is there a general way we can convert from a function with minimum discrepancy to the decomposition with best possible sum?

Computationally, solving the linear problem for discrepancy and that for decompositions seems fairly similar.

I suppose the next thing to think about is what happens if we allow and zero.

Alastair, could you post one or two of your solutions? It would be interesting to see how sparse the nonzero terms are.

Suppose we are trying to find values along the rays in such a way that the discrepancy on all rectangles is bounded. Then in particular this applies to rectangles of width 1. Now consider the rectangle of width that consists of all points where runs from to Then the sum of the values on that rectangle must be bounded. Next, consider the rectangle that consists of all points where runs from to The values along here must be bounded too. Actually, I should say more: the partial sums as you go up the rectangle are bounded. But these are equal to the partial sums for the even in the first rectangle. More generally, we find that the sums along all HAPs of s in the first rectangle have to be bounded. So if EDP is true then we can’t find a function that’s constant along rays and of bounded discrepancy on all rectangles.

This doesn’t quite prove that we can find a decomposition because we have to allow more general functions. I haven’t yet thought about …

Let’s just see what happens if we define if either or is equal to mod 3, if mod 3, and if mod 3. Is this constant on rays? Yes. Oh dear, it seems to have bounded discrepancy on all rectangles.

Phew, that got me worried, but I’ve just realized that it’s NOT constant along rays. So after that moment of madness I’ll continue the interrupted sentence.

… what happens if you do this.