This is a post I’ve been intending to write for several months, but now seems to be quite a good moment, since the issue is in the air somewhat. For example, I’ve just read a post by Michael Nielsen on a similar topic, which itself was responding to things that other people have written. However, he is addressing a different issue: that of the restriction of access to journal articles once they are published. I am more interested in whether mathematicians really need journal articles at all, now that we have the internet. (Just in case anyone hasn’t noticed, this post is not part of the series about first-year mathematics at Cambridge …)
Before I go any further, let me make clear right from the outset that I’m not merely saying, “We can stick our papers on the internet, so let’s forget about journals.” I think that journals still have a vital role to play, even though the internet exists. However, like many people, I do not think it is at all obvious that they will continue to have a vital role to play, so I’d like to discuss two questions.
1. If we didn’t have journals, then what might we have instead?
2. How could the change from journals to whatever replaces them actually take place?
To answer the first question I’d like to propose a model that is similar to models that others have proposed. I know this not from being familiar with the models themselves, but from having read sentences like, “Several people have suggested models where <description of type of model that I am about to propose>.” If anyone can provide me with links to similar suggestions, I’ll be happy to insert them.
What I think could work is something like a cross between the arXiv, a social networking site, Amazon book reviews, and Mathoverflow. I’ll try to describe what it might look like, and then defend the suggestion against several objections that I think are likely to occur to people. (If you still have objections, I’ll be interested to know what they are. But I hope you will read this sympathetically, and for the sake of balance consider the disadvantages of the current journal system as well.)
In an ideal world, this new website would simply be an extension of the arXiv, but if that couldn’t be organized, then instead it would be a separate site with links to the arXiv. Then if you proved a result, you would post it on the arXiv as you do now, but you would also start a page on this website, which would be devoted to your paper. The page would have various subject classifications — possibly the same ones as on the arXiv.
Once your paper was up, anybody who wanted to would be able to respond to it. Posting a response would be a bit like posting an answer to a question on Mathoverflow, and as with Mathoverflow it would be possible to make comments, either on the paper itself or on people’s responses to it. Also as with Mathoverflow, it would be possible to vote up people’s responses, which would give them reputation points.
What would be in a typical response? It could be anything useful: obvious possibilities would be a description of the main results for the non-expert, an evaluation of the paper, an explanation of what the paper contributes to its area, an assessment of how big that “area” is, a list of minor errors, and so on.
In order to motivate what I shall say next, let me list some potential problems with the idea as I have described it so far.
1. Nobody would have any motivation to contribute to the website.
2. If referees were anonymous, then any “reputation” accrued on the site wouldn’t translate into genuine reputation.
3. But if referees were not anonymous, then they might be unwilling to write about the papers they understood best, for fear of offending people they knew.
4. People might not like to have their papers evaluated in public.
5. Some papers might not get evaluated at all.
6. Some papers by well-established people might be more favourably reviewed than equally good papers by less well-established people.
7. If somebody wrote an unjustly negative evaluation, it would be read by everybody who looked up the paper, thereby unfairly damaging the reputation of the author.
8. If someone gets a paper published in a good journal, that really means something. But if all they got was a few comments on a website, that wouldn’t be the same sort of stamp of quality or certificate of correctness.
Let me deal with these points in turn.
1. Nobody would have any motivation to contribute to the website.
One could say the same of Mathoverflow, and yet many people contribute high-quality answers to that. Why do they do so? I can think of at least three possible reasons: the questions are often quite interesting, people like accruing reputation points (however silly that might seem), and people who have a good answer to a question like the chance to display their erudition in public.
Could the same factors work for a paper-evaluation site? Well, some papers would certainly be interesting and a pleasure to comment on, but what about run-of-the-mill papers? I think even these could be of interest to people in the areas concerned, but I have to admit that for many papers, mere mathematical interest alone will not be a sufficient motivation for commenting on them.
Reputation points on the site would, I think, provide a significant extra motivation, which could apply even to evaluations of uninteresting papers. It would take a display of skill and judgment, not to mention work, to evaluate such papers well, and if this was rewarded with reputation points, one would feel appreciated for doing it. And the third motivation, the chance to show off one’s erudition, would be there too.
How does that compare with the current system? Well, I’d say that the main reason I referee papers is that I find it hard to say no. Sometimes this is because I get a good and interesting paper that relates in some way to my work, so I can see that I am a fairly obvious choice of referee and would feel guilty about refusing. But sometimes I get a paper that is not really about anything that I care about but is officially in my area, or an area that I have at some point worked in, and I accept for some much sillier reason, such as that I forgot about the email and when reminded feel too guilty to refuse.
So as the system is now, refereeing is basically a chore: we know we ought to do some of it, because we want our own papers to be refereed — or the whole system would break down. However, because refereeing is private and anonymous, we get almost no recognition for our efforts, so although there is a fairly strong motivation for agreeing to referee a paper (the wish not to feel guilty), there is very little motivation for doing more than a merely adequate job once one has agreed.
2. If referees were anonymous, then any “reputation” accrued on the site wouldn’t translate into genuine reputation.
3. But if referees were not anonymous, then they might be unwilling to write about the papers they understood best, for fear of offending people they knew.
I have a proposal for dealing with these twin problems. Referees would have the option of writing under their real names, or under a pseudonym, or even a bit of both. When you registered for the site, you would give your details and whatever names and pseudonyms you wanted to use. Then any reputation points you earned would be attached to all those names and pseudonyms. In particular, if anybody looked you up (by name) they could find out how much reputation you had accrued, and therefore get some idea of how much you were contributing to the mathematical community in this way.
An obvious objection is that it would become easy to guess who was writing under a certain pseudonym, because you could simply look at users in the relevant area until you found one with the same reputation. To deal with that, I would suggest not giving the exact number of reputation points next to pseudonyms. One method might be to have very broad categories such as “high”, “medium” and so on — the minimum information that would allow one to form a reasonable judgment about how seriously to take the referee. Another might be to add some randomized errors to the reputation points associated with the pseudonyms, just to disguise their relationship with actual names.
So what I am suggesting would allow reputation points to translate into genuine reputation, even if you wanted to write pseudonymously. I would add that in my opinion it would be better if people wrote under their own names except under quite unusual circumstances. If this was part of the ethos of the site, then people would be less inclined to upvote responses offered by people writing under a pseudonym. But if someone felt that they could say what they thought only under the cloak of pseudonymity, then that could at least be an option.
4. People might not like to have their papers evaluated in public.
This is, I think, a much more complicated issue. Suppose you are a talented young mathematician, just beginning research. Suppose also that you lack confidence — you’re quite pleased with your results but you have no idea how good others will think they are. At present, you can get some private feedback by simply submitting to a journal and waiting for the referee’s report. With the system I’m suggesting, you would have your work dissected, and possibly criticized, in public.
I think this is quite a serious potential problem, but I have a suggestion for mitigating it, which is that moderators on the site should adopt a very strong policy of discouraging negative comments, and this too should be part of the ethos of the site. If you think that a result is actually incorrect, then of course you should politely point out the mistake. (Perhaps in that case, to spare people’s blushes, it could be possible to withdraw a paper from the site, even if it is stuck there on the arXiv.) But if you merely find it completely uninteresting, then you should indicate that in a “positive” way, as is done on MathSciNet when reviewers content themselves with a bald description of the results with no further comments.
In particular, unpleasant negative comments by pseudonymous reviewers would be completely contrary to the spirit of the site. An official policy of that kind would lead to their being downvoted, or even, in extreme cases, removed.
5. Some papers might not get evaluated at all.
What happens if somebody submits a worthy but dull paper in an unfashionable area? I’ve had to referee many such papers and my heart sinks every time, because I just don’t really know, and, worse still, don’t much care, how good they are. And yet people’s careers may depend on such papers being accepted by a reasonable journal. Under the new system, would anybody be prepared to look at them?
I have two suggestions for this. One is that the longer a paper has gone unresponded to, the more reputation one would earn for being ready to respond. This could either be done by increasing the reputation you get for each upvote, or it could be done by having badges for, say, getting a response upvoted five times to a paper that had been unresponded to for over a year. In the first case, the premium could carry on and on increasing, making it less and less likely that a paper would languish unread.
The second suggestion is that it would also be part of the way the site worked that a response to a paper could fall short of a full referee’s report. Indeed, I’ve already more or less said that. One of the difficult aspects of the current system is that one has to do a complete job before saying anything at all. With the online system, one could write a response such as, “Lemma 2.1 is already known: see such-and-such a paper,” or “The proof can be simplified if you use …” That would be useful information for the author and for other readers of the paper, and would take a lot less work than reading through and evaluating the whole thing. And that would make it much more likely that people would be prepared to respond to papers that weren’t earth shattering.
6. Some papers by well-established people might be more favourably reviewed than equally good papers by less well-established people.
Possibly true, but it’s equally true of the current system. Also, if you write a report praising a paper to the skies but not giving a good reason for your praise, then your report won’t (or shouldn’t) get those precious upvotes.
Incidentally, I wouldn’t rule out an iterative system for reputation, where upvotes from people with high reputation count for more than upvotes from people with low reputation. This might guard against “cartels” of people from some tiny subarea all being incredibly positive about each other’s papers.
7. If somebody wrote an unjustly negative evaluation, it would be read by everybody who looked up the paper, thereby unfairly damaging the reputation of the author.
The author would have the right to respond to responses, and if they could explain convincingly why the negative evaluation (which in any case is discouraged) was wrong, then that evaluation would get plenty of downvotes and the author’s reputation would remain intact.
8. If someone gets a paper published in a good journal, that really means something. But if all they got was a few comments on a website, that wouldn’t be the same sort of stamp of quality or certificate of correctness.
Let’s think for a moment what getting published in a journal gives us now. To start with correctness, opinions differ about whether a referee is obliged to check carefully what an author writes. But the very fact that opinions differ is enough to tell us that the mere fact that a paper is accepted in a journal is not, on its own, a guarantee of correctness. At best it makes it a safer bet that the paper is correct. Most of the time, what makes one feel that an important result is “truly established” is, these days, not the fact that it eventually appears in a good journal, but the fact that it is read and understood by people in the field, who are likely to be able to sense if there is something fishy about the proof. If the result is sufficiently interesting, then that tends to happen long before the paper comes out.
With an online system of the kind I’m talking about, one could have more transparency about this process. Someone could write a response saying, “I haven’t checked this line by line, but the paper is based on a very nice idea, which I basically understand, and it’s quite clear that the details can be made to work.” That’s how I feel, for example, about some of Tom Sanders’s recent breakthroughs. Somebody else might go further and actually explain the idea. And someone else might say, “I’ve checked the first section carefully and apart from this minor mistake, which can easily be fixed, it’s correct.” And so on. (One would of course trust that kind of comment more if the person making it had plenty of reputation points. And if they vouched for the correctness of results that turned out to be wrong, then they would lose points.)
And how about the stamp of quality? Nowadays, if a paper is published in Annals or Inventiones, we think it must be pretty good, whereas if it is published in [insert name of your favourite not very good journal here], then we don’t. But is that all we do? It certainly gives us some indication of the quality of a paper, but it is a very imperfect measure. Nobody would say that every paper that appears in Annals is better than every paper that appears in, say, the Journal of the LMS, even if it is supposedly a better journal. So at best the stamp of quality is a rather crude measure.
Why do we need this measure? One of the main reasons is that we find ourselves having to judge other mathematicians, notably when hiring them. We get their CVs and publication lists, and we have a look through the latter to see what kinds of journals they are getting their papers published in. But if they are lucky and get an indifferent paper accepted by a very good journal, it will be very difficult for anyone but an expert to know this — and the unfortunate fact is that we often have to judge people who work in areas we don’t know much about. We also have only a rather vague idea of the relative quality of many journals. The one document that could really help us, the referee’s report, is strictly private.
Under the system I’m describing, we would have something potentially far more useful: not just the information of where in some very fuzzy quality scale the paper appears, but actual descriptions of what the paper has accomplished, how it fits into the general aims of the area, whether the techniques are standard, what is truly new, whether the paper is exciting and unexpected, and so on. If you were contemplating hiring somebody, you’d have a ready made reference, written by a wide variety of people, before you even began.
I wondered whether it would be a good idea to have scores for papers. One could for instance vouch for its probable correctness by ticking a box, and perhaps give it marks on a scale of 1 to 5 for quality. (There would be guidelines: for instance, 5 might be, “a breakthrough result worthy of one of the top two or three journals” and 1 might be, “of interest to specialists in a tiny area only”.) Perhaps one could allow a digit after the decimal point too. There would be some pressure to give the right judgments, since if you didn’t then you’d risk getting downvoted. And the paper could get some kind of average score based not just on how many people gave it what score, but also on their reputations and the votes given to their responses.
I think something like that could be made to work, but I also wonder whether a more radical approach is simply to do away with this linear-scale idea (which I think is implicit in the current system) and satisfy ourselves with the opinions given in the responses. It would be a bit like the difference between an innocent/guilty type verdict and a narrative verdict. Instead of reducing the paper to a number, you’d have judgments and descriptions that would tell you all about the paper and give you a much more detailed idea of how good it was.
I’m torn on this point. I’d prefer not to have numbers, but it might in the end be necessary for the purposes of pleasing bureaucrats in other subjects who need what in this country are called “metrics”. (Roughly translated, that means numerical measures of quality.)
After that discussion, let me collect together what I see as the main features of this hypothetical paper-evaluation site.
(1) You post your papers on the arXiv and create pages for them on the site.
(2) People can respond to your papers. Responses can range from smallish comments to detailed descriptions and evaluations (the latter being quite similar to referees’ reports as they are now).
(3) Responses can be written under your own name or under a pseudonym.
(4) You can accrue reputation as a result of responses of either kind, but your pseudonym will have the reputation disguised enough to maintain your anonymity.
(5) Negative language is strongly discouraged. If a paper is uninteresting, it simply doesn’t attract much interest. If it is incorrect, one says so politely.
(6) There is a reputation premium for evaluating papers that have spent a long time not evaluated. (There would be a way of finding these: for instance, you could list all the unreviewed papers in a certain area or subarea in chronological order.)
(7) If you are not registered for the site, or if you are registered but had very few reputation points, then people know that you are not doing your bit for the mathematical community when it comes to the important task of evaluating the output of others. Conversely, if you have a high reputation, then people know that you are pulling your weight.
I can think of a couple more potential problems. One is that cranks could put their papers on this site and perhaps gain some kind of bogus respectability — more so than they can at the moment when their papers don’t get published. I’m not too worried about that, however. In fact, I think this system would be worse for cranks than what we have at the moment, where they can put their papers on the arXiv. Under this system, they could put their papers on the site, but those papers would be followed by responses like, “I’ve checked, and as expected the supposed simple proof of Fermat’s Last Theorem is wrong. In particular, the statement of Lemma 4 is false.”
Another is that somebody might put a paper on the site, then register under a different name and give their own paper a very positive response. I think it ought to be possible to stop that by having some kind of barrier to registration as a reviewer. If any professional mathematician were to try it and get caught, then their (real life) reputation would take a huge knock, as people who have been caught praising their own books on Amazon have discovered.
Now let me turn to the question of how one might actually convert from the current system to an online system such as the above. (By the way, I don’t insist on every last detail of the above — I’m sure it has its flaws, and I’m sure there are some good ideas that I haven’t thought of for features that the site could have.) My answer to this is simple. Somebody gets the website up and running, and people start submitting their papers to it, as well as submitting them for publication in the normal way. Then if the new method works in the way I think it should, it will gradually be realized that the extra information one gets from the fact that a paper has been accepted in a journal is close to zero, and the journal system will wither away. If it doesn’t work, then it’s the site that withers away: it goes down in history as yet another online initiative that failed to take off.
In theory, it would be possible to use the site to form something like “virtual journals”. If you wanted to set up a virtual journal in additive combinatorics, say, then people could ask you to give an official stamp of approval to their paper. The editor would find a referee and instruct them to write a report. If the editor felt that the report was strong enough, then they would post the report under the journal’s name, stating clearly that it had been accepted as up to the standards required of the Journal of Additive Combinatorics. These “journals” could then accrue reputation and be useful collections of papers.
Having said that, the current business model for journals would no longer work, as it would no longer be possible to make money from publishing mathematical articles. While many people would rejoice at the thought of Elsevier not making money out of their papers, there would also be less welcome side-effects. For example, the London Mathematical Society makes a lot of money from its journals, to the benefit of British mathematics. I should probably mark that as another disadvantage of an online system, but if such a system would be better in all other respects, then I think a more sensible response would be to try to come up with new ways of funding learned societies. For example, the money that universities currently spend on keeping their libraries stocked could be given in part to those societies instead.
One final difficulty is that the site would probably be complicated and big enough to need active maintenance, so it would cost money to run. But it could in principle do such a great service to the mathematical community that obtaining the funding should be possible if a proper proposal, perhaps including a beta version of the site itself, could be presented.
Postscript: I’ve just Googled “The future of mathematical publishing” and come up with some interesting links. Here is what Rob Kirby had to say in 1998. Last February there was a conference at MSRI on mathematical publishing, which produced this interesting document. (One thing that interests me about the document is that the prevailing attitude seems to be one of thinking about how the mathematics journal will survive in the new and changed world we now inhabit, rather than thinking about what a system for evaluating research would look like if one were to design it from scratch.) And here’s one more link: a paper called Some thoughts on the near-future digital mathematical library, by Thierry Bouche.
Also, I strongly recommend this post by Michael Nielsen, which has influenced my thinking on these matters.
I forgot to mention one other aspect of an online system that would in my opinion be a huge advantage. It’s that there are many kinds of mathematical communication that do not get published in journals because they are “the wrong kind of thing”. For example, if you write an essay about why certain promising approaches to a problem don’t in fact work, then unless you can express it precisely in the form of a mathematical theorem (a famous example of this is Razborov and Rudich’s Natural Proofs paper), then you won’t get it published. Similarly, if you rewrite an existing proof in a much clearer way, you won’t get that published either, even if what you’ve done is far more useful to other mathematicians than yet another obscure theorem. On this site, you could submit mathematical documents of all kinds, perhaps giving them labels such as “new result” or “exposition of existing result” or “discussion of open problem”. If these were good enough, you would get glowing responses to them. Perhaps this could lead to a cultural shift towards giving more value to mathematical activities other than proving theorems.
Gowers's Weblog 
October 31, 2011 at 7:36 pm |
Tim, you asked for links to similar suggestions. I have a few links you might find interesting, but aren’t necessarily too similar.
On a post at scienceblogs.com [1], Chad Orzel and company propose a “journal of stuff I like”. Perhaps the quickest summary is that each person has their own “personal journal” of links to the arXiv where they leave comments on papers. This might be viewed as your proposal except without an explicit reputation system.
(This is also somewhat similar to the idea of “fantasy journals” [2].)
I believe you will also find the idea of an auction market for journal articles intriguing [3]. Roughly speaking, journals bid on articles from a central repository like the arXiv, using fake money called “academic dollars”.
My source for all of these links is two of Michael Lugo’s blog posts [4, 5].
[1] http://scienceblogs.com/principles/2008/09/science21_the_journal_of_stuff.php
[2] http://marginalrevolution.com/marginalrevolution/2007/08/fantasy-journal.html
[3] http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1005304
[4] http://godplaysdice.blogspot.com/2008/09/journal-of-stuff-i-like.html
[5] http://godplaysdice.blogspot.com/2007/08/some-thoughts-on-current-journal-system.html
October 31, 2011 at 7:38 pm |
Interesting proposal. Yes, various aspects of it have been proposed before – a discussion on the algebraic topology mailing list a couple of years ago led to me setting up a forum for discussing how one might go about creating a review site (http://www.math.ntnu.no/~stacey/Vanilla/rForum/, in case anyone’s interested). It fizzled out.
The main difficulty that I see with any such system is your first point: motivation to contribute. There’s a big difference between answering questions and reviewing papers, and an even bigger difference between asking questions and putting up your paper for review.
I have my own personal “review” site where I keep notes on papers that I read, and I put them in public. But even there, I rarely actually put the notes up – most entries are just bibliographic information to make it easier for me to find the papers again. I know what I think of the papers, and most of the time I don’t feel the need to record more information about them.
That’s why the project proposed in the rForum stalled. I couldn’t think of a way to make such a project useful **at the point of use**. Things like the nLab and MathOverflow and TeX-SX work because they are useful right at the time you use them. Even when answering a question, I feel that I’m gaining something, or the effort required is so minimal. I don’t think I would feel the same when writing a review. I might feel that I *ought* to do it, but there are so many things that I feel that I *ought* to do, that I doubt this one would make it on to the list.
Perhaps if the connection between writing reviews and getting your papers reviewed was stronger. Maybe you can only get one of your papers reviewed when you’ve written enough (up-voted) reviews yourself.
I completely agree that the system needs a complete overhaul, and it’s great to read your views on this.
October 31, 2011 at 7:46 pm |
I certainly agree with the last paragraphs about finding ways to better recognize other mathematical activities than proving new theorems. But this is somewhat independent of the whole proposal, and one could imagine creating a serious online journal of “non-standard” mathematical writing with much less effort.
October 31, 2011 at 7:54 pm |
To handle point 4, one could institute a policy of optionally revealing the paper to only a select group of people (maybe high-rep people in the relevant area) before opening it up for general comment.
October 31, 2011 at 7:55 pm |
I suggest that an alternative realm be considered. Instead of a new model of publishing, how about a new model of research or of exposition? The notion of “paper”, while at present useful, may hinder scientific and other forms of progress; we might do well to suggest an alternative measure of research output. If your goal is to preserve some facets of the old system but ramp it up slightly, then we could discuss your article point by point; I think some of the items you raise belong to a more radical shift: how should a national or global society advance mathematical research, and how should individuals contributing to that have their efforts measured and rewarded? I think you have the boldness to expand the scope of your article.
October 31, 2011 at 10:11 pm
I suppose one alternative to the model of free-standing papers would be a giant wiki, full of cutting-edge results. People could contribute little points, or sections in articles, or whole new articles.
One could do this for other disciplines, too. For mathematics, we could call it Wourbiki.
October 31, 2011 at 8:05 pm |
Here are three other objections to such a website.
1. It neglects the archival function of journals. I feel very confident that the arXiv will be here in 10 years, but I don’t feel at all confident that it will be here in 100 years, and I rather doubt that it will be here in 200 years. Certainly something will be there, but who know what form it will take. But printed books will survive, even if our civilization does not.
2. It would benefit certain areas of math in a way that is not related to the importance of those areas. For various non-mathematical reasons, certain areas of math are very active on the internet, while other areas are not.
3. It would do mathematics departments a serious disservice within the wider academic community. It is already very hard to advocate for greater resources for math departments given that we don’t write huge numbers of papers, we don’t get large grants, and we don’t get citations in the same volume as other other sciences. Imagine having meetings with deans and advocating for extra salary for someone whose only accomplishment is publishing something on a website somewhere!
November 1, 2011 at 11:43 am
1. To begin with, the problem with a site such as arXiv is that libraries cannot buy the content of its hard disc.
2. You can have an area of math not active on the internet for a simple mathematical reason: it may be under competitive development of experimental nature, not yet ready for prime time.
3. The problem with the current system is the pressure to write ever more papers, resulting in referees being brainwashed away from doing their job properly. An interesting trend away from writing a lot of papers making little progress is the posing of open problems in many recent works.
A simple solution at this time which could begin to solve other ills might be for maths departments to disallow its employees from publishing too many papers. Here sites such as arXiv could satisfy the needs of serial writers.
November 2, 2011 at 11:54 am
It seems it is now possible to buy the arXiv content in bulk – see http://arxiv.org/help/bulk_data_s3 – though it’s not clear from that page if it includes all versions of all papers, the underlying software may not be available and the data doesn’t come with any redistribution rights. A more conventional approach for distributing large amounts of data without the bandwidth to deliver it directly to everyone interested would be to use BitTorrent, although some networks may frown on that because of widespread use for copyright infringement and it only works legally if the recipients may redistribute the data themselves.
The same point about getting all the data for preservation would apply to the proposed new site: all comments and associated metadata should be available for bulk download and redistribution (which conflicts with having private data linking pseudonyms with other identities), as should details of server configuration, and the underlying software should be open source.
(But thinking of it as a single site at all is actually quite a conservative approach; a distributed approach might be better where you can have any number of interoperating sites sharing comments and distributing whatever subsets they wish, with whatever associated metrics they wish. Several proposals at https://drumbeat.org/en-US/challenges/beyond-comment-threads/ relating to commenting on news articles actually propose generic distributed commenting systems for any content with a URL, and if such a system takes off you’d then want to define appropriate tags and other metadata to say that comments attached to an arXiv URL are intended as part of the academic review process on that content – not to have yet another separate commenting system.)
October 31, 2011 at 8:27 pm |
Yes, Reddit for papers! I’ve often thought this would be a good idea. In fact, one could get started rather easily, though without all the desired features:
arxivpublishing.reddit.com
November 3, 2011 at 4:34 pm
A “Reddit for papers” has existed for long time. It was Dave Bacon’s scirate.com. It was connected to arXiv and it was very successful in the quantum community which Dave belongs to. Unfortunately since a couple of months scirate is down. I would be interesting to know more about its future.
October 31, 2011 at 8:27 pm |
Very interesting post, Tim. I have a couple of quick thoughts.
First, another way around problems 2 and 3 would be for each member to have (as you suggest) the option to respond anonymously, but not as a pseudonym, simply as ‘Anon’. Any up- or down-votes for such responses would count towards that member. (Only the server would have knowledge of which member made each anonymous response.)
The only disadvantage I can see is that it wouldn’t be possible to know the reputation of the author of an anonymous comment (for example, to inform how seriously you should take the comment). But that could be a worthwhile incentive to post under a real name; and moderators could still take action against the authors of destructive anonymous comments.
Secondly, I’m not sure that the question of how ‘good’ or ‘important’ a paper is need be addressed at all by the site. The fact that some journals are considered better or more important than others is, as you point out, of questionable relevance, and maybe even a distraction from (or excuse to avoid) more objective evaluation of the work they contain. Provided the site is well indexed and searchable, it is likely that important papers would be noticed soon enough and acknowledged as such, in other ways, by the community.
Clearly, though, some gauge of correctness is needed. Your idea of a tick box seems plausible, but it may need to be a little more nuanced: for example, it should be possible to distinguish between an argument that has some flaws that may be mendable and one that is fundamentally broken. (Could something like ‘bugzilla’, as used for tracking software bugs, be employed here?)
October 31, 2011 at 8:32 pm |
On evaluation (or why metrics are necessary not just for non-mathematician bureaucrats):
Nowadays many jobs get 500-1000 applicants. Someone on a search committee has their own teaching and research to do. The search committee member cannot spend 5 minutes per application at the initial phase.
To make the search committee’s job possible, there has to be some way of making a preliminary evaluation of an application in 1 minute or less which throws out about 90% of the applications. This preliminary evaluation does not need to be very accurate, since any applications that pass will get a more detailed look, and, from the hiring institution’s point of view, tossing out a good application in error isn’t much of a problem as long as it doesn’t happen too frequently.
Reading a narrative takes more than 1 minute.
October 31, 2011 at 8:39 pm |
One thing that worries me about proposals like this is the further “gamification” of life. The fact that mathoverflow works like a game, with points and badges, doesn’t really bother me, since the mechanics work reasonably well, and it genuinely is a kind of game that nobody takes all that seriously. However, the game aspects are already enough to put some people off, and I would strongly dislike them too if I thought they really mattered in the world (for example, if employers started judging people by their MO reputations). Basically, I don’t want to participate in a professional community with scores and rules for earning points, unless it’s clear that it’s all for fun or just a convenient way to sort the results.
Partly, this is because games and rules represent exactly what I think mathematics shouldn’t be: a competition, and especially a formalized, rule-based competition. Of course there are plenty of competitive aspects of mathematics, and some degree of competition is unavoidable and even healthy. However, I don’t want to institutionalize or promote it any more than is necessary, and I want to leave open multiple, incomparable, unquantifiable ways of competing. It would be a nightmare if someday, the first thing a search committee does is to rank the applicants according to how many research quality points they have received in the official mathematics forum, weighted according to the reviewing reputation scores of the people awarding the points.
I also think it would be nearly impossible to keep people from manipulating the system, or even to define what counts as manipulation. For example, if you ask the students in your graduate class to go comment on the papers you are covering, some of which are yours, is this a laudable attempt to get your students involved in professional life, or a lamentable attempt to make these papers look popular and well loved? If a forum like this became an important part of professional life, I believe there would be a lot of bitterness over how it was being used or abused, as well as periodic, controversial attempts to clarify or update the rules.
November 1, 2011 at 2:35 am
I had the same thought. It seems that the reason that mathoverflow.net and cstheory.stackexchange.com work so well is precisely because the reputation points don’t matter at all in “real life.”
November 3, 2011 at 7:17 pm
I used to be in favor of the idea of the site that has been described but this comment made me think: Research shall not be a competition but should be done for the results themselves. Putting a paper on a website just to earn reputation seems not a good motivation, but as soon as reputation has some meaning “in real life” this will happen almost certainly.
November 9, 2011 at 12:10 am
“and I want to leave open multiple, incomparable, unquantifiable ways of competing”
This is a very important issue I think – for a variety of reasons, including the point that particular results and whole areas that are not appreciated by the community today may be seen as cool and fashionable in a decade; and vice versa. Indeed diversity of thought is crucial, and Tim’s current proposed system might hit it badly.
But at least “multiple, incomparable” ways of competing can be easily achieved within a modified system, just like the impact factor is being challenged by various new metrics. There are lots of quantitative characteristics that a single paper in such a system could have: correctness and “quality” evaluations by reviewers, as functions on their reputation, on their activity in commenting the paper, on the upvotes on their comments on the paper (in turn depending on various characteristics of the upvoters), etc.; then there are the numbers of downloads and views of the abstract, as functions on various characteristics of the viewers/downloaders; all of these numbers may change for a new version of the paper, and all of these numbers can be made area-specific (for entire math, for an arxiv category, for a MathOverflow tag, etc); and so forth.
This zillion of characteristics of a paper can be combined in a zillion of ways into a single number, or just a few numbers such as likelihood of correctness and a quality rating, which can then be blended with more traditional ratings based on citations. Then if somebody (e.g. from a hiring committee) wishes to translate these data into an overall evaluation of the author, there are again many potential ways of doing it: short and long papers could have same or different weight, there could be various formulas like “five best papers minus three worst ones”, etc. Whichever way you calculate the author’s overall score, it can count as an argument for his reviews, his upvotes and his downloads, with the desired amount of circularity.
So there can be several competing ways to evaluate a job applicant based on the data, which could be built into a system with a possibility of customization. If you’re on a hiring committee or you’re from NSF or from university administration, you could select out of 5 popular metrics, say, or you could devise your own metric in minutes. As long as different metrics are used by different people and by different organizations, this is not going to be any worse than the current model where pretty much everybody has to compete for publication in high-ranked journals (ranked according to the impact factor!) and for being cited. In fact, I wouldn’t call this current model open to “multiple, incomparable” ways of competing.
On “gamification” and “Putting a paper on a website just to earn reputation” I cannot quite agree. Surely when you submit you paper for conventional publication in the Annals or wherever you submit it, it is a kind of a game already? Surely communication with your anonymous referee is an addictive game, is it not? And sometimes people publish papers just to able to apply for jobs and grants, don’t they? How open evaluation process is going to make it worse?
October 31, 2011 at 8:47 pm |
I am familiar with a few similar ideas. Noam Nisan advocated similar ideas for some time and he has some related posts on his blog. Naj economics (a peer reviewed but “not a journal”) is a sort of somewhat similar system http://www.najecon.org/naj/najfaq.htm .
Of course, it is important that people are reading and trying to understand papers of others, and if they have comments they find ways to present them to the authors or to others. This is a valuable thing to do. And, also, understanding an important new paper/idea/concept/technique is one of the best avenues for successful resaerch on your own.
My instinctive reaction to some of these ideas and to the rather detailed suggestion by Tim is quite negative. I think that having such a system will be overall bad (to mathematics, and to mathematicians) and will have bad side effects (to specific groups of mathematicians). On the other hand I think it is unrealistic to think that such a system can be viable so it will simply “not fly”. (The second feature make it less bad that it is bad). Some of the concerns were raised in the post itself (with long but unconvincing responds) and there are other concerns as well.
Maybe some small fragments of the overall suggestion can still be nice.
October 31, 2011 at 8:50 pm |
P.S. Regarding Alexander Woo’s comment: I agree that unfortunately, search committees need a quick way to filter out many applications. However, I’m worried about a scenario in which they all use the same couple of numbers, perhaps with different thresholds. Even aside from individual unfairness, the system would be sure to award more points to certain working/publication styles than others. (For example, it might prefer one big paper to several small ones, or vice versa, but it would surely end up having some effective opinion on the matter.) These distinctions might be quite important in practice, in ways that were not anticipated by the system’s designers.
October 31, 2011 at 9:12 pm |
Nowadays arXiv has a system of endorsement. My first article was a few days ago accepted for publication in a peer reviewed journal, but I can’t get into arXiv because nobody endorses me. So to get into arXiv is harder that to be published!
That say, if we would do it now and base our system on arXiv I would be a looser (and the entire math research system would loose me and my discoveries).
BTW, please endorse me for arXiv:
http://arxiv.org/auth/endorse.php?x=8KIXKG
October 31, 2011 at 9:15 pm |
There already exists a site where anybody can put his scientific article and be commented and reviewed. They call them “a peer reviewed publisher”.
This site is a partial implementation of the idea of Gowers.
There are very few math articles there however.
Sadly I don’t remember the URL of the site.
October 31, 2011 at 10:05 pm |
One tweak that might help with problem 6 would be to have all papers compulsorily published without the authors’ names on them, for the first three months. Then the authors’ names would automatically be disclosed. So the initial comments on a paper would not reflect people’s opinions of the author’s professional standing (except when people could work out who the author was), and it would be a bit obvious if the tone of comments changed once the author’s name was displayed.
November 1, 2011 at 3:53 am
I read papers mostly for one reason: because I am interested in what they say. Without knowing who the author is (and trust me, I could work it out in most cases I care about), I am much LESS interested. I very rarely read papers merely to “evaluate” them. And yes, I agree to referee 10+ papers per year.
October 31, 2011 at 10:06 pm |
This is a great idea, that is, if you want to turn math into slashdot. I would rather keep the old system than have to keep up with thousands of remarks from you and your attention whores.
October 31, 2011 at 10:11 pm |
Tim,
Great post.
Let us begin by saying that the overall topic of choice for this post is incredibly relevant to the current project we are working on. And for starters, you could not be more right about the need for an alternative community for scholars to participate in rather than just academic journals/publishing. While we agree that journals will always have their place in the academic community, we are very interested in creating a blended medium that affords academics the very opportunities which you speak of; a forum to collaborate and comment on the work of their peers in a carefully constructed and highly social arena.
Similarly to what you have suggested, our site aims to provide “reputation” points for those who become very active in the community and give insights and responses that intrigue others. We are also considering measures to make members’ recommendations and reviews of others work more visible to encourage honest and thoughtful participation. We would love for you to visit our site and provide us with feedback. We are working daily to develop new features and are looking to the community for direction.
Come and take a look at http://www.scholasticahq.com. If you like what you see, request a beta, and we’ll send you a code.
Thanks again for the fantastic post and intriguing ideas. Keep up the good work!
Best,
Team Scholastica
October 31, 2011 at 10:11 pm |
Hey Tim. You say you can think of at least three reasons why someone might want to contribute an answer to MathOverflow. I’m not sure that your reasons encompass why I occasionally contribute answers though so I thought I’d offer my reasons here, as perhaps a 4th reason. Nowadays I definitely do _not_ answer a question just because I think I know the answer. Indeed, if there’s a question where I’m pretty sure that any half-decent number theorist can answer, I almost always do not answer. But I am finding now that the reason I’m answering questions is that people occasionally ask technical questions in an area that I know something about and which is not particularly well-documented in standard graduate textbooks — e.g. a question about p-adic modular forms, for example. And now my instinct to be a good teacher and provide some sort of service kicks in — if I can answer quickly and effectively, and I also know that typically a grad student would not know the answer, then I am motivated to answer because it “makes the world a better place” in some sense! I think I sketch my current policy on my “MO home page” or whatever it’s called.
Cheers,
Kevin
November 2, 2011 at 9:22 pm
I was going to say the same, and add that I have many times referred people in face-to-face conversations to one of my MO answers, just to have _someplace_ to send them to a written version.
October 31, 2011 at 10:55 pm |
Thanks for the thought-provoking post. You might be interested in this working paper on crowd sourcing the process of scientific publishing:
http://luca.dealfaro.org/Home/openpublishing.pdf?attredirects=0
November 1, 2011 at 12:16 am |
It’s a recent trend online to do the following:
1) Identify some “old-fashioned” process (say identifying interesting news stories)
2) Invent some sort of game in which users are rewarded by others for their ability to play. The game is built so as to reward behavior that corresponds to doing our process well.
These attempts almost always fall victim to some sort of degenerate behavior where users manipulate the system. Users will find some behavior which is rewarded out of proportion to its actual contribution. Take for example any of the social news sites.
While I’d like to think that the mathematical community could resist falling victim to this same sort of degeneracy I think there will be great temptation, especially if real-life meaning starts to be assigned to scores in this game. If your potential to get a job rides on how many points you have in this game, that’s very strong motivation to game the system as best you can.
Furthermore, even were it possible to prevent people from abusing the system there is the very real question of how best to translate from real contributions to the mathematical community to some sort of point scheme. There are many (probably conflicting) potential ways to rate the relative merit of different contributions.
—————
This being said, there is certainly a lot to be gained from some sort of electronic update to the peer review process. I propose limiting the information that a given user can see about other users/posts to the following:
1) He can see whether a given post/comment has positive/negative/neutral votes, and can see their relative positions (i.e. sort papers/comments by score).
2) Each user is assigned one of a few colors according to his reputation say Red=Negative Yellow=Few but positive Green= many. This is probably not enough, but the point is to give users a way seeing at a glance how seriously to take a comment, but not to incentivize gaining points simply for its own sake.
Perhaps you might give users the ability to see their own reputation against some sort of statistics about the population of other users, but this would be private, decreasing the incentive to get reputation for its own sake, and removing the ability for 3rd parties to form detailed judgements about users from some sort of score.
November 1, 2011 at 12:34 am |
One element of the current publication system that is worth preserving is a publication threshold. A publication threshold means that there is a period in which papers get improved and reviewed by a small number of people and a period afterward in which they remain static are read by a larger audience. The most useful aspect of this is that it delineates a boundary between having authors revise work so that it is a better contribution versus having a static copy that others can refer to and work from.
November 1, 2011 at 1:47 am
I strongly agree with this comment. In the best case, the publication process saves labor for the community as a whole. Acceptance by a journal means that a reader can reasonably assume that a paper has met a certain standard, and that any obvious problems have been corrected – this without having to read and digest a possibly lengthy critique, which may in fact be no longer relevant if the paper has undergone substantial revision in response to the critique.
On the other hand, public reviews mean that hard-working referees can receive credit if they have made substantial contributions to the final paper, which is not possible in the current system.
November 1, 2011 at 1:48 am |
Another valuable aspect of the current system is that, assuming there are multiple referees, the reviews are independent. In a system where reviewing is done in the public eye, it will be difficult to prevent earlier reviews from influencing later ones.
November 1, 2011 at 1:51 am |
There are many interesting suggestions above!
I think the important thing is to think about _incremental solutions_, that are relevant in the context of the current setup, and that can offer immediate benefits. It’s fine to have in mind (possibly disparate) ideals about later stages, but essential that it’s actually possible to _take the first step_.
Two ideas that I like a lot, and which are compatible with (but don’t necessarily require) making the refereeing system more flexible, public and effective, are
1) establishing “selected papers” lists, orthogonal to the journal system, and
2) running explicitly multi-tiered journals.
An editorial board could establish itself, announcing that it will select a certain number of papers (presumably from a particular subfield, perhaps on particular criteria) for their “selected papers” list during each specified period.
There would be no expectation that these papers were or were not published or submitted at any journal. The list could be a simple list, or it could also include some amount of commentary on each paper, presumably solicited by the editorial board from appropriate mathematicians. These comments might or might not certify the “correctness” or “relevance” of the papers, according to the accepted criteria for the list.
An author whose paper was selected for a list would have the option of including this information on their list of publications (for example on grant proposal, job application, or website). This information could be additional to the “standard” information about a journal in which the paper has been accepted.
Such a list could be started with very little overhead. The technological infrastructure required is barely more than a blog.
Initial instances of such “selected papers” lists should presumably come with a sufficiently prestigious editorial board (present company definitely not excluded, eh?) so as to give the idea an initial chance of receiving the necessary attention.
November 1, 2011 at 2:07 am
I really don’t like the idea of “selected papers” lists. As things stand right now, I can submit my paper anywhere I like. I might not get a fair shake, but in my experience most editors make a valiant effort to find an appropriate referee and (not always, but mostly) trust their advice. A list of “selected papers” will just end up being a list of papers by the editor’s friends in their favorite fields.
November 1, 2011 at 2:03 am |
I have been speaking informally about such a system, and I may have mentioned it at the n-category cafe at some point. The gaming aspects of the system and the potential abuses exist within the profession as it is.
In terms of publishing, I think that journals should add value to the product if they wish to publish. In particular, better papers (whatever better means) can be bundled into a journal, and the author receives a request from a journal to publish the work. That aspect alone puts the author in a better place.
Non-anonymous refereeing also makes sure that one is careful with one’s comments.
Finally, it has to be made clear to the bosses that this is how we do business. Axillary measures such as the number of times the article has been downloaded or cited can be added to the reports to the Deans. As long as these are accurate, then they become more meaningful than the existing system.
November 1, 2011 at 2:08 am |
It seems to me that this proposal solves a problem that doesn’t exist, and doesn’t solve the problem that it purports to solve.
There already exists a robust system for obtaining feedback on one’s papers and giving feedback on the papers of others. Our email addresses are at the end of every paper we write; we visit our colleagues to give seminars and discuss our work; we meet regularly at conferences; we have Mathematical Reviews and Zentralblatt. In fact the number of people who are qualified to give insightful feedback on my papers is not so large, and I think I know most of them personally (typical exceptions being recent PhDs and a few people that don’t travel much). If one of these people has something nontrivial that they want to say about one of my papers, I would expect them to email me — and to be honest, I don’t see that changing even if this proposal were to come into effect.
On the other hand there is a certain kind of feedback that I’m very happy to receive (or give), but am loathe to receive (or give) in public, namely the kind of nit-picky commentary that significantly improves the exposition of a paper. The kind of comment that makes you feel like a *pest* if you give it voluntarily, but that you (hopefully) feel responsible for giving anyway if you’re a referee. You know what I mean: “one more sentence of justification here would be helpful”, “when you define X it might be useful for the reader to include example Y as motivation”, “please change lambda in this section to something else because you’ve already got a different lambda two sections ago”, “you impose hypothesis H in section 2 but it seems to me that you don’t really use it until section 3, please modify section 2 to make this clearer for the reader”, “it would look better to tex such-and-such an expression using this command instead of the way you’re currently doing it”, and so forth.
That is: when the anonymous peer review system works, it really works. I’ve gotten some fantastic referee reports that have made genuine improvements to the readability of my papers, and in the other direction I believe I’ve written some referee reports that have made genuine improvements in the papers of others. (Of course it doesn’t happen every time that I receive (or write!) a report.) I would hate to lose this kind of feedback, but I don’t see it happening within the system that is proposed here. It takes a serious commitment of time and energy from an individual, and I think this system would ultimately allow many of the most competent, qualified referees to avoid taking on that kind of responsibility (since when it comes down to it they would rather spend their time doing their own mathematics).
In short, if there’s to be a solution to the journal problem, I’m of the opinion that we need to look for solutions that nevertheless preserve the anonymous peer review system. (That’s even before considering the question of what we should do in order to give official imprimateur to our work, for evaluation by hiring committees, tenure committees, deans, funding agencies etc. +1 to both Henry and Gil.)
November 1, 2011 at 2:30 am |
As long as papers are the only form of credit-granting work, I don’t see the journal changing form significantly. As I see it, the fundamental flaw with your proposal and ones similar to it is that there are too many papers that no one (or almost no one) actually wants to read. Any workable system along the lines of what you propose seems to me to leave many (and perhaps most) papers uncommented on and unrefereed. You can try to fight this problem, but I don’t think the workarounds are going to work better than the current system.
On the other hand, the paper is a somewhat unusual form for mathematical knowledge. Internally the structure of mathematical knowledge is more like a wiki than like a paper. Furthermore, as the number of papers grows rapidly it will become harder and harder to actually find relevant information in papers. I expect to see projects like the stacks project and the nlab grow. A lot of stuff that’s currently written in papers would be much better as a little corner of a big wiki project (that would be refereed by the other people on the project). If large wiki-style projects were the normal mode for communicating “normal mathematics” and papers were reserved for exposition, “revolutionary mathematics,” and other things that you expect lots of people to actually read, then it would be no problem to have a system similar to what you propose.
November 1, 2011 at 2:34 am |
A lot of mathematicians have a lot invested in the “seriousness” of what they do. Journals have a certain gravitas, and I think any suitable replacement has to try to maintain some of that serious feel or else face very strong resistance.
November 1, 2011 at 3:55 am |
Tim, thank you for having this discussion — with the reach of your blog, this really could be a starting point towards real efforts.
—
I strongly agree with Noah Snyder’s first point:The biggest trouble seems to be that we mathematicians (as well as scientists in general) are lacking ideas for overcoming our exclusive dependency on peer-reviewed publications of “new” results to judge a researcher.
This exclusivity seems to have increased pressure to reduce the least-publishable-unit and, more discouragingly, lower the quality of mathematical writing: papers are less about communicating mathematical ideas and more about get-it-past-the-referee-to-increase-publication-metrics.
I have the impression that the focus on new or new-looking results encourages authors to make their work look much harder than it is — hard enough so that a referee doesn’t think it’s “too easy”. We do not value exposition or (heaven forbid) teaching, focusing exclusively on “new” results (I wonder what the mathematical analogue of the so-called decline effect might be).
Finally, this exclusivity is also why publishers can afford to charge ridiculous amounts of money: we lazily rely on their artificial metrics to hire, decide on grants etc.
As David Savitt pointed out, when peer review works, it works great! But given the pressure I think the chances for it to work are getting worse.
The question, I think, should be: can we relieve the pressure? Can we supplement peer-reviewed papers with other tools for evaluating the quality of a researcher? Can we maybe even develop tools that are about people, not metrics? For example, could we have tools that allow a hiring committee to find a referee that it would trust to give a neutral yet informed assessment of the quality of the candidate as a member of the scientific community as well as their ability to actually fulfill the position?
Journals were born out of the necessities of the pre-internet age and its need to limit communication and interaction between researchers so as to be efficient. Now we have countless ways to make the scientific community a reality, to actually go for a picture of the entire researcher.
Mathoverflow (and other such sites) are already on their way to help with this issue: they allow us to get a much more complete picture of a researcher as part of their community. Blogs and blog-like tools can also add to the picture allowing for exposition, teaching and direct exchange between researchers (currently they mostly lack good ways of making the complicated structure of interactions visible when it’s not as focused as a polymath project) as do all social networks if they are public. Video lectures, video conferences and internet-based teaching methods are also quite easy to document these days.
I don’t think we need to create a big centralized place, rather we need to make an effort to make our own activities accessible and connected online. This is hard work — after all, it should be! But I think if we can supplement peer-reviewed papers with other means, from mathoverflow to crowd-reviewed mathematical writing to “social graph”-like tools to teaching we could gain much more than it will cost: relieving the pressure on producing nonsensical papers while finding new ways of doing mathematics.
“Full Disclosure”: Over at Booles’ Rings we are experimenting with wordpress to collect best practices for a personal, academic homepage using decentralized tools to make “scientific community” a bit more true to the name — so this discussion is right up (or down) my alley so to speak.
November 1, 2011 at 2:23 pm
your link doesn’t work for me
November 1, 2011 at 6:21 pm
Any system needs to provide good tools for 1 minute evaluations of job applicants, or search committees will invent their own bad tools.
Can we have tools that allow a hiring committee to find a trusted neutral referee? Yes, if the hiring committee is considering 20 applicants. If the hiring committee is considering 600 applicants (as many US search committees do nowadays), any such tool takes too much time at the initial stage.
Would it be great if the search committee got some idea of each applicant as a mathematician rather than as a bunch of metrics? Yes, but with 600 applicants, that’s a already full month of full time work for one person.
November 1, 2011 at 7:24 pm
@Lukasz the link should have been http://boolesrings.org
@AlexanderWoo: no matter what tool we develop, nobody can force people to use it. If a department choose not to invest time and money in hiring suitable people for a position, we can’t change that.
I’m not saying that it won’t be more expensive (on some level), I’m saying that it’s necessary to make good hiring decisions — and I do think the current system is working more and more poorly.
I’m also obviously not saying that we have the tools already to decide “in 1 minute”.
—-
However, given what we have, here are some random ideas not to be taken too seriously:
* Of course, take a look at publications, h-index and what not (btw, for which position do you have 600+ applicants where these values are meaningful?)
* But also take a look at reputation points on MO, math/tcs/theoreticalphysics/stats stackexchange, orexchange etc.
* Also try to take a look at the “academic social network”: conference activity, the social graph on facebook, g+, twitter restricted to mathematicians — who knows, maybe all living field medalists are following the applicant on twitter? That’s a number… the number of video lectures the applicant has posted, the comments, the activity on their blog, the amount of people commenting on their blog, the google rank of their homepage.
In short: you should be able to look at the applicant’s academic homepage and see if they’ve taken steps to offer information that completes the picture — this is the applicant’s job and we should all take it more seriously (right now, we don’t have to because a publications list ordered by impact factor does the trick).
I’m sure this last part will give you a reason to throw most out in under 1 minute if that’s what you’re really after…
November 1, 2011 at 8:35 pm
Peter : Why on earth is someone’s facebook account or twitter feed or blog or MO reputation at all relevant to hiring them? That’s just absurd. And why do you think that the current system is working “poorly”? As far as I can tell, it works just fine, though of course there are too many good candidates and not enough jobs, so some people don’t get the jobs they deserve/want. But there is really nothing that can be done about that, at least not in the short term.
November 1, 2011 at 10:07 pm
@AndyP I think in your other comment you’re pointing to the crucial question:
As was already pointed out, peer review is still reasonably good at what it is supposed to do: check correctness of results (maybe even “importance” though this is, in an average paper, in the eye of the beholder).
As I wrote in my original comment, the problem is, in my eyes, that we exclusively rely on this system to assess mathematicians.
But producing lots of papers in the current system does not, in my humble opinion, imply that a person is qualified for a job at an academic institution.
In times of a shrinking mathematical community (the UK and Canada give us clear signs), we should hire people that can actually sustain our scientific community — and this requires more than just producing new results, it means be part of and shape a scientific community.
As Tim Gowers pointed out in his afterthought, there’s much to be gained from changing our culture so as to value other research activities as well, e.g., “non-new” research papers such as exposition, simplification etc. I would simply want to extend this to other activities.
But yes, I stand by those crazy ideas:
* a user’s reputation points at MO tell you something about them — most importantly that there’s content to check out their academic qualities in detail later.
* a blog such as Nuit Blanche can single-handedly keep you updated on an entire emerging field of mathematical research
* A discussion on google+ can lead to a quick resolution of a complicated mathematical claim.
etc.
November 1, 2011 at 4:19 am |
Lovely post. As you say, there has been a great deal of interest in similar ideas, particularly the idea of decoupling the publication/dissemination of articles from their certification. I’ve submitted a paper on this recently; you can find a bunch of previous work in the lit review. For now it’s living on a gDoc here: https://docs.google.com/document/d/1xDOy9GXXrUFc9TUIR2C470DTau8JEgZ9k-SMNIx5pb8/edit?authkey=CMeCqOYD
November 1, 2011 at 4:20 am |
Sorry, last comment deployed unexpectedly…here’s my actual name 🙂
November 1, 2011 at 4:41 am |
i’m intrigued by the idea, in general terms.
One difficulty which I see is the following: for this model to work, it would need to involve a LOT of people, and involve broad swathes of mathematics. Were such a project to be restricted to rather specialized areas of mathematics, I could see it devolve into a wiki in which the same N (a small number) of individuals publicly commented on each other’s work. Over time, this could be undesirable. Yes, even now one may be able to identify a referee- but there’s a reasonable chance a paper may go to someone not in the author’s circle. There is very little incentive to go look at, and review publicly for reputation points, papers in a different specialty.
This difficulty is ameliorated by having a model similar to that of MathOverflow- which is a centralized forum for all research-level mathematics. However, that has its own issues. There’s the formidable challenge of ‘founder bias’ of expertise. It’s highly likely such a project will become successful around papers in a few areas. It will, I think, be a challenge for papers in other areas to ‘break in’, as it were.
So, both difficulties are addressed by having your suggestion implemented separately for different areas of mathematics (getting similar in flavor to our current specialized journal system). There’s some optimal collection of MSC tags per site, but that’s another discussion.
November 1, 2011 at 5:53 am |
> 4. People might not like to have their papers evaluated in public.
and also
> 6. Some papers by well-established people might be more favourably reviewed than equally good papers by less well-established people
Why not allow (or even require) papers to be published under a pseudonym, with the option to claim credit later if the paper is well-received? The system could track this and provide a way for the paper author to verify his authorship. (4chan allows this, though pure anonymity is nearly always preferred there.)
November 1, 2011 at 9:20 am |
[…] How might we get to a new model of mathematical publishing? […]
November 1, 2011 at 10:01 am |
I would like to make a general observation about “(online) communities” (of which a new model of mathematical publishing is certainly an example) and “gamification” (used to create incentives, for instance with reputation points):
Creating functional communities is extremely difficult. We all take Mathoverflow for granted now, but it took two decades since the inception of the internet for it to be born. The creators of StackOverflow stumbled upon a formula that works (i.e. resonates with people), and took a lot of effort to refine and perfect it.
Tim, the detailed suggestions you make are all well-thought out and reasonable, but they are useless in the sense that the proof of the pudding is in the eating: herding people (mathematicians) into a community (“peer review community”) is a strictly experimental science (which is mostly practiced by game designers these days). To get a system like the one proposed off the ground, you need a dedicated group of people to create it and constantly refine it by trying out all your suggestions, well, and all their negations.
And there is still no guarantee that this will work as intended. The most likely result is a peer-review ghost-town. For instance compare Wikipedia to Citizendium. Same goal, slightly different method, huge difference in success. It is hard to get right.
(Ironically, I don’t see an incentive for anyone to actually build such a system at the moment.)
November 1, 2011 at 12:09 pm |
An additional twist that could help reduce manipulation (like up-voting your own or your friends’ papers) as well as reduce any blind reliance on numeric scores coming out of the proposed review website is the use of a “personalized page-rank”-like reputation system.
The reputation/ranking scores that I see will be calculated and shown specifically for me as a function of the sources that I designate as trusted (e.g. I start by trusting all Fields medalists, or by trusting the editorial board of virtual journal X). From this starting point, reputations of others are recursively calculated (X gets a bit of my-personalized-reputation if he was reviewed positively or up-voted by someone that already has some of my-personalized-reputation).
Different university departments, grant agencies, etc may have their own initial sources of trust, including specific ones for specific sub-areas. A self-congratulating clique can be easily disregarded as a source of reputation, and no single metric (like impact-factor) would cause world-wide uni-think.
November 1, 2011 at 12:37 pm |
Yes. Let’s do it.
November 1, 2011 at 1:39 pm |
[…] (science?) publishing. One might consider voting and government in a similar framework (that is to pose the two […]
November 1, 2011 at 1:40 pm |
[…] (science?) publishing. One might consider voting and government in a similar framework (that is to pose the two […]
November 1, 2011 at 3:13 pm |
Couple of opinions after reading the article and the comments. Very generally, I would like a system facilitating the currently exisiting communal work of recognizing good articles, but not a system which produces numbers (by whatever algorithms) by which then people are judged.
1) To minimize possibilities of abusing the system easily, everyone should register under his real name (and provide an academic email address; in the long run public key certificate).
2) I agree with Henry Cohn’s comment. If you think that reputation on mathoverflow makes people more keen to answer then let’s keep it, but let it have no influence on real life. The informal “who is who” knowledge based on what mathematicians actually think based on interactions with other mathematicians and their articles, should remain the only reputation system which counts in real life. I fear anything else is too prone to “gamification”.
3) The reviews should be allowed to be published under either the real name or a pseudonym. If it is published under a pseudonym, I don’t see a point of showing to the other users what’s the reputation of the one who posted it. This is because the only real reputation is as is point 2), in particular a nickname with a number is not more helpful than just a nickname, in the “a priori” judgment of whether the answer will be useful. However, if an anonymous answer is up-voted then the reputation points should go towards the person who answered.
3b) Up-voting of reviews should not be anonymous. If I see a good review, knowing that someone I value high up-voted it is much more useful than just knowing that someone up-voted it. For obvious social reasons, down-voting – if allowed – should remain anonymous.
4) I would largely keep the current system of journals, i.e. the users would form journals within the site, and these journals would with time gain prestige (again: this would be only informal prestige).
The hiring committees would be basing their decisions on these new journals.
I know that the hiring commitees, beaurocrats, etc. would start inventing new measures (or use the old ones) for these new journals, but this is a separate problem. In my opinion the problems which we’re trying to solve mainly are: a) getting independent from big publishers, b) facilitating the communal work.
November 1, 2011 at 3:30 pm |
I have no idea why posts like this seem to bring out so many comments from people who are not professional mathematicians. Why do they care so much about how mathematicians communicate?
In any case, I’ve given some more thought to this. I have to admit that I have no idea why people constantly claim that the journal system is broken. It seems to work just fine to me. The only real issues I’ve heard people bring up are 1. the open access issue, and 2. the cost issue. But we’ve broken the backs of the major publishers on open access in the sense that no serious publisher doesn’t allow you to post your papers to the arXiv or to your webpage. And as for cost, the solution is clearly to form more journals which are owned by nonprofit agencies controlled by mathematicians (like Geometry and Topology).
I guess people also complain about how long it takes for things to get refereed, but that seems unavoidable if we want referees to actually carefully read our papers.
Aside from the above two issues, what exactly is this suggestion supposed to accomplish? Social networking is the fad of the moment, but why should we rearrange our professional lives around it?
November 2, 2011 at 3:31 pm
Here’s my attempt to answer this question: http://agtb.wordpress.com/2011/11/02/the-problem-with-journals/
November 3, 2011 at 6:55 pm
Andy-
There are hiring and promotion decisions around the world being made on the basis of which journal people publish papers in. Is the journal system really so reliable and fast that you think that’s a good system?
November 1, 2011 at 6:07 pm |
Two points. First, disguising reputation might be a much harder problem than you might think. Just considering how many purportedly anonymized data sets in the past (search, movie preferences, authorship of anonymous manuscripts) have fallen to clever data miners, I’d be very reluctant to count on any disguising system that doesn’t come with an actual proof of security.
Second, many of the potential problems you’ve identified in getting people to participate, doing so without causing friction with colleagues, and so on stem from the fact that reviews and submitters are the same set of people–you are a reviewer of other’s work but will also be submitting your own work.
Perhaps a different model is worth considering. For instance, when you want to know whether a new movie is worth seeing, do you ask George Lucas? No, you ask Roger Ebert. Why not the same for mathematics (and science in general)–why not have the role of math critic?
A math critic would be someone who can read and understand mathematical research papers, and can advise authors on how to improve the paper, but who does not produce mathematical research of their own.
November 1, 2011 at 6:08 pm |
+1 to Andy P, particularly the second paragraph.
(Regrettably) it’s not clear to me that there’s even a consensus in the profession that a revolution is necessary. For at least a decade I’ve heard people daydream about the mass resignation of the editorial board of Inventiones (simultaneous with the formation of an analogous online or not-for-profit journal), but it hasn’t happened yet. I have to imagine that many or most members of that editorial board have given careful thought to what they are doing and why they are doing it, but they are still doing the work.
Inventionicide is one way that the revolution could start. Alternately I can imagine, say, someone campaigning for and winning the presidency of the AMS on a for-profit-journal-killing platform, and then doing the hard work of setting up the organizational and legal infrastructure that might allow this to happen.
One more thing: a publication system in which we simply put all our work online for free seems to be leaving a lot of money on the table! Print journals are profitable enterprises because there are huge library budgets which can be spent on journal subscriptions. Rather than getting out of that game entirely, the long-term best option for the profession might be (as Andy P suggests) print journals run by professional societies and other mathematician-run nonprofit agencies that would direct the revenue back into the profession. To my mind that’s where we should be heading.
November 1, 2011 at 7:04 pm
On the last paragraph, in such a system we’d still want some way for people in poor countries (and perhaps also people at poor schools) to get access. I don’t think this is a hard problem (allowing open access for ISPs in most countries should be straightforward technically, and lifetime AMS memberships could give individual access).
As a mathematician I agree with you that we should try to increase our cut, but as a citizen and a taxpayer I think the NSF should force all funded research to be publicly available for free.
November 1, 2011 at 7:21 pm
Sure. I don’t mean that we should take steps backwards on open access, but open access and revenue-generation do not seem to be mutually exclusive (as the for-profit journals have taught us over the last decade).
I learned recently why one can use exactly 16 of one’s AMS points towards one’s AMS membership annual fee. An AMS membership for a mathematician in a third-world country costs $16/year, so this policy allows third-world mathematicians to get free AMS membership in exchange for writing two Math Reviews per year. Point being, access in the developing world is not a new problem and AMS has already put some thought into this sort of thing.
November 1, 2011 at 7:37 pm
Oh, I see the confusion — where I wrote “in which we simply put all our work online for free”, I meant “in which the sole distribution method is free and online”.
November 1, 2011 at 9:16 pm
But once everything is actually on the arXiv doesn’t the money go away? Libraries won’t keep paying for journals if the full content of the journal is available without paying. (I think this was born out in practice for Annals and Geometry&Topology.)
November 1, 2011 at 6:30 pm |
I’ve been writing a ‘manifesto’ for a post-journal academic world, but this post largely sums up exactly what I would like to see. A few additional points:
1) You mention that an upvote from a highly upvoted person should count for more than from a less upvoted person. I propose, in addition, something more dynamic, which I think would greatly help in the hiring process. The system should track who I personally upvote and downvote, to get a feel for who I trust and who I don’t trust. Then, when I’m evaluating a candidate for a job I can have the system re-weight his upvotes according to who I trust and don’t trust. This provides many benefits, but the most important is that it protects the system from decay. If I recognize a certain circle of mathematicians colluding, I can personally downvote them and have their influence erased from the scores of candidates I am considering, even if the greater mathematical community has not agreed that this circle is rotten.
2) This system could work for academia in general, not just mathematics. The resulting website would not be substantially more complex. I believe it would be easier to find funding for a more ambition project.
3) I’m not convinced that this website could just be put up and expected to become popular, even if the website were perfectly implemented, and even if everyone would be better off if everyone switched. There is a valley to cross. I think the best way to cross this valley is by a pact, hopefully led by very strong mathematicians. The pact would simply say that for 1 or 2 years they will not publish anywhere else but this system.
November 1, 2011 at 6:50 pm
1 or 2 years is too short. Many very strong mathematicians don’t publish more than 1 paper every 1 or 2 years.
If you want a system for academia in general, you really need to learn a lot more about all the differences between how different disciplines operate.
November 1, 2011 at 9:02 pm |
“But if you merely find it completely uninteresting, then you should indicate that in a “positive” way, as is done on MathSciNet when reviewers content themselves with a bald description of the results with no further comments. ”
My MathSciNet reviews are always a summary of results, very rarely with additional comments, criticism or praise. I always thought that this is a professional way to review a published paper, and did not realize that people like you would come to the conclusion that I found it to be “completely uninteresting”. I hope that most people don’t read so much into the meaning of MathSciNet reviews.
November 1, 2011 at 9:18 pm
I should clarify — I meant the kind of “review” that just copies the abstract of the paper. Having said that, I do sometimes read MathSciNet reviews to see whether I see any phrases like “this interesting result” or “solves an old problem” or anything that might indicate some real enthusiasm on the part of the reviewer. But I don’t always take the lack of an expression of interest to be the expression of a lack of interest.
In any case, my main point was more the converse of how you interpreted it: under the system I suggest, if you don’t find a result interesting, then instead of saying that it is uninteresting, you could merely not say that it is interesting.
November 1, 2011 at 10:51 pm |
A few comments: (a) some people are really put-off by MO-style reputation points and badges, so these aspects should be an opt-in (i.e. by default a logged-in user wouldn’t see any, neither his nor those of others ; only users who tick a box would have those stats displayed next to their name and they would only see those of other people who ticked the box too) (b) hiring commitees wanting to access those stats would have to obtain the user’s approval, and agree not to ever disclose them (c) it is true that many Mathscinet reviews have simply the abstract copied whithout specifying why (is the abstract really offering sufficient insight? or perhaps the reviewer didn’t really read the paper?or is it a polite way to say it is uninteresting?) so the possibility that several people at different times add comments on parts of the paper would indeed improve that matter.
November 2, 2011 at 12:55 am |
Please keep in mind that any system has to work not only for research universities but also for liberal arts colleges and regional comprehensives, as well as institutions in the developing world, which have a large range of different expectations (as well as support) for research productivity in hiring, tenure, and promotion.
November 2, 2011 at 1:47 am |
I think that the reviews on mathscinet that just copy the abstract of the paper are automatically generated. This just means that mathscinet could not find a reviewer, or that their selected reviewer did not respond sufficiently fast to the review request. It does not mean that the paper is uninteresting.
November 2, 2011 at 2:44 am
Actually, as may already have been pointed out earlier, reviewers are explicitly given the option of “cut and pasting the abstract as a summary” when they are assigned papers to review.
November 2, 2011 at 11:19 am |
[…] interesting contribution that I read recently is by Tim Gowers on his blog: How might we get to a new model of mathematical publishing? It think his ideas are quite interesting, although futuristic. Maybe we could get there step by […]
November 2, 2011 at 2:29 pm |
[…] Tim Gowers destroys and rebuilds mathematical publishing. […]
November 2, 2011 at 3:30 pm |
[…] Gowers recently suggested an answer to “How might we get to a new model of mathematical publishing?” which I highly recommend. While there has been much talk for years now on how to replace […]
November 2, 2011 at 10:01 pm |
[…] Gowers wrote an interesting post where he proposed in surprising many details an Internet mechanism (mixing ingredients from the […]
November 3, 2011 at 8:55 am |
Let me first say that I haven’t read the comments and I know that at least some of the things I will say should be a repetition of what is said before. Also, let me say that I would be really happy to see such a site. But, also let me say a few reasons why I think this might fail:
First, as you mentioned yourself, people’s job and payment depend on their papers being published. So, they would not leave a well-established journal for a website anytime soon.
Second, mathematicians are not an isolated community (at least they’d better not be). So, if there is going to be a central way to distribute national funds, there should also be a metric to compare the activity and productiveness of a field to those of another field (and it doesn’t matter how much you might dislike metrics). They need some measure to compare and they will not let go of this easy measure anytime soon again. So, mathematicians also need to stick to the journals at least as long as other fields do so.
Third, as you yourself point out, there are a lot of varieties to how to setup the system, a lot of parameters to set and a lot of options to enable or disable. I think it should then be easy to see that there is no universally agreed setting that everyone prefers. So, why would you think that clones of this system will not appear all over the place? (all with different settings and, of course, different user bases and probably covering different sub-fields). How do you measure the papers in different clones against each other? Wouldn’t you have to define a (much harder) metric again?
Fourth, again, this is a system which is going to have effect in real-world, i.e., people’s jobs and payment. So, who would administer the system? What about dishonest administrators selling points or tampering with negative reviews? You cannot compare it to math-overflow in this case because math-overflow has no serious real-world repercussion. Even if there is no such behavior (which I hope is the case), the system is completely susceptible to such interpretations and such interpretations alone have the capacity to make the system fail.
Fifth, scientists are, generally, anarchists by nature. They don’t like centralized systems where there is a single point of control. Journals are decentralized by nature and so much less contentious in this regard.
November 3, 2011 at 2:55 pm
Even in a non-anonymous system such as mathscinet reviews can be dishonest. There are examples there of on one hand, cartel self-support, on the other, of genius independents getting nasty comments from jealous but much weaker competitors. Actually, why should mathscinet not allow reviews from established researchers, even if just to straighten things out.
November 3, 2011 at 11:32 pm
“genius independents getting nasty comments from jealous but much weaker competitors”
That seems a bit hyperbolic to me. Any chance you’d care to furnish the links, for those of us who have MathSciNet access?
November 3, 2011 at 11:48 am |
I think this could be simulated in a manner analogous to “The Peter Principle revisited”. It might make an interesting area of game-theory.
November 3, 2011 at 5:48 pm |
[…] Andrew Stacey. The incentive system I proposed (roughly speaking, Mathoverflow-type reputation points) will not […]
November 3, 2011 at 5:48 pm |
[…] Andy P. Even if a new system is demonstrably better for mathematicians, it still needs to be taken […]
November 3, 2011 at 6:52 pm |
[insert name of your favourite not very good journal here]
My favourite is the Antartic Journal of Mathematics.
November 3, 2011 at 11:35 pm
AARGH! MY EYES! MY EYES! It’s Zombie GeoCities!
November 3, 2011 at 8:53 pm |
[…] to spell out the problems with the current academic journal publishing system, and pointed out to Tim Gowers’ post that suggested how it can be replaced by a combination of arXiv and a math-overflow-like online […]
November 4, 2011 at 4:30 am |
For a while there has been a notion of “overlay journals” in which papers are posted on the Arxiv and also vetted as part of a normal journal submission process seems to handle a lot of the complaints about the long time to publication that have pushed people to suggest crowd-sourced reviewing as suggested here.
Overlay journals don’t get around sloppy journal refereeing (which is still much better than one gets in reviews for selective conferences and the kind of review I would be likely to produce if it were merely a rating system) but it does clearly emphasize the journal’s role as being primarily in the reputation and vetting business rather than in the dissemination business. This seems compatible with the crowdsourcing approach but I don’t see crowdsourcing as an alternative model because of its incentives:.
I have no trouble with not being publicly acknowledged.for my refereeing work – what is meaningful to me is really the editor’s acknowledgement of the private favor that I am doing for them by writing a careful review. That lack of a specific third party for whom I am doing favor makes the crowdsourcing approach less appealing to me as a reviewer.
November 4, 2011 at 6:48 pm |
I tried something very similar to this with the quantalk project (http://quantalk.org) – basically we created a system where people can upload papers, like arxiv.org, but where reviews and indeed other comments can be made and are permanently recorded — essentially like an extension of the paper.
The system was actually used in a (largely successful) experiment by a real journal, the International Journal of Quantum Information. Several papers submitted to a particular special issue were reviewed on quantalk, using our system whereby referees can choose whether to be anonymous or not.
There were some less encouraging outcomes, though. Specifically we found when the site is used simply as a place to discuss science (so that that authors to not have much at stake) then authors will often not bother to respond to comments/questions.
November 8, 2011 at 3:50 pm |
[…] How might we get to a new model of mathematical publishing? (gowers.wordpress.com) […]
November 9, 2011 at 8:27 pm |
[…] the biggest splash last week was the post of Timothy Gowers on what we would come up with if there weren’t any journals. It was followed up by Computational Complexity, Gowers himself, and the Geomblog, that also gave a […]
November 11, 2011 at 12:13 pm |
What you described seems like blogs. So no new software or website are needed.
November 20, 2011 at 1:39 am
I agree that new software is needed. Let me explain why.
If mathematicians are sceptical about working on original results, then assuming they are right, what is the cause of it? Here is an experience I had not so long ago.
I was working on an equation in high-temperature superconductivity. After a many years effort, I concluded with what I thought was a sensational finding of a new solitary wave in closed form, which was not detected by numerics in a paper I was following up on – maybe because the solution contains three irrational numbers. I posted our paper on arxiv. After some time, I was contacted by the main author of that numerics, who expressed interest in sth else in our paper. A bit later, he contacted me again, saying that the closed form solution was found a few years earlier by his countrymen. Since I think I am very meticulous about the literature, I decided I would rather believe that we live in some kind of parallel universe reality, and that the solution was stolen by passing thoughts back in time.
Modern nonlinear science is more adventurous than physics and mathematics, e.g., because with cutting edge analysis it can come up with results that the physicists think can’t exist. That means that it is confident with whatever maths comes out of a physical derivation. E.g., classical dispersion relations can quickly give non-causal evolutions.
What is missing in our research is the software to time stamp our results without making them public. A serious institution would have to do it, much like the way money is printed. At this time the only solution in this direction is to write one page, submit it to arxiv for a time stamp, and have them reject it. But that’s impossible with what they accept, isn’t it?
November 20, 2011 at 1:55 am
Actually, Ralph, there’s been an essentially perfect timestamping service, available for free, since the early days of the internet. Have a look at .
November 20, 2011 at 1:55 am
Actually, Ralph, there’s been an essentially perfect timestamping service, available for free, since the early days of the internet. Have a look at http://www.itconsult.co.uk/stamper.htm
November 20, 2011 at 3:06 am
5. I. T. Consultancy Limited reserves the right to specifically withdraw the use of the Stamper service from any person or body without notice.
I was suggesting a serious institution. Actually to be more precise, several, so that there is good chance that some will survive in the long term.
November 20, 2011 at 4:37 am
Indeed. My recollection, however, is that one can easily download some data from their website that effectively serves as an irrevocable (barring a very wide conspiracy) certificate of your timestamp. They’ve thought about this.
November 21, 2011 at 12:23 pm
Ralph, your request makes perfect sense for me. I was going to recommend the same service as Scott Morrison did. I recall that there was a service (some student’s project) with the same purpose. I will look further if you are interested.
What is your definition of “seriousness”? 😉 If a service is free, it can be revoked at any time. Nobody is obliged to work for you for free, you know.
If you are concerned about truthfulness of time stamps, there is a cryptographic proof that the chain of time stamps is increasing. I had in mind a more complex scheme, where many witnesses check time stamps with their clocks. A single agent (even government) does not provide a truthful service, this is commonly accepted.
November 21, 2011 at 1:24 pm
beroal, I don’t have a cryptographic education, however, I have just found this at http://en.wikipedia.org/wiki/Trusted_timestamping
The idea of timestamping information is actually centuries old. For example, when Robert Hooke discovered Hooke’s law in 1660, he did not want to publish it yet, but wanted to be able to claim priority. So he published the anagram ceiiinosssttuv and later published the translation ut tensio sic vis (Latin for “as is the extension, so is the force”). Similarly, Galileo first published his discovery of the phases of Venus in the anagram form.
My question is then: If this paragraph means that Galileo followed up, then what software would prove it?
November 21, 2011 at 8:47 pm
“If this paragraph means that Galileo followed up, then what software would prove it?”
Sorry, I don’t understand.
November 14, 2011 at 5:05 pm |
[…] reading this blog who does not also read Tim Gowers, but in case you missed it, here’s his blog post proposing a hypothetical alternative publishing model in mathematics: essentially, a massive website combining the functionality of arXiv, Math Overflow, and more. […]
November 17, 2011 at 6:57 pm |
[…] interviews, it seems to be grudgingly accepted to be the least bad form of evaluation. Recently Timothy Gowers raised the possibility of an alternative system on his blog, which led to much fevered debate (I […]
November 18, 2011 at 11:26 am
A bit off-topic, but just out is an interesting review of combinatorics titled Aftermath http://arxiv.org/abs/1111.4050 by an author who published 290 papers. In particular a mention of Green-Tao.
On-topic: MathSciNet doesn’t distinguish between paper and book citations. However, if there is a well-defined area that is sufficiently diverse, there is no leadership and books, so original results may seem trivial to hiring committees, because no one has taken profit on them. It is quite easy to spot classical results from e.g., just the body language of the first page. In this way, dozens of thousands of job applications can be evaluated very quickly, so abandoning the cv/letters requirement could result in a dramatic improvement of the quality of our research, if funding could be shifted to supporting competitive phd students – nowadays the brightest people are seldom interested in science. That this would also result in a much higher unemployment rate for phds seems to be desirable, so that they don’t cluster just in finance to the detriment of us all.
November 27, 2011 at 5:40 am |
A few additional ideas:
(1) The web site should maintain paper versioning system. Reviewer’s remarks could result in the author updating the paper. In such a case reviewers should be rewarded by noting this in their public profile.
(It seems that for this feature it would be better for the site to store the papers rather than provide pointers to arxiv).
(2) Here’s one idea on how to resolve the problem of unreviewed papers. I suppose that the reviewing reputation will become very valuable. People who are not first rate mathematicians may become
very valuable reviewers. It is possible that such people will establish real life careers almost solely on their reputation as reviewers,
so a new class of people within academia may emerge separating the
producers from the reviewers.
The system may therefore require reviewers to occasionally review an arbitrary paper in their field as a precondition to submitting future reviews, (I realize this is tricky and will likely meet opposition).
(3) On the other hand when reviewing is rewarded, we can expect many more people to actively review papers. When you are teaching a graduate level class, you may assign your students to review several recent papers. This will increase productivity tremendously. Student work will immediately add value.
(4) It seems to me that the business model of private companies that make their profit off scientific publishing is parasitic. It seems that they do not add value. Sometimes the opposite is true. I recently had to publish a paper according to ACM publishing guidelines and found that the formatting requirements were very time consuming.
The current peer review system is broken. It may still be working within narrow subfields, but it will not cope with the increasing volume.
Web-based scientific publishing is a necessity. The exact mechanisms
will evolve gradually and the ideas outlined above could be used as starting points.
November 27, 2011 at 8:43 pm |
[…] In particular Tim Gowers has been asking similar questions for peer review in mathematics (see here and here) As we have seen above, such a system is likely top be highly controversial as well as […]
November 28, 2011 at 10:02 pm |
[…] A Popular Solution One popular solution to do this is a combination of Reddit and the ArXiv for academics. Researchers can post their papers online and then other people can leave comments and reviews of the paper. Everyone has a reputation score and the influence of one’s comments depends on their reputation. Maybe papers can get assigned scores, so anyone can score the paper, but the weight of their score depends on their reputation. That way, on my CV I can write down all of my papers along with their scores, so that others can quickly glance at my CV and get an idea of how important my research is. This is the basic idea but obviously there are a lot of details so that one cannot game the system. I’ve spent some time thinking about this and I think that if you implement it carefully in enough you can make it work. Timothy Gowers also seems to think so and he has thought about many of the details. If you are interested, please read his blog post, here. […]
November 29, 2011 at 1:44 pm |
[…] (matematico, medaglia Fields) si sono palleggiati una discussione, qualche settimana fa, su come cambiare l’attuale modello di pubblicazione scientifica nella matematica (peraltro, è la seconda volta, 2 anni fa una loro interazione aveva portato alla […]
December 13, 2011 at 2:37 am |
Seeing how some of my professor colleagues handle their referee duties, I would say that peer review under an editor is way more effective than voluntary reviewing. Maybe we should aim for a mixed model, i.e. let anyone write public reviews on some common blog page (that should also be sorted by topic) and put “coerced” reviews into some journals.
Maybe we also need two-party negotiations about prices. So far publishers unilaterally dictate prices for journals and libraries deciding one-by-one have little power (beside dropping a journal) on price decisions.
Best
elf32
December 13, 2011 at 9:01 am
Even your description of the pathetically small power wielded by libraries is an overestimate. Elsevier has a policy of bundling together collections of journals and telling libraries that they can have all of them or none. So dropping a journal is not as simple as it seems. (I’m not certain that this policy still exists, but it certainly used to, and caused an uproar when it was introduced.)
December 13, 2011 at 10:38 pm |
Issue 4 seems to me the most important to address:
4. People might not like to have their papers evaluated in public.
How about instead, the reviews only to be visible to a small number of people, including the author(s), all people who have commented on the same article, and the editorial board of the journal to which the article is submitted for publication.
Indeed, if I submit a paper, I would not mind getting even negative responses (especially if I would have the opportunity to respond to them), provided I know the discussion stays “within the community”, within the group of people who care about the article. But I would not like Joe Doe from the street who has no idea about mathematics in general to read inner workings of a paper that is not yet in its very final form.
I look at arxiv.org as announcing a result, and at journal publication as confirming its correctness. Both are necessary, but the former can not be a substitute for the latter.
Your suggestion can remodel the reviewing process for the journals that care to follow your method.
Also, I don’t understand the purpose of reputation points. (I know, this is a statement, not a question. 🙂 ) What if somebody comments only once in a while, but the comments go to the core, while another person responds every day about spelling? Both types of responses should be welcome, but it seems to me unjust to issue reputation points.
December 13, 2011 at 10:57 pm |
Please disconsider my remarks in the last paragraph. You address these issues here:
which I am only now reading.
December 14, 2011 at 4:47 pm |
Related blog post (and journal article) in the physical sciences: http://bjoern.brembs.net/news.php?item.812.11
December 19, 2011 at 10:33 pm |
Does anyone know of a decent paper published by International Press – modulo seven in DPDE by Y. Charles Li, the editor of DPDE.
December 22, 2011 at 11:57 am |
Interesting post. I have proposed a very similar idea several months ago to a colleague, namely a cross between arXiv and a social networking site. I also suggested to get the idea rolling initially via a blog/journal publishing arXiv articles, peer-reviewed in the blog comments – for the sake of technical simplicity. So essentially what you’ve described.
Having experienced other social networks with a creativity streak – mostly in art like Deviantart – the danger of “gamification” or “score-hunting” does arise as many here have realized in advance. Then there is the issue of getting lost in the crowd unless the ranking algorithm is phenomenal – hardly ever. Any ranking system would reduce a person to a score, but how could the scoreless / no-reputation newbie researcher be noticed for something ingenious? People would just follow the big fish, because there would be way too many little fish tiring their eyes. (Just consider, we all follow Terence Tao’s blog, but who follows ours?)
So I have got to the point where I discarded the above idea altogether (though I still am in a dilemma whether to create such a blog with a limited group of researchers inviting each other). Instead I asked myself, why are we doing research in the first place? Is it not to contribute to something greater than ourselves… the development of mathematics / science? Do we really need to be credited for our contributions? Does it have to be an ego game? How about a world of research where no-one cares about who did what? See where I’m getting at…? Wikipedia! Sure, there are wiki sites on math, but what if that’s all research would become?
The above idea is a little naive in its brevity and humility. The actual website would have to be a little more sophisticated than a wiki, to account for measuring contributions and the relevance of those contributions (one will still need to put something on their resume to land a job). Possibly a thumbs up/down system per each wiki edit per user (stars don’t work btw). If an edit gets too many downs, it would get removed. Let’s just keep it simple. (No wiki discussion pages = battlefields either, they are soul-killing.) Then on one’s personal profile, the contribution stats could be shown, exportable for resumes. So quantity vs. quality would be displayed analytically.
As far as the edits themselves, a user would have total freedom as on a wiki. Brief to longer edits ok. An edit would be highlighted as new for let’s say 3 days, at which point the votes would decide if it remains included in the body of text. But if a person gets too many thumbs down on edits, they could be identified as potential spammers by the admins. The site should require a scan of one’s master’s degree and edits would be only permitted in that field of science, as verified by admins upon registration. (Thus no need for a weighted voting system, since users would be professionals by default, and a newbie’s vote is potentially as accurate as others’.) The site could be peer-invitation only in its infancy, to ensure coherence of material through existing collaborative relationships. The first researchers invited should be prime representatives and authors of a field, so their writing would serve as the foundations of the website. We would always see where research is at by just scrolling to the end of an article. There would be no need to write intros to topics as we did in papers, since the info would all be there.
So to sum up the idea, research would become a coherent evolving online encyclopedia striving for absolute complete knowledge, just like it should have always been in real life in my opinion. Let’s admit, the current journal system has always been just a substitute for the real thing, that we couldn’t yet make happen without the technology… but now we can! What do you think? Worth a try?
December 22, 2011 at 2:11 pm
“(No wiki discussion pages = battlefields either, they are soul-killing.)”
IMHO banning discussions is not a way to do science. It’s a way to do brainwashing. 🙂
I like to compare blog posts to articles and wikis to textbooks. Blogs offer news, wikis offer the core. Lets not mingle them.
December 23, 2011 at 5:48 am
It’s a technical detail, but here is an illustration of my point http://www.informationisbeautiful.net/visualizations/wikipedia-lamest-edit-wars/. The current wiki discussions system can get too personal, which a thumbs up system would eliminate.
As far as blogs vs. a wiki, it’s not a matter of comforting analogy, but long-term functionality.
December 23, 2011 at 11:42 am |
Dear József
What you write is quite interesting “research would become a coherent evolving online encyclopedia”, something similar also came to my friend’s mind some years ago, but may be it is too radical or it least may be such project should coexist with Tim’s proposal. And wikipedia may be considered as a kind of good prototype for this.
Best wishes, Alex
January 4, 2012 at 7:49 pm
Dear Alex,
I think we are currently in brainstorming phase and no proposal is solid. So we must try to come up with an ideal philosophy before action, and I perceive this to be Tim’s effort as well. The ideal one will undoubtedly seem radical by current standards, since the journal system is hopelessly outdated and not in line with the potentials in our new online world. The old system will still have its adherents like classical music, and co-existence of the old and the new is inevitable. Perhaps the new way could take the shape of a movement, with a manifesto of guiding principles.
Best wishes, Jozsef
January 5, 2012 at 9:57 am |
[…] community. As an entry to the discussion on the role of journals you may consider the post “How might we get to a new model of mathematical publishing?” by Timothy Gowers (and the follow-up “A more modest approach”) and also some […]
January 23, 2012 at 5:55 pm |
Wow… lots of comments above! I admit I didn’t read all of them… Anyway, a great post. Here’s a couple of comments:
– The reviews could initially show only to the reviewers and the author
– Someone who reviews a lot but doesn’t get reviews could maybe trade his rep points? Sounds a bit iffy though…
– Force login with google, fb, openID etc. The web is going away from anonymity anyway.
-I also think that the rep point system should not be taken too seriously. Number of citations should of course still weigh a lot more that some reputation points. But gamification isn’t all that bad IMO.
-Also, it should NOT be too complicated. It would only end up in an uncontrollable shitstorm…
(and pardon me for not having read Tim’s later post yet either)
February 2, 2012 at 10:37 pm |
[…] already had a go at suggesting a system, back in this post and this post. Another system that has been advocated, which I also like the sound of, is […]
February 17, 2012 at 1:56 am |
[…] also (things I have found after writing the above): Gowers’s How might we get to a new model of mathematical publishing? Gowers’s more modest proposal This excellent thread at Math 2.0 Nikolaus Kriegeskorte’s […]
February 29, 2012 at 2:31 am |
[…] exactly in these days many scientists and thinkers (notably Timothy Gowers, a Field medalist 1, and Michael Nielsen, the author of Reinventing discovery, an interesting book […]
May 2, 2012 at 6:30 am |
Sadly I’ve not had time to read this post and all the comments in detail, but wanted to pitch an idea into the mix on this whole issue of academic publishing and the peer review process. From what I can see individual researchers, and the institutions that they operate within are all getting ripped off one way or another by the commercial publishers, with the result that no-one best interests, other than perhaps the financial interests of the publishers’ shareholders, are best served.
To me this situation is crying out for a cooperative model, whereby the people and organisations that wish to produce and consume the knowledge come together to jointly meet their needs.
The Guardian piece that prompted me to visit here (http://www.guardian.co.uk/science/2012/apr/09/frustrated-blogpost-boycott-scientific-journals) suggests that UK universities alone are spending something like £200 million annually with publishers currently. Clearly this money could be used instead to build and maintain a not-for-profit system that could do a far better job at a far lower cost.
As is pointed out above, an online platform would serve this approach very well.
May 23, 2013 at 8:07 am |
[…] Gowers, some time ago, in this post, […]
April 30, 2014 at 3:04 pm |
[…] Frontiers in Computational Neuroscience special issue, Tim Gower’s influential blog post, “How might we get to a new model of mathematical publishing?,” or Faculty of 1000 Prime, the highly respected post-pub peer review […]
September 20, 2014 at 9:33 pm |
[…] are not necessarily original (for example, I have been inspired by posts and related discussions as this one and its follow-up), and I have never taken any real action to see whether they could be tweaked […]
September 30, 2014 at 3:54 am |
[…] ideal world” as Timothy Gowers, Royal Society Mathematician and Fields Medalist summarized in his terrific vision to revolutionize scholarly communication and publishing. It will be interesting to find out how this will also improve the motivation of reviewers to do […]
January 2, 2015 at 2:25 pm |
[…] raised some years ago by Field’s medalist Timothy Gowers. In 2011 Gowers asked in one of his blog posts how we might get to a new model of (scientific) publishing, however focused, but principally not […]
October 21, 2015 at 10:59 am |
[…] for a boycott of the traditional commercial scientific publishers, in particular Elsevier. (See here, here and here for more on that.) So anything he does regarding Open Access and the use of open […]
October 23, 2015 at 11:30 am |
[…] for a boycott of the traditional commercial scientific publishers, in particular Elsevier. (See here, here and here for more on that.) So anything he does regarding Open Access and the use of open […]
February 8, 2018 at 12:12 pm |
[…] Timothy Gowers: Gower’s Weblog Blog, 31.10.2011 – https://gowers.wordpress.com/2011/10/31/how-might-we-get-to-a-new-model-of-mathematical-publishing/ […]
February 8, 2018 at 12:50 pm |
[…] ideal world” as Timothy Gowers, Royal Society Mathematician and Fields Medalist summarized in his terrific vision to revolutionize scholarly communication and publishing. It will be interesting to find out how this will also improve the motivation of reviewers to do […]
February 9, 2021 at 8:01 pm |
life requires
How might we get to a new model of mathematical publishing? | Gowers's Weblog