hckrnws

I think it's wrong to defend the current state. Even journals like Nature are game-able to a large degree. You can publish quite mediocre science in them if you socialize with the right editors at the right times. High impact journals are too much of a social club to be trustable, and social actions take too much energy from scientists.

eLife is the only high profile journal that experiments with the process. Their experiments may be in vain, because they still need to come up with an alternative measure that funding agencies will use to distribute grants. You can't replace peer gatekeeping with nothing, there has to be something else. That said, the "publish then review" model is great, it is a straightforward mechanism that has worked on the internet for decades. We have AI that can weed out the obvious garbage , and the open review process can enhance papers greatly.

The irony is that big publishers create new open access journals with fast review time to jump on the bandwagon. They know it all sucks and they don’t want to miss the opportunity to make money.

Publishing reviews along with a paper sounds amazing to me, that’s a part you never get to see in journals when you’re not a participating reviewer. I’d love to have some reference points of expert opinion and critique come with each paper.

I can understand the fears about submission quality. People do prefer submitting to ‘elite’ journals with low acceptance rates, because getting accepted becomes a validation. I wonder if publishing the reviews might take care of that anyway, if the audience will tend to read only the papers with high reviews.

Maybe they could have the best of both by publishing a curated set of the top 10% of papers by review scores in a best-of version of the journal, and still publish everything in the main journal. The move to consider publishing everything makes sense in this new world where all papers are on Arxiv, and they frequently make rounds in the press and social media before getting reviewed. I would welcome the eLife changes if it helps quell the trend of completely unreviewed papers vying for attention.

The review process is noisy and subject to much social engineering, lobbying, editorial PR, and quid pro quo if also serving as reviewer.

Reviews are inherently noisy because there are usually no more than three (over-taxed, unpaid) reviewers. But papers are now often highly multidisciplinary. Luck of the draw applies with force at n = 3 or even n = 10.

Reviews DO improve quality of final product, especially in the better journals. But there are still huge gaps in reviewing due to the underlying assumptions about “good” science.

For example, the majority if experimental studies using mice and rats use only a single genotype or strain of animal and are therefore essentially n = 1. My inclination is to reject a great majority of such studies without serious review because they are just fancy case reports.

So if you lucked out and did NOT get me as a reviewer you may be good to go at Science, Nature, or Cell. If you did not luck out then you will have to fight with the editor-in-chief to overcome my fundamental dismissal of case reports in experimental biology.

I support the eLife direction. We have squandered 20 years already and the hegemony gets stronger. Eisen is a bold publication experimentalist.

I don’t work in your area, but my observation is: you seem to have a strong opinion about the validity of certain experimental work. Moreover, your opinion does not appear to be broadly shared by your fellow reviewers, and (here I make a second presumption) also by many experimental scientists submitting work to these venues. I don’t know if your opinion is valid or incorrect - that is incidental to the discussion - but I sure wish that reviewers would agree on basic standards in advance, rather than surprising scientists with disagreements about them after researchers have done enormous amounts of work and spent public funds. The situation you’re describing (basically, having a debate about standards in the wrong place) seems very harmful.

Do review comments get publicly shared under the previous model?

I'm curious if a large part of this push-back is "I don't want to be held publicly responsible for my review comments..." (the over-taxed, unpaid ones)

... which leads inexorably back to the place publishers don't want to go -- reviewers deserve to be well-compensated, for doing a time-intensive, hard job well. And especially because they create the bulk of a curated journal's value.

Nature publishes review history for some. Also, F1000 follows this model, although not officially a journal.

There are definitely some wrong things with peer reviewing and scientific publishing, but this is hardly a solution. Sure it’s one way to do it; eLife is probably going to turn into a kind of arXiv with reviews. That is, a repository, not a journal. Whether that is good or not for them I guess it’s subjective, but it’s certainly not the solution.

I think one interesting model is the one by PLoS (especially with One). As long as the methods are valid, the research is original, and the language is minimally appropriate, the paper gets published. While they strive to eliminate the kind of subjective bias you get from small tightly knit communities (where subjective/invisible criteria are enforced in an informal way), they at least try to clear out the obvious junk. As a consequence, they too have become a half-repository, but at least one with a certain entry barrier. Then, at some point during the year, they make a collection of highlights or special picks for the previous year, which kinda work like what a conference would do.

It'd be interesting if you'd give arguments for why you think it's not a solution.

> As a consequence, they too have become a half-repository, but at least one with a certain entry barrier.

eLife will still have an entry barrier, but passing that barrier doesn't give you access to publication, but to a positive rating in the assessment of the significance and rigour. In other words, it will still perform the function of highlighting potentially relevant research, but it doesn't block non-highlighted research from being accessible.

I think that’s only PLoS One.

My experiences with PLoS Biology and PLoS Computational Biology have been great, but very similar to other journals.

Yes. The idea of PLoS One is that people can subject directly to it, but also it serves as a second chance for papers submitted to the higher tier PLoS journals -- an editor can reject the paper from those while still suggesting that the authors submit it to PLoS One (if the reason for rejection was that it wasn't that exciting but still solid research).

The same people who made PLoS are the ones reforming eLife. Exact same.

Well, it's Michael Eisen, yes, and he was a co-founder of PLoS, but it's not the "exact same people" other than that.

That’s a weird change. Publishing reviews alongside papers is good. Removing the “stamp of approval” from the whole process is weird. Imagine if every student who managed to enroll into a university would receive a diploma; and every course would only provide a review of a student instead of grades. That would just make everything much harder.

Yes, it may seem wrong to put things into black and white buckets; but many our decisions inevitably are binary and we have to make them no matter how hard it is.

The presence in a journal is not a grade. It is a "pass".

Perhaps some journals only let grade "A" papers pass, but sometimes that same journal has an editor who might let their buddy's article slip through. If one is part of the old boys club, this is a nice situation.

What eLife wants are signed grades from editors and reviewers. The editor is supposed to attach a brief summary in a few sentences summarizing the reviews. Hopefully one might bother to read a sentence or two to evaluate a paper instead of merely looking at the cover of the journal.

eLife's move here is basically a statement that the system is corrupt. Those wanting to fight the corruption are trying to increase transparency and reduce arbitrary decisions.

The whole point is that journal prestige is harmful and fake. It's prestige by association. That's not how science should work.

The problem is that science has not figured out how to work without this system.

Sure it has. The modern “scientific publishing” industry is much younger than science.

Yeah, it's the formal academic hiring system that hasn't yet figured out how to work without this. But of course, that's what eLife is trying to figure out: they're trying to separate recognition (in the published assessment) from publication. The big question will be whether those assessments will be able to play that role in academic rewards, because that's the root of the incentive system that preserves the status quo.

Being able to produce a bazzilion papers of low quality is also new.

Lolol

Yeah we have. It's called reading the paper and applying judgement.

Maybe you do that.

But my HR department which made me write "publish X papers in top-tier journals (see list)" as a condition in my probation agreement most certainly does not.

And alas, as long as I'm under probation, their opinion carries far more weight in the debate than any moral argument you could come up with.

Yes I do that, thank you.

Thanks for clearly stating how journal prestige is harmful to academics and scientists.

I didn't say I disagreed with you, let alone I approve of this situation.

The point was that utopian idealism that ignores reality on the ground is not as useful as you might think.

Your comment was as useful as me saying if I was president I'd give people free money.

> It's called reading the paper and applying judgement.

Many important papers these days are interdisciplinary. Like, medical students doing bioinformatics and data analysis. But it’s impossible to be proficient in all the methods. So, many parts of these papers are impossible to assess for a wide audience, trust and faith in a journal quality becomes most important.

I don't do my science by faith.

Then you probably don’t do science in 2023.

In some fields, there are thousands of submissions daily to Arxiv. Good luck staying up to date.

That's not a problem to be fixed through fake journal prestige.

When was the last time you were hired like that?

About 8 months ago :)

The change just means that the stamp of approval is not the journal name, which is fine. If you think about it, that system is a bizarre and lazy way to attribute value to a piece of work. The book publishing model figures out which books are great without caring at all about who publishes something (for the most part)

They're not removing the stamp of approval:

> with a short editorial assessment of the work’s significance and rigour

However, not getting the stamp of approval will no longer lead to the work not being accessible to people.

But every paper can already be accessed by anyone if its authors decide to put it on arxiv, GitHub, their personal website or anywhere else they wish.

So does that mean that the only practical difference is that now when you list your papers for a promotion or tenure review you will write “eLlfe (approved)” instead of “eLife”? Then what is all the fuss about?

1. There’s still some hurdle before new-eLife will send your paper out for review. This is not necessarily “quality” or “impact” level; they say it will depend on whether they can provide a helpful and effective review.

People had very mixed experiences with old-eLife’s editorial decisions, and some felt it was a bit clubby: papers from some labs sailed through to acceptance while similar work from others got editorially rejected (without review) for seemingly minor reasons. Their policy against requesting new experiments could certainly be used as a cudgel: an editor can just say you’re not convinced by a control and the paper’s DOA.

Thus, they are both uniquely positioned to make sweeping changes and have people doubt them.

2. The reviews use a controlled vocabulary, so I think you’d say eLife (excellent approach, major impact) or whatever the words are.

Yes, but now the authors don't need to manually decide to also share their research elsewhere - something they've (in many, but not all, disciplines) been trained not to do as publishers traditionally wouldn't allow it.

The fuss is about publication and approval being seen as linked in the academic hivemind. Even if you're able to see them as separate steps yourself, that doesn't mean that the academic world at large sees them as such, so if this fails to gain widespread acceptance, the reputation of eLife-the-journal might suffer, and having been published there will no longer look as good on your CV as it currently does.

In other words, the people who are afraid of reputational loss are the ones who have benefited the most from the broken system of academic publishing.

I won't cry for them.

The main question is: will those people be able to prevent this change from happening. From the article, it looks like they've been able to delay it already.

It will also change the process from the perspective of an author. If you are not fully sure about the impact of your publication, you might start submitting it to the journal with the highest possible impact. And, after a rejection, continue submitting to the next best journal, until acceptance. The new process by eLife would hence massively cut the workload on authors, besides editors and reviewers.

> every course would only provide a review of a student instead of grades

Would it? You'd think that a hiring manager would rather see a description of what the student accomplished, and with what level of skill, rather than a rather opaque letter grade.

Of course, that would make it tough to run "cattle car" courses with thousands of students being graded with multiple-choice tests. I don't think that would be a bad thing, either.

> Of course, that would make it tough to run "cattle car" courses with thousands of students being graded with multiple-choice tests.

It will be easier with GPT providing the evaluation.

What makes you think it actually can provide an evaluation?

Ugh stop. Not every conversation has to be about GPT.

This is an interesting development. The current publication process is quite broken, including due to the outsized importance given to the broken peer review system.

Wrt that: the NeurIPS conference did a test once, splitting the PC in two and having both review the same batch of 100 papers (and arrive independently at a decision). Roughly speaking (top of my head), the two sub-PCs agreed on the top 15% and the bottom 25%. That is: both recognised the obvious good stuff and the clear rejects.

For the remaining 60%, it basically was 50-50: one sub-PC would arrive at accept and the other at reject.

Which, to me, points out 2 things: (1) there is something very broken for about 60% of submissions... (2) but the process works as we want for the other 40%.

So I don't think accepting everything would be ideal, but I'm more than happy to try.

I'd say the experiment (relying on the summary provided here) shows what works: - Papers that are obviously poor are rejected. - Papers that are uncontroversially good are accepted.

From a CS point of view, the problem is that in many communities, what is 'uncontroversially good' depends on subtleties only known to insiders who are willing to comply with the quirks of a community. And most communities are very defensive when it comes to shielding their top venues from relative outsiders, who are not sufficiently close to the inner circle of key players. As a consequence, getting the top-venue stamp (required to please the administration) becomes a social game. I think having a more open review process and stronger editorial control (vs. reviewer power) can mitigate this problem to some extent, for the following reasons: 1. When a paper is good, but upsets the subjective feelings of reviewers, the system should be implemented in a way that there is a high social cost for killing a paper because of subjective/cosmetic issues. Having venues where papers that are sent out to reviewers are most likely reasonable and should only be killed because of 'hard' flaws creates such a system. 2. When a paper is bad and the editors push it through, publishing details of the paper's editorial flow would impose social costs on the editors. (The reviewers, who may be potentially early in their career and may need protection, may even remain anonymous, the editors, who typically enjoy a strong standing in the community, should not.)

In CS, journals tend to operate a bit closer to this model than conferences, which I think makes the journal submission and revision process more meaningful. To conclude, I don't think getting rid of rejects is the solution, but rather editorial policies that encourage reviewers to either expose fatal technical flaws in a paper or help the authors improve the presentation to achieve a better end-result.

This sounds reasonable to me as a lay person.

Humans have flaws that operate under the surface and it's good to have a system that minimizes the effect of ego on the dissemination of ideas.

> So I don't think accepting everything would be ideal, but I'm more than happy to try.

An important caveat is that this experience isn't accepting everything; it's publishing everything (or at least, everything that satisfies the desk check and is sent out for peer review). The whole point is separating evaluation from publication; they'll still distinguish works that they think are significant and rigorous from those they do not, but all works will be accessible. That's the mental shift they're trying to make happen.

(Disclosure: I volunteer for a project with a similar goal, that has received funding from eLife in the past: https://plaudit.pub.)

Looking at https://plaudit.pub and trying to understand it. It seems like a technical solution trying to solve a social problem (which I dare say is impossible). There's too much politics in academia for researchers to objectively endorse work of other research groups.

Thanks for taking a look. I think the problem you signal is already present in peer review as well, unfortunately: the politics also influences reviewer's accept/reject recommendations. With Plaudit, at least, the endorsements are public, which we hope will mitigate phenomena like review rings.

Blinded peer review at least enforces some degree of impartiality and anonymity, though obviously challenging in practice.

It tries to enforce that, but as you said, there's very little proof that that works, and strong indications that it does not: https://absolutelymaybe.plos.org/2015/05/13/weighing-up-anon...

Anonymity obviously can't be provided when reviews or endorsements are public, but partiality can at least be detected more easily.

as a long-time NeurIPS reviewer: the problem is that most submissions are kind of "meh", and everyone knows it. They're usually well-done papers (in the sense that the math checks out and the empirical evaluation is is done well enough), but it's pretty obvious that the paper isn't going to be influential. Mostly, these are papers of the "this is a cute idea that'll give you 0.1-2% improvements on some benchmarks people usually look at" variety. So depending on how strongly a reviewer believes that this might actually become influential if someone REALLY, REALLY digs down on this idea or not, you'll give it a "barely passing" or a "barely rejecting" grade. That's what most of these 60% usually are.

I don't think the problem here is actually the "broken peer review system". I think this is a fairly natural development: if your PhD advisor insists that you have a few "good papers" before they'll agree to let you graduate, and "publication venue" is used as a proxy for "is a good paper" (which is very reasonable, because it might take years before better proxies such as "citation count" give a good signal), then you'll submit your paper to top tier venues and hope for the best. I don't really know how to fix this: Establishing things like TMLR (Transactions of Machine Learning Research -- a rolling-release journal meant to have roughly the quality of NeurIPS) might be a good way forward. Students get their stamp of approval by publishing there and NeurIPS can significantly increase their acceptance threshold. But once you do that, you'll risk that TMLR doesn't count as "good enough venue" anymore....

> (1) there is something very broken for about 60% of submissions

My conclusion is that there is an inherent subjectivity involved in judging paper "quality". Which shouldn't be that surprising; if there were objective criteria we probably wouldn't need humans to do any judging. I dunno if you want to call that a broken system or not, but if this outcome is unavoidable then trying to change it won't improve anything. (I suspect that 60% isn't some magic lower limit and improvement is possible, but I suspect the lower limit is 20-40% at minimum).

Realistically, this means that a single acceptance/rejection shouldn't decide the fate of a persons whole career. And they generally don't, it will be an average of many.

I review articles for an NPG journal. I have seen articles submitted that are essentially advertisements from a company. They don't bother to read the journal's guidance to authors or anything else. I don't see how promising to publish a review alongside that is a good idea. They're just looking for free press and you're rewarding their bad behavior. No one will see the review. They'll see the glossy reprint the company hands out at their booth at the conference, with your journal's emblem on it. I'm not against companies submitting articles, far from it, actually. But how am I going to confidently review a single spaced PDF? I don't even have room to jot notes. I am against a complete lack of respect for the process.

In a case like that, my advice to the editor is not a review, it is to simply reject for failing to follow guidance to authors. Now, it may be that other journals have a better desk review process before sending out to reviewers, but I don't know.

They will only publish what they review. Desk rejects like this are still possible.

I don't expect that anyone will see this now, but the eLife Board of Directors has issued a statement in support of the quest to upend science publishing: https://elifesciences.org/inside-elife/9d38cb80/elife-s-new-...

I submitted a paper to eLife a month before they announced the change. (It was published last month.) I'm worried that 5 years from now, hiring committees will think of eLife as "not a real journal".

It seems unlikely that members of tenure/promotion and grant-review committees would have a high regard for candidates who publish in journals that accept papers that garnered negative reviews.

We already have blogs, arxiv, and similar. The main addition here seems to be that editors write something in addition to peer reviews. Where will they find the time to write these meta-reviews, and to deal with the inevitable complaints that may follow? This is asking a lot, particularly when it will be in service of a journal unlikely to have a good reputation.

We already publish in journals that ‘garner’ negative reviews; we don’t get to see the review scores. Editors already comment on papers, and we don’t see their comments. Everyone knows that some mediocre papers are accepted and some excellent papers are sandbagged behind the scenes, for less than honorable reasons. Maybe publishing the reviews and editorial comments would shed some valuable light?

Why wouldn’t tenure and grant committee members take into account high review scores in a journal that has high status and maintains an excellent review system? As long as eLife can maintain their status, maybe this could be a model for a positive change in science publishing, without making it any harder for people to get tenure, promotions, or grants?

Side note, it seems like this thinking that we need to have peer validation gate-keeping the publishing process is part of the reason for the reproducibility crisis; the reason nobody dares writing reproduction papers is because it’s so hard to get them published. Could help if journals accepted these by default?

> Side note, it seems like this thinking that we need to have peer validation gate-keeping the publishing process is part of the reason for the reproducibility crisis; the reason nobody dares writing reproduction papers is because it’s so hard to get them published. Could help if journals accepted these by default?

I think that's also where their distinction between significance and rigour comes into play. Reproductions might not be "significant", but if they're rigorous, they can still get that stamp of approval.

eLife is a treasure. eLife isn't just another "reputable brand" in publishing; they're aiming for a better publishing model. Thus, fears like

> They worried it would diminish the prestige of a brand they’d worked hard to build

are disappointing to hear. eLife's good brand is a way to get their foot in the door, to change the system from within, but the dependence on good journal brands in the academic world is the primary reason that publishers are able to extort the academic world and lock the outcomes of publicly-funded research behind paywalls.

It's hard to not become the thing you set out to fight once you're in the system, but I appreciate that eLife and Eisen are still pushing to change it from within, and I hope they succeed, and that they overcome pushback from the vested interests.

The problem of toxic reviews would be mitigated by publishing them. It’s a well known issue:

https://www.science.org/content/article/rude-paper-reviews-a...

With something like Scientific Reports having first-decision time of half a year this is inevitable.