With thanks to MV, YC, DB, WM, MH, and to Claude (Anthropic) and ChatGPT (OpenAI)
The problem: more papers than reviewers, and an AI echo chamber
Conference submissions are growing faster than the pool of qualified reviewers. Every program chair now faces the same arithmetic: more papers to evaluate, the same number of experts to evaluate them, and fewer hours of careful human attention per paper. Our review process has to spend that attention where it changes the outcome.
Generative AI makes the shortage worse while offering a tempting fix. Authors use AI to produce and submit more papers. Reviewers, overloaded, use AI to review them. If both sides lean on the same models, we converge on an equilibrium in which papers are acceptable but mediocre, because the models reward what resembles the corpus they were trained on, and novelty is limited by what has already been published.
Some conferences and journals respond by screening for and penalizing papers that appear to use generative AI. We disagree. AI detectors are unreliable, their false positives fall hardest on authors writing in a second language, and provenance is not what we care about. Quality is. The question for our community is how to use AI and people together so that we publish better papers, not more papers.
Our design principle is simple. AI focuses on decisions that have a checkable right answer: is the citation real, does the proof hold, is the dataset described. Humans handle judgment calls: is this interesting, does it matter. And a set of exceptions guarantees that a paper no AI model would rate highly can still reach a human reader.
A three-layer cascade
- AI desk-reject screening. AI checks every submission against a fixed list of objective failures and rejects the clear cases.
- Multi-agent AI ranking. Several independent AI agents evaluate, score, and rank the papers that survive desk-reject screening. The top-ranked papers advance to the last phase.
- Human pairwise evaluation. Human attention is deliberately lightweight: Reviewers rank papers in pairs within their topic area. The comparisons are combined statistically into a final ranking using the Bradley-Terry algorithm.
More detail on each layer is described below.
Layer 1: AI desk-reject screening
Desk-reject screening checks a paper against a fixed, published list of objective failures. Each check is one a reader could verify in minutes: a cited paper that does not exist; a claim that the code or data is available when the link is dead; results tables whose numbers do not match the text; a proof with a gap at a specific step; text copied from a prior paper; a submission outside the conference’s scope.
The screener does not judge quality. It is not asked “is this a good paper” but “does this paper fail any of these checks,” and for each failure it must quote the specific evidence. Because every rejection cites checkable evidence, every rejection can be appealed with a counterexample, and a human decides the appeal.
Layer 2: Multi-agent AI ranking
Papers that pass desk-reject screening are evaluated by several AI agents built on different model families. Each agent scores and ranks the papers in a topic area independently. The ranks are combined, and the top papers in each area advance to human review. Disagreement among the agents is recorded, because it is a useful signal later (see the novelty exception).
Layer 3: Human pairwise evaluation
Human reviewers today assign absolute scores, such as 1 to 5. Those scores are known to be unreliable: the same paper scored by two committees lands on opposite sides of the acceptance line roughly half the time, and reviewers differ by a point or more in how harsh they are. Absolute scores also demand something a reviewer cannot do well, which is to hold the whole field’s distribution in their head and place one paper in it.
Comparison is easier and more consistent. Given two papers in the same area, a reviewer can say which makes the larger contribution far more reliably than they can say what either is worth in isolation. Reviewer harshness cancels out, since a harsh reviewer and a lenient one will usually agree on which of two papers is stronger even if they disagree on both scores. The individual comparisons are combined statistically, using the Bradley–Terry model.
Acceptance is already a comparison. A conference accepts a fixed fraction of what it receives, so every acceptance decision is implicitly “this paper over that one.” Pairwise evaluation makes that comparison explicit, and asks it directly of a person best placed to answer.
Isn’t this apples to oranges? Comparisons are only asked within a topic area, assigned by the same area-chair process that assigns reviewers today. A reviewer is never asked to compare a market-microstructure paper with a fraud-detection paper. Within an area, the question is the one an area chair already answers when deciding which borderline paper to take: which of these two advances the field more? Where a paper truly has no peers, which is rare, the area chair places it directly.
Reviewers are also not asked to compare two whole papers from scratch. Each reviewer receives a small batch of four or five papers in their area, orders them, and writes one sentence on each adjacent pair explaining the ordering. That is less work than writing four or five independent reviews, it yields more comparisons per hour of reading, and the sentences give every author feedback.
Exceptions and protections for authors
The cascade spends human attention on the papers the AI layers rank highest. The exceptions exist for the papers the AI layers would get wrong: unconventional work, work that departs from the published literature, and work rejected on a mistaken premise. Each one routes a paper back to a human.
First-author exception. Most authors submit one paper, and for them this exception means their paper goes directly to human ranking, bypassing the AI ranking entirely. Authors who submit several papers must choose one to protect. That choice is the point: it asks authors to signal which paper they stand behind, and it caps the number of papers that bypass the AI stage at one per author rather than one per submission. An author who believes a paper is unconventional, and therefore likely to be underrated by models trained on the existing literature, has a guaranteed route to a human reader for it.
Desk-reject appeals. Every desk rejection must state an explicit, falsifiable reason, such as a specific hallucinated citation. Authors can appeal only with a specific counterexample, which a human reviewer evaluates. A denied appeal returns the paper to desk rejection; an upheld appeal sends it on to the AI ranking.
Novelty exception. Papers that the AI ranking placed below the cut are given a second look for novelty, because models trained on the published literature will tend to underrate work that departs from it. Two signals are used. First, disagreement among the AI reviewers: when independent models give the same paper very different ranks, that is evidence the paper is hard to evaluate, which is exactly where human judgment adds the most. Second, distance from the literature: the paper’s embedding is compared with published work in the field, and papers far from any cluster are flagged. Neither signal alone is reliable, since an incoherent paper is also far from everything, so a paper must show both. The “top” flagged papers then proceed. The test’s purpose is to guarantee that unusual work reaches a human, not to reward unusualness for its own sake.
What this costs in human time
Suppose 1,500 submissions. Desk-reject screening removes 20%, leaving 1,200. The AI ranking advances the top third in each area, 400 papers, and the exceptions add perhaps 100 more, for 500 papers reaching human review. Each is read by three reviewers in batches of five, so 1,500 reads, or 300 reviewer batches: one batch each for a program committee of 300, or two each for a committee of 150. Under today’s process, 1,500 submissions with three reviews each would be 4,500 reviews. The human load falls by about two thirds, and every one of those reads is a comparison between papers that have already cleared the objective bar.
What authors get back
Every author receives: the result of each screening check, with the evidence if any failed; their paper’s AI rank band and the agents’ agreement or disagreement; and, if the paper reached human review, the sentences reviewers wrote comparing it with its neighbours. That is more specific feedback than most rejected papers receive today.
Open questions
- Which model families should serve as the AI reviewers, and how do we keep authors from tuning papers to them?
- How should the AI layers be validated before they decide anything?
- Which models may see unpublished manuscripts, and under what confidentiality terms?
