PremiseCheck Efficiently Fact-Checks Human and AI Peer Reviews

Meta description: PremiseCheck flags factual errors in review comments with ReviewFact, reaching near GPT-5.5 AUROC at lower cost and latency across three fields.

Tags: PremiseCheck, ReviewFact, AI review, peer review, Natural Language Processing

Featured image: Abstract technology style, review comments and paper fragments, no real people, no company logos, no text.

Background: Why review comments need checking

The OpenLAIR PremiseCheck project is a system for checking whether review comments accurately reflect a paper. Its starting point is simple: review comments, written by humans or AI, sometimes misstate what the paper says. The project aims to flag those comments cheaply.

This matters because a wrong factual claim in a review can affect how authors, reviewers and readers understand a paper. The system is not designed to judge whether a review is fair, useful or well written. Its purpose is to point people to comments worth checking. That means the tool is positioned as a triage aid, not as a final reviewer.

Development process: From comments to premises and evidence

The method breaks a comment into testable statements. An LLM rewrites each comment as explicit premises without reading the paper. A small decision model then checks each premise against the paper. The weakest premise sets the score, and each flag names the premise the paper contradicts.

Because the score names the contradicted premise, a reader can inspect that premise rather than the whole comment. This structure gives each flag a concrete reason: the premise that fails. It also separates the step that turns a comment into premises from the step that checks those premises against the paper.

The project describes ReviewFact, a dataset of 9,267 comments on 217 papers from Natural Language Processing (NLP), machine learning and the natural sciences. The comments are labeled for factual errors under one definition, drawn from the expert annotations of ReviewCritique, ReviewScore and PeerReview Bench. Cost and latency are measured alongside accuracy, so the comparison is not only about error ranking.

Latest results: Comparable accuracy at lower cost and latency

For individual comments, PremiseCheck shows Area Under the Receiver Operating Characteristic curve, or AUROC, comparable to holistic GPT-5.5 in all three fields at about half the cost. On NLP reviews with a no-reasoning decomposer, the comparison is 63.1 versus 63.2 AUROC at ~25% of the cost and 45% of the latency.

For whole reviews, fusing batched PremiseCheck and GPT-5.5 scores gives 71.5 AUROC at 28% of the reference cost. The per-comment result compares PremiseCheck with holistic GPT-5.5, while the per-review result combines batched PremiseCheck scores with GPT-5.5 scores. The measured results include both per-comment and per-review performance, so the method is evaluated as a comment-level flagger and as a component that can be combined with a larger model.

Current phase: A check pointer, not a reviewer replacement

The project states clear limitations. The task is hard, with AUROC ≈ 63 on NLP for both systems. The ML and science subsets are small, with wide intervals, and judgment errors are out of scope. PremiseCheck is meant to point people to comments worth checking, not to replace reviewers.

The project remains focused on factual error flags, not on replacing human judgment. Its resources are public: code at https://github.com/OpenLAIR/PremiseCheck, data at https://huggingface.co/datasets/songdj/ReviewFact, and a project page at https://openlair.github.io/premisecheck/. Questions and counterexamples are welcome.