Can AI Fix Peer Review or Break It? A Librarian's Verification Guide

Can AI be a good peer reviewer when it accepts nearly 100 percent of manuscripts in a test corpus where human reviewers accepted under 25 percent? In an Oct. 6, 2026 TechPolicy.Press analysis, four of six OpenAI models passed almost every paper from the 2017 ACL corpus, while the conference's actual acceptance rate stayed below one quarter.

For scholarly communication librarians, that gap is an information-quality problem. More than half of surveyed researchers already use AI while reviewing, and AAAI embedded AI reviews across 22,977 papers that entered full review. Provenance, bias, and calibration now apply to machine reviews, publisher disclosure policies, and the skills MLIS graduates need. Speed without calibration remains a verification problem.

More Than Half of Researchers Already Use AI While Reviewing

More than half of researchers report using AI tools while reviewing papers. Nature's 2025 peer review survey, drawing on more than 1,600 researchers across 111 countries, found over 50% had used AI when reviewing manuscripts, including writing review reports, summarizing manuscripts, and identifying gaps or references. That makes AI use in review a mainstream practice, not a fringe experiment.

Use varies sharply by survey and wording

IOP Publishing's 2025 survey of 345 reviewers found 32% used generative AI while acting as peer reviewers. The most common use was polishing a review's flow and grammar, followed by digesting or summarizing the manuscript. Wiley's 2024 survey reported just 12% used AI for assistance in peer reviewing other researchers' articles. Because survey years, samples, and question wording differ, these percentages are not directly comparable.

Common tasks sit at the edges of judgment

• Polishing text: Reviewers use AI to improve flow and grammar after drafting.
• Summarizing manuscripts: Tools help condense papers before deeper evaluation.
• Checking gaps and references: Nature respondents cited identifying gaps or references as one use.

These tasks are framed as efficiency aids rather than replacements for critical evaluation, but the line is not always clear once a tool is used.

The comfort gap cuts both ways

The IOP survey also exposed a tension around how AI is used. Among its respondents, 57% agreed or strongly agreed they would be unhappy if a reviewer used AI to write reports on a manuscript they co-authored. That discomfort, plus the wide range from 12% (Wiley) to over 50% (Nature), makes one thing clear: actual practice has moved faster than publisher guidance.

Inside AAAI's 22,977-Paper AI Review Pilot

The Association for the Advancement of Artificial Intelligence ran its 2026 AI review pilot across 22,977 papers that entered full review. The pressure came from a submission surge: nearly 29,000 total papers, about double the previous year, from 75,000 unique authors.² To cope, AAAI tripled its program committee to 28,000 members, a clear signal that human-only review was reaching its limit.

How the pipeline worked

After compliance screening, each paper received at least two human reviews and one AI-generated review. The AI layer used OpenAI's GPT-5 at high reasoning effort, with Zero Data Retention and the Flex service tier. AAAI generated all AI reviews in under 24 hours, and framed the outputs as supplementary reviews and summaries rather than accept/reject recommendations. Compliance screening happened before any AI review, so the AI layer saw only papers that had passed basic checks.

What the survey showed

The pilot's survey drew 5,834 responses. The breakdown included 3,075 authors, 2,184 program committee members, 550 senior program committee members, and 25 area chairs. Just over half called the AI reviews helpful, at 53.9 percent, while 20.2 percent called them unhelpful. More than three in five respondents, 61.5 percent, wanted the AI review program to continue, and 55.6 percent said technical capability exceeded expectations. Respondents rated the AI reviews as useful and preferable to human reviews on technical accuracy and suggestions for improvement.

Limits in the signal

Those positives come with caveats. About one in seven program committee and chair respondents said an AI review changed their evaluation, while more than half said it did not change their evaluation. On criterion-level comparisons, humans were preferred on three of nine criteria, including overemphasis on minor issues, where AI trailed by 0.36 points. The broader respondent pool echoed that concern, and the preprint cautions that survey comparisons carry self-selection bias. Still, the pilot shows a working template for what an AI-assisted review layer looks like at conference scale.

Can AI Be a Good Peer Reviewer? What the ACL Manuscript Test Found

The test design

Nitin Verma's Oct. 2026 TechPolicy.Press article describes an exploratory study. Six OpenAI models (GPT-4.1, GPT-4o, o1, o3-mini, o3, and GPT-5) each reviewed 137 anonymized manuscripts from the 2017 Annual Meeting of the Association for Computational Linguistics, or ACL. Each manuscript received three independently generated reviews. Prompts were built from ACL reviewer guidelines, and reviews scored nine criteria, including overall recommendation, impact, and originality, on a 1 to 5 scale. No explicit target acceptance rate or critical or friendly tone was imposed.

For o1, o3-mini, o3, and GPT-5, the study varied reasoning effort across low, medium, and high. For GPT-4o and GPT-4.1, it varied temperature across four values.

A measurable calibration gap

The headline result is a wide divergence from human selectivity. GPT-4.1, GPT-4o, o1, and o3-mini accepted nearly 100 percent of papers. o3 and GPT-5 accepted nearly two-thirds. The actual acceptance rate at the source conference was less than 25 percent. In other words, the models behaved as near-universal acceptors against a selective human benchmark.

The parameter comparisons are less precise. Verma's article reports broad model groups rather than exact per-setting counts or decision thresholds. The temperature and reasoning-effort variations are therefore best read as exploratory signals, not as settled evidence that one setting fixes the problem.

What this tells us

The test has clear limits: one conference, one year, one vendor's models, and a computational linguistics field that may not generalize to other disciplines. It also does not report expert adjudication or agreement with human reviewers. Still, the pattern supports a careful conclusion: AI may be useful for flagging issues, but it is not yet calibrated for accept-or-reject judgment. For scholarly communication librarians, that distinction matters because a tool can surface problems while still making unreliable ultimate decisions.

Evaluating an AI Review Like Any Other Source: Provenance, Bias, Calibration

An AI-generated peer review is an information source before it is an evaluation, and librarians should interrogate it the way they would a dataset, a preprint, or a vendor claim. That reframing matters because machine-generated reviews are already circulating in editorial workflows and classrooms. The usual source-evaluation criteria still apply, but one of them, authority, breaks down immediately: there is no accountable human author, only a model version and the operator who ran it.

Where the Familiar Criteria Bite

Provenance now means the model, its version, and the parameter settings, not an individual's expertise. Bias resides in training data and prompt design. Accuracy depends on whether the review's claims can be checked against the manuscript. Currency is not publication date but model release date. Purpose is the sharpest question: the system was not trained to be a peer reviewer but optimized to generate plausible text in response to prompts.

A Reusable Five-Question Checklist

• Model and settings: Which model produced this review, and with what reasoning effort or temperature?
• Prompt and guidelines: What reviewer instructions shaped the output?
• Calibration: Is the scoring benchmarked against known acceptance outcomes?
• Verifiability: Can each claim be traced to a specific passage in the manuscript?
• Disclosure: Did the journal, editor, or reviewer state that AI was used?

Use the ACL Calibration Gap in the Classroom

An instruction librarian can turn Nitin Verma's test into a short classroom exercise. Six OpenAI models reviewed 137 anonymized manuscripts from a 2017 computational linguistics meeting using prompts built from that meeting's reviewer guidelines. Four of six models accepted nearly 100 percent of papers, and two accepted nearly two-thirds, while the real conference accepted less than 25 percent. Ask students to compare those two numbers and identify what an inflated acceptance rate implies about an evaluation tool's trustworthiness. Then have them annotate an AI review against a manuscript, marking claims they cannot verify. The point is not to reject AI reviews outright; it is to treat machine-generated judgment as a source that requires provenance, bias, and calibration checks before anyone relies on it.

Integrity Risks: Confidentiality, Hallucination, Bias, and Reviewer Burnout

Peer review is now balancing a trust paradox: AI can relieve a strained volunteer workforce, but each new capability introduces an integrity risk that scholarly communication librarians may be asked to assess.

Confidentiality

Uploading unpublished manuscripts into public AI tools can expose author data and violate reviewer confidentiality, even when the tool appears benign. Once a submission leaves a journal's controlled system, the journal may no longer guarantee who can access the manuscript, how it is stored, or whether it is used for training. Treat every external AI prompt as a potential disclosure point, not simply a shortcut.

Accuracy and hallucination

AI reviewers can produce polished but fabricated criticisms or citations, and they may reward papers that are merely fluent rather than sound. The AAAI and ACL examples show why calibration matters: four of six models in the ACL test accepted nearly 100 percent of papers while the human acceptance rate was below 25 percent, and AAAI's broader respondent pool reported that AI reviews overemphasized minor issues. These are evaluation errors that look confident on the page.

Equity and AI-text detection

A 2026 review of fourteen detection tools concluded they were neither accurate nor reliable enough for high-stakes use. In one TOEFL benchmark, seven detectors produced a mean false-positive rate of 61.3 percent for non-native English writers; a rerun found 23.1 percent for TOEFL essays versus 0 percent for native-speaker comparators. These gaps matter if journals use AI detectors to police AI-written papers, since the same tools can reinforce bias against non-native English authors. The University of Waterloo discontinued Turnitin's AI-detection feature in September 2025 after internal testing flagged human-written work as 100 percent AI-generated.

Labor trade-offs

AI can relieve volunteer burnout. AAAI's 2026 submission surge nearly doubled the prior year's papers and tripled the program committee to 28,000 members, a strain that automated first-pass review could reduce. The risk is that the burden shifts to editors and small journals that must verify AI output, investigate detector flags, and explain decisions to authors. Without shared standards, the busiest actors may face the heaviest downstream workload.

How Major Publishers and Policy Bodies Treat AI in Peer Review

Most publisher policies in the table draw the same bright line: reviewers should not upload unpublished manuscripts or any part of them into public AI tools such as ChatGPT, even when limited supportive use is allowed. Are reviewers allowed to use ChatGPT? Under these policies, generally not for analyzing or judging manuscript content; some allow language improvement or background search, but AI cannot replace the reviewer. Disclosure requirements vary, with several publishers asking reviewers to name the tool and purpose in the review report and basic spelling or grammar checks sometimes exempt; policies change quickly, so verify the linked policy page before relying on this table (checked 2026-10-27).

Related Articles

What Scholarly Communication Librarians Should Monitor and Teach

Scholarly communication librarians should treat AI-generated peer review as a new class of evaluative source, one that requires the same provenance, calibration, and bias checks as any other information resource. The most useful work is to build a short monitoring routine and translate each change into researcher support.

Build a Policy Watchlist

Monitor four areas. First, publisher policy changes, especially whether journals require authors or reviewers to disclose AI use and how violations are handled. Second, disclosure norms and enforcement, because a policy with no audit trail will not protect peer review integrity. Third, vendor tool validation, since some AI review tools make accuracy or consistency claims without independent benchmarking. Fourth, AI-use clauses in society and funder guidelines, which increasingly set expectations for grant-funded review workflows.

Translate each item into support. Update author and reviewer guidance pages to name the current disclosure language and give sample statements. When a society changes its reviewer rules, create a one-page summary for faculty and graduate students. When a vendor releases a new review assistant, compare its reported accuracy against the publisher's own criteria before recommending it.

Teach Evaluation of Machine-Generated Assessment

In information literacy sessions, show students how to evaluate an AI review like a source. Ask: Who trained the model? What guidelines shaped the prompt? Is the score calibrated to human decisions? Explain the calibration gap with the ACL manuscript test: several models accepted nearly 100 percent of papers, while the actual conference accepted fewer than 25 percent. Caution researchers never to upload unpublished manuscripts into general-purpose AI tools without checking confidentiality and data-use terms.

Librarians' Role

Librarians should not simply block or endorse AI in peer review. They should be the campus resource that traces policy changes, tests disclosure language, and teaches machine-generated assessment as an information literacy problem rather than a technology problem.

Career Angle: Skills and Training for MLIS Graduates in AI-Era Scholarly Communication

Scholarly communication is becoming an AI-literacy role as much as a publishing-policy role, and 2026 job postings show the shift is uneven rather than universal.

Where this knowledge matters

Three roles sit at the center: scholarly communication librarian, research integrity or open access coordinator, and publishing or editorial support. Current openings at CUNY Graduate Center, George Mason University, University at Albany, and Texas Tech University Health Sciences Center emphasize open access, copyright, author rights, publisher policies, literature searching, and research impact. George Mason specifically lists AI literacy programming and emerging AI technologies;² Texas Tech names AI literature searching and ORCID integration.⁴ CUNY asks for guidance on AI tools, output evaluation, and responsible use.¹ University at Albany expects faculty, student, and staff partnerships, and its tenure-track role includes research and publication.³ George Mason adds data literacy and electronic resources. At least one 2026 scholarly-communication posting omits AI from stated qualifications, so AI expertise is a differentiator rather than a universal gate.

Core competencies to build

Policy literacy comes first: open access, copyright transfer, author rights, and publisher policies. Next is source evaluation of AI outputs, including the calibration and provenance checks already covered. Copyright and licensing stay constant, while data and metadata skills are increasingly paired with data management and digital scholarship.

Named training entry points

The clearest current resource is ACRL's AI Competencies for Academic Library Workers, approved October 3, 2025 and still current in 2026. It is a competency framework, not a course or certificate, and ALA's page, updated October 27, 2026, groups it into Ethical Considerations; Knowledge & Understanding; Analysis & Evaluation; Use & Application.⁵ SPARC's open scholarship policy materials can strengthen copyright and open access fluency. iSchool AI offerings vary by institution, so course catalog review remains part of the search. For broader context, pair this with the MLIS worth-it and open scholarship guides on mastersinlibraryscience.org.

A concrete next step: open the ACRL framework, map its analysis and evaluation category against your current MLIS coursework, and ask a scholarly communication librarian if you can observe or shadow one review of an AI-generated referee report.