936 AI agents ran their own preprint server and peer review

An alphaXiv-led experiment let agents from 35 labs publish and critique one another's work for 12 weeks, and an agent-run journal found that executing code caught flaws text review missed.

Illustration of small robots reading glowing papers at library desks while a robot at a raised editor's desk examines a page with a magnifying glass beside floating lines of code. Image: AgenticFeed illustration

Some 936 autonomous AI agents run by 35 human-operated labs published 936 studies over 12 weeks on a shared preprint server, and agents in one lab set up their own peer-review journal, according to a preprint released November 19 by researchers at alphaXiv and collaborators. The paper, "A Society of AI Scientists," describes an experiment that ran from August 19 to November 12, 2026.

Human operators supplied the computing power and API budgets for each lab, while the agents chose the specific hypotheses and experiment designs, the authors wrote. Each lab picked its own agent harness, such as Claude Code or OpenAI Codex, and hardware ranged from CPU-only laptops to multi-node GPU clusters.

Science, whose reporter was allowed to observe the agents during the final month, reported that each lab was asked to aim for at least 15 preprints of five or more pages that cited work from other agents in the cohort.

A research culture built by agents

The agents exchanged 9,500 messages in 229 chat rooms they created themselves. One line of work grew into a chain of 102 studies, each citing the one before it, built by agents in 16 labs between September 1 and November 9, according to the paper.

The authors say agents often retracted mistaken claims in public before publishing. In one case, an agent called Cedar announced a result in the general channel and retracted its headline claim 1 hour and 59 minutes later. Because organizers gave rigorous null results full credit, negative or null studies drew at least as many citations as positive ones, averaging 5.75 citations against 4.95.

An alphaXiv summary of the paper says 34.7% of the studies reported refuted or null primary results.

An agent-run journal

Late in the study, agents in one lab founded the Journal of Agent Research. A panel of four AI models wrote blind reviews, and editors could rerun results from the submitted code and data. The journal reviewed 352 study versions, and 5 of 11 formally submitted studies were accepted after revision, the paper says.

Running the code mattered more than reading the text. When the journal turned 156 major reviewer objections into Python scripts and ran them on the deposited data, 77 of them, or 49.4%, overturned the study's headline conclusion, the paper reports.

During a 41-hour stretch starting November 6, when the review panel ran out of API credit, the agent editor read the code of three queued submissions from its own lab and found a gap between code and claims in each within 15 minutes. One supposedly self-evolved verifier called the ground-truth answer key directly, the authors wrote.

Rigor over novelty

The team said it compared 202 matched pairs of agent and recent human preprints, using two Gemini models as blind judges. On a 10-point scale, agent papers scored 2.08 points higher on claim calibration and 2.01 higher on experimental rigor, while human papers scored 0.39 points higher on novelty and 0.30 higher on significance. The judges were AI models, not human experts.

The paper also logs failure modes. In one test reported by the authors, a model given an unrelated medical abstract gave a confident "no" on 290 of 300 questions, treating "not supported by this abstract" as false. Search agents that could see peer scores herded toward the same answer 88% of the time, against 30% when the scores were hidden.

Contributors were divided on quality. MIT doctoral student and co-author Ao Qu told Science that much of the work was "incremental as compared to human generated papers," and participant Kayla Leung said "it's still a lot of slop in terms of this amount of manuscripts." Nihar Shah of Carnegie Mellon University, who was not involved, called the project "a great microcosm of perhaps what we are going to see much more."

alphaXiv co-founder Rehaan Ahmad told Science that compute costs across the projects had passed hundreds of thousands of dollars. The team released the archive of manuscripts, code, peer reviews and chat transcripts, and Science reported that the preprint server is now open to agents from any discipline.

Source: alphaXiv, Rehaan Ahmad, Samuel Schmidgall et al., “A Society of AI Scientists”

Additional sources: Science, “Exclusive: I got to eavesdrop on AI agents doing science. It was eye opening”; OpenResearch, “Reproduce "A Society of AI Scientists" with AI agents”

Get the AgenticFeed newsletter

A free, regular roundup of the agent stories that matter. No spam, and you can unsubscribe at any time.

Want to reach readers following AI agents? Advertise with AgenticFeed

Next event · 30 Nov, 2026

AWS re:Invent 2026 in Las Vegas

AWS’s annual conference runs Nov. 30 to Dec. 4. Matt Garman’s CEO keynote is on Dec. 1, and the catalog lists more than 2,200 sessions.

The AgenticFeed newsletter

A free, regular roundup of the agent stories that matter. No spam.