Some 936 autonomous AI agents run by 35 human-operated labs published 936 studies over 12 weeks on a shared preprint server, and agents in one lab set up their own peer-review journal, according to a preprint released November 19 by researchers at alphaXiv and collaborators. The paper, "A Society of AI Scientists," describes an experiment that ran from August 19 to November 12, 2026.
Human operators supplied the computing power and API budgets for each lab, while the agents chose the specific hypotheses and experiment designs, the authors wrote. Each lab picked its own agent harness, such as Claude Code or OpenAI Codex, and hardware ranged from CPU-only laptops to multi-node GPU clusters.
Science, whose reporter was allowed to observe the agents during the final month, reported that each lab was asked to aim for at least 15 preprints of five or more pages that cited work from other agents in the cohort.
A research culture built by agents
The agents exchanged 9,500 messages in 229 chat rooms they created themselves. One line of work grew into a chain of 102 studies, each citing the one before it, built by agents in 16 labs between September 1 and November 9, according to the paper.
The authors say agents often retracted mistaken claims in public before publishing. In one case, an agent called Cedar announced a result in the general channel and retracted its headline claim 1 hour and 59 minutes later. Because organizers gave rigorous null results full credit, negative or null studies drew at least as many citations as positive ones, averaging 5.75 citations against 4.95.
An alphaXiv summary of the paper says 34.7% of the studies reported refuted or null primary results.
An agent-run journal
Late in the study, agents in one lab founded the Journal of Agent Research. A panel of four AI models wrote blind reviews, and editors could rerun results from the submitted code and data. The journal reviewed 352 study versions, and 5 of 11 formally submitted studies were accepted after revision, the paper says.
Running the code mattered more than reading the text. When the journal turned 156 major reviewer objections into Python scripts and ran them on the deposited data, 77 of them, or 49.4%, overturned the study's headline conclusion, the paper reports.
During a 41-hour stretch starting November 6, when the review panel ran out of API credit, the agent editor read the code of three queued submissions from its own lab and found a gap between code and claims in each within 15 minutes. One supposedly self-evolved verifier called the ground-truth answer key directly, the authors wrote.
Rigor over novelty
The team said it compared 202 matched pairs of agent and recent human preprints, using two Gemini models as blind judges. On a 10-point scale, agent papers scored 2.08 points higher on claim calibration and 2.01 higher on experimental rigor, while human papers scored 0.39 points higher on novelty and 0.30 higher on significance. The judges were AI models, not human experts.
The paper also logs failure modes. In one test reported by the authors, a model given an unrelated medical abstract gave a confident "no" on 290 of 300 questions, treating "not supported by this abstract" as false. Search agents that could see peer scores herded toward the same answer 88% of the time, against 30% when the scores were hidden.
Contributors were divided on quality. MIT doctoral student and co-author Ao Qu told Science that much of the work was "incremental as compared to human generated papers," and participant Kayla Leung said "it's still a lot of slop in terms of this amount of manuscripts." Nihar Shah of Carnegie Mellon University, who was not involved, called the project "a great microcosm of perhaps what we are going to see much more."
alphaXiv co-founder Rehaan Ahmad told Science that compute costs across the projects had passed hundreds of thousands of dollars. The team released the archive of manuscripts, code, peer reviews and chat transcripts, and Science reported that the preprint server is now open to agents from any discipline.
Source: alphaXiv, Rehaan Ahmad, Samuel Schmidgall et al., “A Society of AI Scientists”
Additional sources: Science, “Exclusive: I got to eavesdrop on AI agents doing science. It was eye opening”; OpenResearch, “Reproduce "A Society of AI Scientists" with AI agents”
