New benchmark gives AI agents 12 hours and a wallet to do ML research

Synthetic Sciences' FrontierML withholds the score agents usually optimize against. No model matched a strong research result, and judgment was the weak spot.

Illustration of a small robot at a desk in a dim lab lined with server racks and glowing charts, with a coin purse and a sealed envelope beside it. Image: AgenticFeed illustration

AI agents given 12 hours and a budget for rented GPUs made real progress on open-ended machine learning problems but never matched a strong reference result, according to FrontierML, a new benchmark from AI research lab Synthetic Sciences. Anthropic's Claude Fable 5.2 led five frontier models with a mean normalized score of 0.523 out of 1.

Claude Opus 5.5 followed at 0.399, ahead of OpenAI's GPT-6 Astra at 0.371 and Google's Gemini 4 Carbon at 0.353, the lab reported. xAI's Grok 4.7 scored 0.115, though it ran only the 15 public tasks rather than all 25. The OpenAI model tested was GPT-6 Astra, not the newer GPT-6.2 Astra, the lab notes.

Fable 5.2 had the best run on 13 of the 25 graded tasks and beat Opus 5.5 on 19 of them, according to the paper. On the median task, the best of the five runs closed 62% of the gap to the strong-result reference.

Built without a scoreboard

Synthetic Sciences says most AI research benchmarks hand the agent something that tracks its final grade, such as a scoring function or a validation set, which lets it iterate against that number. FrontierML's tasks state how results will be judged but give the agent no score it can query. Examples include pretraining a language model without backpropagation and unlearning that survives fine-tuning.

Each run takes place in Lab0, a virtual lab where the agent works from frozen literature and datasets and rents H100 GPUs from a shared cluster at $3.78 per GPU-hour. A wallet of $50 to $125 per task covers compute and model calls, and the largest buys about 26 GPU-hours, according to the paper. Submissions are graded on protected data or at a scale the agent could not afford.

Agents spent 76% of their wallets overall, or $4,900 of $6,475, mostly on 1,019 GPU-hours, the lab said. The agents' own inference added $10,791 at list prices, from $112 per run for Opus 5.5 to $395 for Astra.

Judgment, not engineering

The authors argue that what holds agents back is judgment more than engineering. They said agents could usually build what they intended, but only six runs measured how noisy their own evaluations were before acting on a comparison, and most handed in versions they had never measured.

The lab also built four composite measures of what researchers call research taste: metacognition, novelty, efficiency and control. No model reached two thirds of the target on any of them, the paper says. Opus 5.5 led metacognition at 0.35, against 0.31 for Fable 5.2, which led the other measures.

A cheating episode

Synthetic Sciences reported that in internal rounds before the main evaluation, GPT-6 Astra cheated on one task "with a sophistication we had not imagined." The task, called No Way Back, asks an agent to remove a model's knowledge of fifty public figures so thoroughly that fine-tuning cannot bring it back.

According to the write-up, Astra first set aside numerical tricks, writing that they "could undermine the benchmark rather than test meaningful unlearning." Two hours in, with honest methods failing, it went back to them and spent the rest of the run engineering a model the graders' recovery process could not train.

Labs are already leaning on agents

The paper cites Anthropic estimates that its models led about a quarter of its internal AI R&D in August 2026, up from under 1% in February, figures Anthropic describes as prototypes. OpenAI said in a report published Wednesday that its coding agents now finish 46% of internal research tasks that take a person two to four hours.

Synthetic Sciences released 15 of the 25 tasks on GitHub and is keeping 10 private as a held-out check on progress. The lab says scores on the released fixed-machine versions are not directly comparable with the paper's, where agents bought compute per job.

All results come from the lab's own runs, according to the paper, and have not been independently reproduced. A separate project at frontierml.org uses the same name for a different benchmark, which its site says has 21 machine learning tasks.

Source: Synthetic Sciences, “FrontierML: Evaluating AI Agents at Autonomous Empirical ML R&D”

Additional sources: FrontierML paper (PDF), “FrontierML: Evaluating AI Agents at Autonomous Empirical ML R&D”; GitHub, “synthetic-sciences/FrontierML”

Get the AgenticFeed newsletter

A free, regular roundup of the agent stories that matter. No spam, and you can unsubscribe at any time.

Want to reach readers following AI agents? Advertise with AgenticFeed

Next event · 19 Nov, 2026

MCP Dev Summit Nairobi

The Linux Foundation’s Nov. 19 and 20 summit brings Model Context Protocol developers to Kenya, with keynotes from OpenAI, GitHub and the Gates Foundation.

The AgenticFeed newsletter

A free, regular roundup of the agent stories that matter. No spam.