Cognition, the company behind the Devin coding agent, released FrontierCode 2.0 on Oct. 27, a harder version of its benchmark for judging whether AI-written code is good enough to merge. The top model, Anthropic’s Claude Opus 5.5, scored 45.5%.
Cognition wrote in its announcement that agentic software engineering “remains far from solved.” Based on its work with Devin, it said models still struggle to infer what users want, respect constraints nobody wrote down and anticipate the long-term effects of their changes.
Merge, don’t just pass
The first FrontierCode asked a question most benchmarks skip. Instead of checking only whether code runs, it asked whether a project’s maintainer would accept the change, Cognition wrote when it introduced the test. Its criteria cover correctness, test quality, scope, style and fit with a codebase’s conventions.
More than 20 open-source developers built the tasks from repositories they maintain, spending over 40 hours per task, according to Cognition. A solution that fails any blocking criterion, the kind of problem a maintainer would treat as a hard stop in code review, scores zero.
Artificial Analysis, which runs independent model evaluations, added FrontierCode to its site earlier this year under the question of whether AI models “write code maintainers would merge.”
Version 2.0 widens the test.
Cognition says its tasks come from 55% more repositories and include security patches and speed improvements. Reference solutions change 67% more files and 89% more lines than in FrontierCode 1.1, in codebases that are 25% larger. Tasks also carry 43% more blocking criteria.
The company says it checked its own answer key more closely this time. In 80% of tasks, the reference solution now differs from the original pull request in the upstream repository, up from 60%. During one automated audit, an agent found a security flaw in an upstream project, which Cognition says it reported to the maintainers after human review.
A large share of the new criteria test what Cognition calls thoroughness, meaning things an agent should infer even when the prompt doesn’t say them. One example is fixing every copy of a bug when the prompt names only one.
Another is supporting all the input types that similar code already handles. Thoroughness blockers rose from 168 in version 1.1 to 501 in version 2.0, Cognition reports.
The company contrasts this with benchmarks such as Terminal-Bench 4.0, whose tasks spell out the approach in detail. Those are useful for measuring raw model intelligence, Cognition wrote, but users increasingly want agents that can take an informal request and work without repeated corrections.
The scores
Every model scored lower on the new version. Claude Opus 5.5 fell from 54.6% on FrontierCode 1.1 to 45.5%, and GPT-6 Astra dropped from 53.3% to 42.8%. Grok 4.7 fell from 47.6% to 28.6%, and Gemini 4 Argon fell from 47.5% to 26.3%.
Cognition’s own SWE-2 model scored 23.9%, down from 50.0%. Among open models, DeepSeek V4.1 Flash scored 19.3% and Kimi K3 scored 17.7%.
Cognition says the new version separates models more clearly by capability. On cost, it singled out GPT-6.1 Sol for strong results relative to its price, and said open-source models, “even more so for western OSS models,” have the most room to improve.
Source: Cognition, “FrontierCode 2.0”
Additional sources: Cognition, “Introducing FrontierCode”; Artificial Analysis, “Announcing FrontierCode on Artificial Analysis: can AI models write code maintainers would merge?”
