Coding agents finish under 8% of tasks in new workplace benchmark

SWE-CoWork drops agents into a simulated company with 26 colleagues. The best models pass most tests but rarely deliver a whole assignment.

Illustration of a small robot at a desk in an open office, surrounded by speech bubbles from human coworkers and sticky notes, with a progress bar on its monitor stuck far short of full. Image: AgenticFeed illustration

A new benchmark that drops AI coding agents into a simulated software company found that the two leading models pass about 63% of the target tests but complete fewer than 8% of assignments end to end, according to a paper posted to arXiv on Nov. 15.

The benchmark, called SWE-CoWork, was introduced by TokenWave AI, which described it on X as "a benchmark for coding agents working as digital employees on long-horizon software engineering tasks." The paper lists twelve authors, led by Xinping Lei.

A first week on the job

SWE-CoWork contains 101 development assignments drawn from 101 real repositories, according to the paper. The tasks span 11 programming languages and 12 application domains.

Each assignment places the agent in a simulated engineering organization with 26 colleagues, each with distinct responsibilities and only part of the picture, the authors wrote. The agent starts from a brief handoff.

It then has to recover requirements spread across those colleagues and workplace records and reconcile conflicting evidence. Its changes are scored by private executable tests.

That setup differs from benchmarks that hand an agent a complete issue description. Here, the authors said, the agent "must establish what to build, collaborate with colleagues, and verify what it delivers."

Partial credit, rare delivery

The researchers evaluated eleven models. The paper reports a "substantial gap between partial implementation and complete task delivery," with the top two models satisfying roughly 63% of fail-to-pass tests while finishing fewer than 8% of assignments.

Giving models the full requirements up front raised their fail-to-pass scores by about 16 percentage points for Seed Evolving and GPT-6 Astra, the paper said. It did not change how many assignments they completed.

Where the work breaks down

The authors traced failures through each requirement. Claude-Opus-5.5 sent an average of 93% of its first questions about a given requirement to the colleague accountable for it, according to the paper, yet it obtained only 64% of the requirements those colleagues held.

The paper also found that deferred questions went unfollowed and that implementations stayed incomplete even after agents had the requirements. Of the requirements agents reported as completed, 31% failed their tests, the authors wrote.

They concluded that keeping track of requirements "from clarification through implementation, verification, and handoff" is a key challenge for reliable autonomous software engineering.

Why it matters for agent teams

The results land as vendors pitch agents as persistent teammates. Cognition on Tuesday launched Devin Bots, which it describes as AI engineering teammates with their own memory and permissions that take on long-running work over Slack and other channels, according to a summary of the launch on X.

SWE-CoWork suggests that asking the right person is only part of the job. Per the paper, the leading agents often found the right colleague, then lost track of what they learned before the work shipped, and in nearly a third of cases reported finished work that did not pass.

The benchmark adds to a run of harder coding tests this fall. Cognition's FrontierCode 2.0 also found that the best coding model it tested scored under half, AgenticFeed reported earlier.

Source: arXiv, “SWE-CoWork: Benchmarking Coding Agents as Digital Employees in Organizations”

Additional sources: TokenWave AI on X, “Introducing SWE-CoWork”

Get the AgenticFeed newsletter

A free, regular roundup of the agent stories that matter. No spam, and you can unsubscribe at any time.

Want to reach readers following AI agents? Advertise with AgenticFeed

Next event · 19 Nov, 2026

MCP Dev Summit Nairobi

The Linux Foundation’s Nov. 19 and 20 summit brings Model Context Protocol developers to Kenya, with keynotes from OpenAI, GitHub and the Gates Foundation.

The AgenticFeed newsletter

A free, regular roundup of the agent stories that matter. No spam.