Researchers led by Zhengxi Li and Junyi Zhou posted SkillForgeBench to arXiv on Oct. 10, a benchmark of 842 tasks that measures whether AI agents meet every requirement of a job. The best agent they tested, built on Claude Opus 4.8, fully completed 40.9% of them.
The nine authors list affiliations including MBZUAI, City University of Hong Kong, Tencent’s WeChat AI and Mila in Quebec. The first two authors contributed equally, according to the paper.
The paper argues that a reliable agent has to satisfy every operational constraint, such as required steps and formatting rules, “rather than produce superficially plausible deliverables.” Every missed constraint forces a person to step in, the authors write.
They describe a trade-off in existing tests. Benchmarks written by experts have strict checks but are costly to author and cover narrow domains, while crowdsourced ones cover more ground with weaker grading. SkillForgeBench is meant to get both breadth and depth.
How the tasks are built
The benchmark uses agent skills as blueprints. A skill is a packaged set of instructions, written by people, that tells an agent how to do a particular kind of work. The authors say skills already capture expert workflows and professional conventions, which makes them a useful base for test design.
For each skill, the team searched the open web for real files and real user requests, then turned them into task instructions packaged in sandboxes, which are isolated computer environments. The skills behind the final tasks span five domains: technology, business, creative work, harness behavior and knowledge work.
Each task is graded against two kinds of rubric. Completion rubrics check the final result, and process rubrics check how the agent got there.
To check the automated pipeline, two expert reviewers rated a sample of 100 tasks and more than 1,600 rubric items. The paper reports 95.9% agreement overall between the humans’ judgments.
What the scores show
The authors ran eight models through the same agent framework, OpenHands, so differences would come from the models rather than the software around them. Besides Opus 4.8, the field included Claude Sonnet 4.6, Claude Haiku 4.5, Gemini 3.6 Flash, GLM-5.2, Kimi K2.6, Qwen3.5-397B and Tencent’s Hunyuan-3.
Measured one rubric item at a time, the agents looked capable, with pass rates from 70.0% to 85.7%. When every requirement had to be met, strict completion fell to between 12.6% and 40.9%.
Giving agents the relevant skill helped the strongest models most. Opus 4.8 improved by 19.2 points with skill guidance, compared with 5.8 points for Haiku 4.5, the paper reports. The authors say skill grounding “widens rather than narrows” the gap between top and bottom models.
Agents also met process rubrics far less often than completion rubrics. For Haiku 4.5, the strict process rate was under 3%.
The authors conclude that completion-only checks “overestimate real-world agent reliability.” The paper runs 38 pages, including appendices with the prompt templates used to build the benchmark.
Source: arXiv, Zhengxi Li et al., “SkillForgeBench: Evaluating Agent Reliability Grounded in Agent Skill Specifications”
Additional sources: arXiv (full paper), Zhengxi Li, Junyi Zhou et al., “SkillForgeBench: Evaluating Agent Reliability Grounded in Agent Skill Specifications (HTML full text)”
