AI agents now pass 41% of real freelance jobs on a labor benchmark

Claude Fable 5.2 leads the Remote Labor Index from Scale AI and the Center for AI Safety, nearly doubling the next model's score on 240 paid Upwork projects.

Illustration of a friendly robot working through a tall stack of freelance job folders while a human reviewer with a clipboard checks finished folders marked with green ticks. Image: AgenticFeed illustration

AI agents can now complete about two in five real freelance projects to a standard a paying client would accept, according to the Remote Labor Index, a benchmark run by Scale AI and the Center for AI Safety. Anthropic's Claude Fable 5.2 tops the public leaderboard with an automation rate of 41.25%, close to double the 21.25% scored by Anthropic's Opus 5.5 in second place. When the index launched in October 2025, the best agent tested, Manus, scored 2.5%.

The result has been circulating widely on X this weekend, where it surfaced as a trending story on Nov. 15. Anthropic first reported the figure as 41.3% in its Fable 5.2 launch materials on Nov. 9, noting that the run used the model's "High effort" setting. The two numbers are the same score rounded differently.

What the benchmark measures

The index is built from 240 projects collected from 358 verified freelancers on Upwork, according to Scale Labs. The projects span 23 of Upwork's 64 work categories and include game development, product design, architecture, data analysis and video animation. Each one comes with the original brief, every input file the freelancer used and the deliverable that the client accepted.

Together the projects were worth $143,991, Scale Labs says. The average job took a human 28.9 hours and paid $632.60, while the median job took 11.5 hours and paid $200.

Trained reviewers grade the work by hand. They compare each AI deliverable with the human original and ask whether it "would be accepted by a reasonable client," Scale's methodology page says.

A project counts toward the automation rate only if it meets or beats the human work on that test. Three reviewers judge each project and a majority vote decides, which Scale says produces 94.4% agreement between reviewers.

How fast scores have moved

Below the leader, the board is tightly bunched. OpenAI's GPT 6 Astra scores 20.83%, Anthropic's Fable 5.1 scores 17.92% and Google's Gemini Argon 4 scores 17.08%, according to the leaderboard.

DeepSeek V4.1 Flash, a new entry, scores 9.58%. The benchmark's own site says the automation rate has doubled roughly every 3.3 months.

When the index launched, its authors reported that agents failed mostly on quality. In 45.6% of failed projects the work was judged amateur, and in 35.7% the submission was incomplete, with problems such as truncated videos or missing files.

Corrupt or wrong file formats showed up in 17.6% of failures, the authors reported. Successes clustered in audio tasks, image generation, report writing and data retrieval.

Limits on what the number means

The authors say the score is not a measure of how much remote work AI can replace. Projects that need direct contact with a client, such as customer service or tutoring, were left out.

So were jobs needing physical labor or results judged over a long period, such as search engine optimization. The 240 projects were also filtered down from 550 candidates for completeness and reproducibility, Scale says.

An analysis published by Pebblous, an AI data company, raises a separate question about scoring. Scale's page says 230 private projects are used for leaderboard evaluation, while all 23 published scores divide evenly by 240 and none by 230, Pebblous found.

Pebblous also reported that Anthropic's system card lists the Fable 5.2 result as 99 of 240 projects.

What gets scored is also an agent setup, not a bare model. In the original evaluation, agents worked in standard command-line or computer-use environments with a budget of up to $30 per task, according to Scale. They could call OpenAI models to generate images and speech and a Google model to generate video. Pebblous said that setup was rebuilt partway through the series of results, which it argued makes comparisons over time less direct.

Even the top-scoring agent fell short of the client standard on 58.75% of projects, the leaderboard shows.

Source: Scale Labs, “Remote Labor Index (RLI) leaderboard”

Additional sources: Center for AI Safety and Scale AI, “Remote Labor Index: Measuring AI Automation of Remote Work”; Anthropic, “Claude Fable 5.2 and Claude Mythos 5.2”; Pebblous, “Remote Labor Index 41.25%: Denominator Doesn't Match”

Get the AgenticFeed newsletter

A free, regular roundup of the agent stories that matter. No spam, and you can unsubscribe at any time.

Want to reach readers following AI agents? Advertise with AgenticFeed

Next event · 19 Nov, 2026

MCP Dev Summit Nairobi

The Linux Foundation’s Nov. 19 and 20 summit brings Model Context Protocol developers to Kenya, with keynotes from OpenAI, GitHub and the Gates Foundation.

The AgenticFeed newsletter

A free, regular roundup of the agent stories that matter. No spam.