Data notebook company Deepnote has released the Agent Initiative Benchmark, a test of whether AI agents know when to speak up at work without being asked. In results covering 12 frontier models, OpenAI's GPT-6 Astra scored highest at 8.97 out of 10, and almost every model struggled most with knowing when to stay silent, Deepnote said.
"An AI teammate should know when to help without being asked," the company wrote on its research page. Deepnote said it builds a workspace where agents and people work on data together, which is why it wanted the measurement.
A fake company with six years of records
Real company data is too sensitive to benchmark on in public, so Deepnote built one. PageMantle is a synthetic website-builder business that started with a seed check in 2020 and grew to 71 employees and 100 enterprise customers by 2026, according to the benchmark page.
Its records span 11 services, including Slack, Linear, Notion, HubSpot, Stripe, PostHog, BigQuery and Snowflake, for about 10 GB of data. Deepnote has published the dataset on Hugging Face and the simulated APIs and sample tasks on GitHub.
Each run drops an agent into the company Slack as a conversation starts. The agent can investigate across the connected services, then decide whether to post and where the message belongs.
If the problem is already handled, the correct answer is silence. Deepnote gave the example of sales promising a Friday launch in one channel while engineering moves the release a week in another.
According to Deepnote, there are 50 situations across eight task types, from finding facts to checking whether an action follows company rules. Every model ran with identical instructions and tools under a 180-second budget at high reasoning effort. Two LLM evaluators scored each run from 0 to 10.
Helping is easy, holding back is not
When help was needed, a message went out nine times out of ten, Deepnote reported. When silence was right, models held back only six times out of ten. For the same model, "speaking out of turn cost about 1.8× as much as staying quiet," the company wrote.
GPT-6.1 Sol placed second at 8.65 and Anthropic's Claude Fable 5.2 third at 8.17, according to the results. Claude Opus 5.5 scored 7.84 and answered fastest among the leaders, at about 29 seconds. Grok 4.7 finished last at 4.51, behind Gemini 4 Argon at 5.60.
Price did not track quality. Sol cost $0.112 per task, Deepnote said, against $0.693 for Astra and $1.87 for Fable 5.2.
Deepnote said the three most expensive models finished third through fifth. It estimated a month of watching one company's Slack at $281 on Sol and $4,684 on Fable 5.2. Per-task costs across the field ranged from $0.007 for GPT-6 Luna to $3.02 for Claude Fable 5.1, a 430-fold spread.
More searching also did not help. Gemini 4 Argon averaged about 35 tool calls per task and Grok 4.7 about 36, the most in the field. Those two hit the 180-second limit without sending anything on 27 and 18 of their 50 runs, respectively, according to the results.
Advice for agent builders
Deepnote recommended faster models for live threads where someone is waiting, and slower, more thorough ones for digests and monitors. It also told builders to cap an agent's search and "force a decision," either a message or silence.
The company listed limits on its own work. It said the panel has only 50 tasks and every model ran at a single effort level.
The test also compares models rather than harnesses, and Deepnote said a different prompt or tool policy "will move every number here." It said it is now building a harness benchmark and plans a larger task set in the next version.
Source: Deepnote Research, “Agent Initiative Benchmark”
Additional sources: GitHub, “deepnote/agent-initiative-benchmark”; Deepnote, “Research notes from agent developers”
