A team at ETH Zurich posted HPCAgent-Bench to arXiv on Oct. 14, a benchmark of 689 code kernels for testing AI agents that rewrite code to run faster. Their headline result is about cost. Across 40 matched setups, adding something like a new tool or skill significantly changed speed only twice, but it changed token cost by up to 3.0 times.
Yakup Koray Budanaz is the lead author, and Torsten Hoefler, who leads ETH’s Scalable Parallel Computing Lab, is the senior author. One co-author is from Singapore’s A*STAR Institute of High Performance Computing. A kernel, in this context, is a small, performance-critical piece of code, such as a loop that does most of a program’s math.
The authors argue that current benchmarks are too narrow to show what each part of an agent setup contributes. Most fix a single chip, programming language or agent harness, the software that runs the model as an agent. They cite an earlier re-evaluation that found only 6.11% of implementations labeled as faster were significantly faster when timed over 30 runs.
How it’s built
The kernels come from loop-level optimization challenges, real high-performance computing codes and machine learning models. The HPC sources include widely used scientific packages such as Quantum ESPRESSO and CP2K.
Each kernel is written once in NumPy, a Python math library, and the framework derives C and Fortran reference versions from it. Agents can submit code through a shared C interface or Python and run it on CPUs or GPUs, on one machine or several.
An outside judge checks each answer on hidden, randomly generated inputs and times it. The authors say this closes off ways agents have gamed earlier tests, such as overfitting to visible inputs.
What the agents did
The experiments used open-weight models including Qwen3.8-27B and Kimi-K2.7, with Claude Code as the default harness. Agents reached up to 4.2 times the speed of automatic parallelizers, which are compilers that spread work across processor cores.
The optimizations mostly held up on different hardware. When the team moved answers from an AMD MI300A chip to an Nvidia GH200, between 93% and 99% of those without device-specific code still ran correctly. The ranking of per-kernel speedups also stayed consistent, with a rank correlation of at least 0.84.
The paper introduces a metric called intervention efficacy, which compares paired runs with and without an added tool on speed, success rate and cost. A second metric, weighted token cost, reports input, cached-input and output tokens under stated weights.
The authors found that the choice of weights can decide whether a token saving looks significant. For that reason, they publish the weights they used and release every token count, and they propose this as a standard for reporting cost.
The paper also notes that repeated runs of the same agent, model and prompt return different results. It reports geometric means with confidence intervals for that reason.
Source: arXiv, Yakup Koray Budanaz et al., “HPCAgent-Bench: A Framework for Sound Benchmarking of Code-Optimizing Agents, Harnesses, and Skills”
Additional sources: arXiv (full paper), Yakup Koray Budanaz et al., “HPCAgent-Bench: A Framework for Sound Benchmarking of Code-Optimizing Agents, Harnesses, and Skills (HTML full text)”
