OpenAI has built an internal system that hands ChatGPT performance problems to Codex agents, which work a slowdown from the first alert to a proposed fix and then confirm whether performance recovered. In a November 19 engineering post, the company said the system, called the Performance Factory, tracked more than 1,700 investigations and found the root cause in more than 600 cases in just over three months, about 17 automated investigations a day.
The post, by OpenAI technical staff members Okay Zed and Cyrus Forbes, presents the system as a pattern for running agent workflows in production. "The staggering pace of agentic development has made traditional approaches to performance engineering unsustainable," they wrote.
How the factory works
OpenAI says the system has four parts: a task tracker with configurable workflows, a dispatch loop that assigns ready work to agents, an internal API that runs Codex in the cloud, and subscriptions that wake an agent when something changes, such as a GitHub review comment or a finished deployment. Engineers define each workflow with a configuration file listing allowed states and transitions, plus Markdown prompts telling agents what to do at each step.
Each task keeps its state and findings outside any single agent's context, so work can pass between agents and engineers. VentureBeat, which received the report as an exclusive, said agents record the relevant metrics and queries rather than raw profiling data, so the next agent can rerun them to check the results.
The agents run under a service identity inside OpenAI's network, which lets the company scope their permissions and see which actions came from its performance agents, according to the post.
The first workflow, called the Regression Workflow, starts when a performance alert fires. An agent uses a shared skill called perf-forensics to find when a metric first got worse and which requests were affected, then examines deployments and feature flag changes before reading the relevant code, the post says.
Agents checking agents
OpenAI found that agents sometimes stopped at the first plausible explanation. It added an adversarial review, a separate Codex session told to disprove the diagnosis.
In one case, an agent blamed failing ChatGPT conversations on image download errors, but the reviewer noticed that the service responsible for locating files had already recovered while image requests kept failing, so it reopened the investigation. The reviewer currently sends back about 25% of diagnoses, VentureBeat reported.
In a manual review of 100 root-cause suggestions, OpenAI said it graded 70% as accurate, with the other 30% needing refinement or not fully establishing causality.
Fixes remain the bottleneck
"Getting fixes right has been harder than finding the cause," the authors wrote, adding that agent-proposed changes have sometimes been more complicated than needed or introduced new issues. OpenAI now runs a second adversarial review on each proposed fix.
Of the more than 600 investigations that reached a root cause, about 400 led to a proposed fix, according to figures OpenAI's performance team gave VentureBeat. About 50 shipped fixes were influenced by or matched agent suggestions, and about 15 were the exact pull requests Codex wrote.
An engineer still reviews each pull request and decides whether to merge it. After deployment, an agent checks whether performance returned to baseline and opens a new task if recovery is only partial. OpenAI also runs a daily agent that scans recent runs for tool failures and access errors, then opens tasks to fix recurring problems.
OpenAI said it is working on synthetic performance tests so agents can measure fixes before review, along with workflows that hunt for slowdowns when no alert fires, such as spotting autoscaling settings that keep more replicas running than a service needs. The company is exploring whether agents could deploy some low-risk fixes without human approval, VentureBeat reported. "Today, the bottleneck may be waiting for an engineer to review and merge the fix," the post says.
Source: OpenAI, Okay Zed, Cyrus Forbes, “How OpenAI's Performance Factory keeps ChatGPT fast”
Additional sources: VentureBeat, Ben Dickson, “Exclusive: OpenAI reveals Performance Factory, a Codex-powered software optimization system that keeps ChatGPT fast”
