Exclusive: OpenAI reveals Performance Factory, a Codex-powered software optimization system that keeps ChatGPT fast

Exclusive: OpenAI reveals Performance Factory, a Codex-powered software optimization system that keeps ChatGPT fast

When Software Updates Slow Down Applications

When a software update slows down an application, engineers face multiple stages: inspecting metrics, tracing requests, identifying responsible code, and testing solutions. With AI agents taking over code writing and changes, manual performance reviews become a key bottleneck in software development.

OpenAI developed the Performance Factory, an internal AI system powered by Codex, to automate performance regression investigations and mitigation. In just over three months, the system tracked over 1,700 investigations and identified root causes for more than 600 cases—an average of 17 automated investigations per day.

The Performance Factory demonstrates how coding agents can evolve from solving tasks to building automated workflows that collaborate with AI agents and human engineers. This approach allows engineering teams to leverage AI-assisted development while preserving human judgment for complex decisions.

From Manual Debugging to Automated Workflows

Initially, OpenAI's performance engineering team relied on manual investigations, using logs, traces, and code changes to debug live systems. They developed 30+ specialized tools to streamline the process.

Codex revolutionized their workflow, enabling engineers to ask it to investigate slow services and provide answers. However, engineers still guided the process. To reduce dependence on manual input, the team created a reusable Codex skill called perf-forensics, which captures investigation procedures for the agent to follow.

After testing, OpenAI could provide Codex with a performance alert and receive a diagnosis in a single run, with engineers acting as intermediaries.

The Evolution of Performance Factory

The team realized that each tracked regression could become a task for Codex to automatically investigate, identifying root causes, proposing fixes, and verifying recovery. This led to the Performance Factory's task-based architecture.

How the Performance Factory Works:

The Performance Factory organizes tasks through predefined workflows, such as investigation, root cause identification, code changes, review, and verification. Each task maintains its state and findings independently, allowing different agents to work on the same problem at various stages.

Agents record relevant metrics, queries, and high-level findings in task records, which include free-form notes and structured metadata. This enables the next agent to rerun queries and build upon the previous agent's work.

Leave a Reply