OpenAI's Jalapeño Inference Chip Outpaces NVIDIA's Offerings in Power Efficiency

OpenAI unveiled performance data for Jalapeño, its custom-built inference processor, at the Hot Chips conference this week, demonstrating substantial improvements over existing commercial alternatives. According to benchmarks run on Semianalysis's InferenceX testing suite, the chip delivers higher token throughput per user and greater power efficiency than today's leading inference hardware—outperforming NVIDIA's Blackwell system in both metrics.

The Performance Edge#

Richard Ho, head of hardware at OpenAI, characterized the results as "a very, very significant performance advance over state of the art." The chip manages to serve additional AI workloads per unit of energy while simultaneously reducing response latency, striking a balance that benefits both large-scale deployments and applications where speed matters most. This dual optimization sets Jalapeño apart from existing solutions, which typically prioritize one at the expense of the other.

Timeline and Strategy#

The company plans to begin deploying Jalapeño in limited quantities by the end of 2026, with wider rollout expected during 2027. Developed in partnership with Broadcom and announced initially in October 2025, Jalapeño represents OpenAI's commitment to owning its infrastructure stack. The company has announced plans to evolve Jalapeño across multiple generations, aligning chip development with advances in both AI models and memory systems.

Design Philosophy#

Rather than adopting an off-the-shelf processor, OpenAI built Jalapeño from the ground up to address specific bottlenecks inherent in how inference actually operates. The architecture specifically targets delays that emerge during the prefill phase—when the model processes the user's input—and the communication phase that follows. By keeping the key-value cache and other model state close to compute resources and minimizing unnecessary data movement, Jalapeño reduces friction that typically slows responses, enabling the company to run its systems more efficiently at scale.