AI
OpenAI's Chief Scientist Proposes Perplexity as a Way to Measure and Pace AI Progress
Jakub Pachocki suggests a technical metric could serve as a shared benchmark for controlling the speed of AI development across the industry.
TechCrunch — AI
OpenAI's chief scientist is advocating for a standardized measurement to help the industry coordinate on controlling how fast AI systems advance. In a recently published post, Jakub Pachocki argues that perplexity — a technical gauge of how accurately a model can predict the next piece of information — could become a shared metric for tracking and limiting AI development speed.
The Case for Perplexity as a Benchmark#
Pachocki contends that traditional performance benchmarks quickly become irrelevant as AI capabilities improve. By contrast, perplexity remains a stable indicator of model quality: lower values signal better predictive power. The metric has tracked OpenAI's model progression from GPT-3.5 through GPT-4 and into more recent reasoning systems, with significant capability jumps occurring at certain perplexity thresholds.
According to Pachocki's analysis, when perplexity dropped below 0.8, researchers began observing transformative breakthroughs in areas like cybersecurity and mathematics. This pattern suggests the metric could function as an objective measure of when AI systems cross into genuinely new capability territory.
A Framework for Coordinated Restraint#
The broader context for this proposal stems from heightened concern within the AI community about unconstrained development. Industry figures have warned of existential risks, and Anthropic's CEO recently published a framework to "pace the frontier" of AI research — an idea that gained support from OpenAI's Sam Altman and Elon Musk. However, meaningful coordination requires a common language for measuring progress.
Pachocki suggests that if companies adopted perplexity as a unified benchmark for general intelligence, they could then commit to voluntary caps: agreeing not to exceed certain perplexity thresholds until safety monitoring and behavioral understanding reach specific, measurable milestones. This would create a mechanism for collaborative self-regulation without needing top-down enforcement.

