TL;DR
OpenAI has published internal measurements of pretraining progress and argues that model perplexity could help compare progress across AI developers and inform shared safety standards. The company links lower loss to past capability gains, but says future abilities and the timing of further advances remain uncertain.
OpenAI has published internal measurements of its models’ pretraining perplexity, arguing that the metric could help AI developers compare progress and build shared safety standards. The company says the measurements offer a more durable way to track the underlying predictive performance of models as downstream benchmarks change, while acknowledging that the capabilities associated with future progress are uncertain.
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
Perplexity measures how well a model predicts the next token in a held-out dataset; lower values indicate better predictions. OpenAI says it has tracked this measure over time, including for models beginning with GPT-3.5, and is now making some internal results public. The company says it weighed the competitive cost of revealing more about its position against what it sees as the need for industry-wide safety measures.
In the post, OpenAI associates successive improvements in its reported log-perplexity with capabilities seen in its models: it gives approximately 0.99 for GPT-3.5 and approximately 0.91 for GPT-4, and says its models have since reached losses below 0.8. It links those stages to fluent conversation, productive use of chain-of-thought reasoning, and capabilities in mathematics and cybersecurity. These are OpenAI’s account of its own models; the post does not establish that a particular metric value reliably produces a particular capability.
OpenAI also says that a model at about 0.76 log-perplexity began producing results on open mathematics problems that the author describes as groundbreaking. The post projects that further changes in pretraining research could come as soon as 2027 and anticipates new capabilities in a roughly 0.68–0.7 range. Those are forecasts and expectations, not confirmed future outcomes.
Perplexity and Intelligence A shared measure?
OpenAI published internal pretraining measurements and argues they could help developers compare progress and shape shared safety standards. The numbers offer a signal, not a guarantee of future abilities.
What the metric measures
Perplexity measures how well a model predicts the next token in a held-out dataset; lower values mean better predictions.
A model learns from a large dataset.
Perplexity tracks predictive performance on held-out data.
Prompting, fine-tuning, and reinforcement learning can elicit or shape what a model does.
OpenAI’s reported progression
The company associates lower log-perplexity with capabilities observed in its models; the pattern does not prove a value causes or guarantees an ability.
| Reported level | OpenAI’s account |
|---|---|
| ≈0.99 | GPT-3.5; fluent conversation |
| ≈0.91 | GPT-4; productive chain-of-thought reasoning |
| Below 0.8 | Later models; mathematics and cybersecurity capabilities |
| ≈0.68–0.7 | Range where OpenAI anticipates new capabilities |
Why a shared measure could matter
Task benchmarks can change or saturate as models improve. Comparable pretraining measurements could give researchers and policymakers a common reference.
- Track predictive performance over time
- Compare progress if developers use shared methods
- Inform discussion of common safety standards
- What a model can reliably do
- How safely it behaves
- How much oversight development requires
What remains unresolved
The supplied material does not include the underlying chart or enough detail to independently compare the measurements.
- No standardized reporting or audit process is specified.
- No formal industry standard, timetable, or commitments from other developers are announced.
- The possible 2027 research acceleration and future capabilities are projections, not confirmed outcomes.
If developers report comparable pretraining measurements, researchers and policymakers may gain a way to track changes in models even as task-specific benchmarks become outdated or saturated. OpenAI argues that perplexity could provide a common reference point for discussing model progress and shaping shared safety standards.
The proposal matters because a measure of predictive performance could be relevant across different applications and development methods. But a common number would not, by itself, show what a model can do, how safely it behaves, or how much human oversight its development requires. The post presents perplexity as an informative measure of the model’s underlying predictive ability, not a complete assessment of those questions.
Products Worth Considering

Apple iPad 11-inch (A16), 128GB, Wi-Fi, Silver

DJI Mini 4K Drone with 4K UHD Camera

LG UltraGear 34-Inch Curved QHD Gaming Monitor, 160Hz, FreeSync Premium
As an affiliate, we earn on qualifying purchases.
From Prediction to Capabilities
OpenAI describes model development as pretraining a system on a large dataset, then using methods such as prompting, fine-tuning, and reinforcement learning to elicit or shape capabilities. It argues that pretraining progress is a useful place to look because downstream evaluations and benchmarks can change as models improve.
The post connects this view to the history of OpenAI models, describing capabilities associated with GPT-2, GPT-3, GPT-3.5, and GPT-4. It also cautions that the appearance of capabilities at particular perplexity levels was not predictable in advance. The examples are presented as a rationale for tracking the metric, rather than proof that the same pattern will continue.
OpenAI further raises the possibility that AI systems could contribute to algorithmic research on pretraining. The author says rapid advances in mathematics make a sudden acceleration in deep-learning research conceivable, perhaps by 2027, while noting that the fields differ and that this remains a projection.
Limits of the Reported Measurements
The supplied material does not include the underlying chart, full dataset details, or enough information to independently compare OpenAI’s measurements with those of other developers. It also does not specify a standardized reporting process, how evaluations would be audited, or which safety decisions should follow from particular values.
OpenAI says capabilities may emerge unexpectedly as perplexity falls, but the post does not show that a given value guarantees a specific ability. The timing and nature of future capabilities, the prospect of AI-driven advances in pretraining research, and the projected 2027 acceleration all remain uncertain.
Toward Comparable Safety Measures
OpenAI frames publication of its measurements as a first step toward a shared scientific basis for tracking progress and setting safety measures. The source does not announce a formal industry standard, a timetable for one, or commitments from other developers. The next concrete development would be whether other organizations publish comparable measurements and whether researchers agree on datasets and reporting methods that make comparisons meaningful.
Where I land
I see value in publishing a measure that could help people track pretraining progress across time, especially if task benchmarks lose usefulness as models improve. OpenAI’s release is a useful contribution to that discussion, but the measurements described here are not yet enough to support a reliable comparison across developers.
The strongest counterargument is that a simple predictive metric may offer a clearer, more stable signal than an ever-changing collection of task tests. That case would persuade me more if multiple developers reported results using shared datasets and methods, with independent scrutiny and evidence showing how the metric relates to capabilities and risks. Until then, I would treat perplexity as one informative measure, while keeping claims about future abilities and timelines explicitly uncertain.
Key Questions
What does perplexity measure?
It measures how well a model predicts the next token on a held-out dataset. Lower perplexity means better average predictions on that evaluation.
Why does OpenAI want developers to track it?
OpenAI argues that pretraining measurements may remain informative as downstream benchmarks change or become saturated. It says shared measurements could help developers compare progress and inform common safety standards.
Does a particular perplexity value guarantee a capability?
No such guarantee is established in the source. OpenAI says capabilities have appeared alongside past improvements, but also says those developments were not clear in advance.
Is an AI research acceleration by 2027 confirmed?
No. The post says a major acceleration in pretraining research could conceivably happen as soon as 2027. That is the author’s projection, not a confirmed event.
Source: OpenAI
