AIThis post was created with the assistance of artificial intelligence (AI).

TL;DR

OpenAI has published internal measurements of pretraining progress and argues that model perplexity could help compare progress across AI developers and inform shared safety standards. The company links lower loss to past capability gains, but says future abilities and the timing of further advances remain uncertain.

OpenAI has published internal measurements of its models’ pretraining perplexity, arguing that the metric could help AI developers compare progress and build shared safety standards. The company says the measurements offer a more durable way to track the underlying predictive performance of models as downstream benchmarks change, while acknowledging that the capabilities associated with future progress are uncertain.

Buying for a business?Offer from Amazon

Get business pricing on tech for your team

  • Business-only prices and quantity discounts
  • Tax-exempt purchasing
  • Multiple users, one account, clear invoices
As an affiliate, we earn on qualifying purchases.

Perplexity measures how well a model predicts the next token in a held-out dataset; lower values indicate better predictions. OpenAI says it has tracked this measure over time, including for models beginning with GPT-3.5, and is now making some internal results public. The company says it weighed the competitive cost of revealing more about its position against what it sees as the need for industry-wide safety measures.

In the post, OpenAI associates successive improvements in its reported log-perplexity with capabilities seen in its models: it gives approximately 0.99 for GPT-3.5 and approximately 0.91 for GPT-4, and says its models have since reached losses below 0.8. It links those stages to fluent conversation, productive use of chain-of-thought reasoning, and capabilities in mathematics and cybersecurity. These are OpenAI’s account of its own models; the post does not establish that a particular metric value reliably produces a particular capability.

OpenAI also says that a model at about 0.76 log-perplexity began producing results on open mathematics problems that the author describes as groundbreaking. The post projects that further changes in pretraining research could come as soon as 2027 and anticipates new capabilities in a roughly 0.68–0.7 range. Those are forecasts and expectations, not confirmed future outcomes.

At a glance
announcementWhen: Published in an OpenAI post; the source…
The developmentOpenAI has released internal pretraining perplexity measurements and called for the metric to help establish a shared scientific basis for AI progress and safety standards.
AI progress · Briefing

Perplexity and Intelligence A shared measure?

OpenAI published internal pretraining measurements and argues they could help developers compare progress and shape shared safety standards. The numbers offer a signal, not a guarantee of future abilities.

≈0.76
Log-perplexity linked to reported open math results
≈0.99
Reported for GPT-3.5
≈0.91
Reported for GPT-4
2027
Possible timing of faster research progress

What the metric measures

Perplexity measures how well a model predicts the next token in a held-out dataset; lower values mean better predictions.

1
Pretrain

A model learns from a large dataset.

2
Measure

Perplexity tracks predictive performance on held-out data.

3
Shape capabilities

Prompting, fine-tuning, and reinforcement learning can elicit or shape what a model does.

OpenAI’s reported progression

The company associates lower log-perplexity with capabilities observed in its models; the pattern does not prove a value causes or guarantees an ability.

Reported levelOpenAI’s account
≈0.99GPT-3.5; fluent conversation
≈0.91GPT-4; productive chain-of-thought reasoning
Below 0.8Later models; mathematics and cybersecurity capabilities
≈0.68–0.7Range where OpenAI anticipates new capabilities

Why a shared measure could matter

Task benchmarks can change or saturate as models improve. Comparable pretraining measurements could give researchers and policymakers a common reference.

Potential value
  • Track predictive performance over time
  • Compare progress if developers use shared methods
  • Inform discussion of common safety standards
What it cannot show alone
  • What a model can reliably do
  • How safely it behaves
  • How much oversight development requires

What remains unresolved

The supplied material does not include the underlying chart or enough detail to independently compare the measurements.

  • No standardized reporting or audit process is specified.
  • No formal industry standard, timetable, or commitments from other developers are announced.
  • The possible 2027 research acceleration and future capabilities are projections, not confirmed outcomes.
WatchMeaningful comparisons depend on shared datasets, reporting methods, independent scrutiny, and evidence connecting the metric to capabilities and risks.
The takeaway
Perplexity may be a useful signal of pretraining progress, but it cannot yet support reliable cross-developer comparisons or settle questions about safety and future abilities.

A Shared Measure for AI Progress

If developers report comparable pretraining measurements, researchers and policymakers may gain a way to track changes in models even as task-specific benchmarks become outdated or saturated. OpenAI argues that perplexity could provide a common reference point for discussing model progress and shaping shared safety standards.

The proposal matters because a measure of predictive performance could be relevant across different applications and development methods. But a common number would not, by itself, show what a model can do, how safely it behaves, or how much human oversight its development requires. The post presents perplexity as an informative measure of the model’s underlying predictive ability, not a complete assessment of those questions.

Products Worth Considering

LG UltraGear 34-Inch Curved QHD Gaming Monitor, 160Hz, FreeSync Premium

LG UltraGear 34-Inch Curved QHD Gaming Monitor, 160Hz, FreeSync Premium

As an affiliate, we earn on qualifying purchases.

From Prediction to Capabilities

OpenAI describes model development as pretraining a system on a large dataset, then using methods such as prompting, fine-tuning, and reinforcement learning to elicit or shape capabilities. It argues that pretraining progress is a useful place to look because downstream evaluations and benchmarks can change as models improve.

The post connects this view to the history of OpenAI models, describing capabilities associated with GPT-2, GPT-3, GPT-3.5, and GPT-4. It also cautions that the appearance of capabilities at particular perplexity levels was not predictable in advance. The examples are presented as a rationale for tracking the metric, rather than proof that the same pattern will continue.

OpenAI further raises the possibility that AI systems could contribute to algorithmic research on pretraining. The author says rapid advances in mathematics make a sudden acceleration in deep-learning research conceivable, perhaps by 2027, while noting that the fields differ and that this remains a projection.

Limits of the Reported Measurements

The supplied material does not include the underlying chart, full dataset details, or enough information to independently compare OpenAI’s measurements with those of other developers. It also does not specify a standardized reporting process, how evaluations would be audited, or which safety decisions should follow from particular values.

OpenAI says capabilities may emerge unexpectedly as perplexity falls, but the post does not show that a given value guarantees a specific ability. The timing and nature of future capabilities, the prospect of AI-driven advances in pretraining research, and the projected 2027 acceleration all remain uncertain.

Toward Comparable Safety Measures

OpenAI frames publication of its measurements as a first step toward a shared scientific basis for tracking progress and setting safety measures. The source does not announce a formal industry standard, a timetable for one, or commitments from other developers. The next concrete development would be whether other organizations publish comparable measurements and whether researchers agree on datasets and reporting methods that make comparisons meaningful.

Where I land

I see value in publishing a measure that could help people track pretraining progress across time, especially if task benchmarks lose usefulness as models improve. OpenAI’s release is a useful contribution to that discussion, but the measurements described here are not yet enough to support a reliable comparison across developers.

The strongest counterargument is that a simple predictive metric may offer a clearer, more stable signal than an ever-changing collection of task tests. That case would persuade me more if multiple developers reported results using shared datasets and methods, with independent scrutiny and evidence showing how the metric relates to capabilities and risks. Until then, I would treat perplexity as one informative measure, while keeping claims about future abilities and timelines explicitly uncertain.

Key Questions

What does perplexity measure?

It measures how well a model predicts the next token on a held-out dataset. Lower perplexity means better average predictions on that evaluation.

Why does OpenAI want developers to track it?

OpenAI argues that pretraining measurements may remain informative as downstream benchmarks change or become saturated. It says shared measurements could help developers compare progress and inform common safety standards.

Does a particular perplexity value guarantee a capability?

No such guarantee is established in the source. OpenAI says capabilities have appeared alongside past improvements, but also says those developments were not clear in advance.

Is an AI research acceleration by 2027 confirmed?

No. The post says a major acceleration in pretraining research could conceivably happen as soon as 2027. That is the author’s projection, not a confirmed event.

Source: OpenAI

You May Also Like

Three Fronts: Inside the Week the AI Race Stopped Being Abstract

AIThis post was created with the assistance of artificial intelligence (AI).Google assembles…

HiRO-ACEv2: Simulating Extreme Weather At 3 Km Resolution For A Fraction Of The Cost

Hugging Face says HiRO-ACEv2 can simulate global 3 km rain, wind and pressure data at about 1/300th the cost of X-SHiELD.

Who Owns Claude AI? Anthropic, Investors And Control – HowStuffWorks

Anthropic develops Claude, but the supplied material does not establish its ownership structure, investors’ stakes or control arrangements.

The Delegation Ladder: The Four Agentic Loops, and What Each One Lets You Stop Doing

AIThis post was created with the assistance of artificial intelligence (AI).“Designing loops…