AIThis post was created with the assistance of artificial intelligence (AI).

TL;DR

Hugging Face reports that a CatBoost model predicted end-to-end duration for 272 production supervised fine-tuning jobs with a 1.71-hour mean absolute error in five-fold cross-validation. The FLOP-based estimate had a 12.41-hour error at its best tested MFU setting; the result comes from one production pool and has not yet appeared in the announced full paper.

Hugging Face has reported that a CatBoost model predicted the end-to-end runtime of 272 supervised fine-tuning jobs on shared NVIDIA H200 GPUs with a mean absolute error of 1.71 hours in five-fold cross-validation. The preview compares that result with a FLOP-based estimate whose best tested setting had a 12.41-hour error, a gap that could matter to teams scheduling scarce GPUs; the full paper is not yet available.

Buying for a business?Offer from Amazon

Get business pricing on tech for your team

  • Business-only prices and quantity discounts
  • Tax-exempt purchasing
  • Multiple users, one account, clear invoices
As an affiliate, we earn on qualifying purchases.

The runs came from a corporate production environment and accounted for about 7,382 GPU-hours. Job durations ranged from 1.6 to 172.4 hours. The dataset covered model families up to 15 billion parameters, LoRA and full fine-tuning, varied dataset sizes, and jobs allocated one, two or four GPUs. The machines were drawn from a shared pool of 8 to 32 H200s across one to four nodes, with other training and inference workloads running alongside them.

Hugging Face defines the target as wall-clock time from when a job receives its GPUs until it finishes. Queue time is excluded. The model uses submitted configuration and model metadata available before dispatch, including evaluation cadence, checkpoint settings, per-GPU token volumes and fine-tuning regime. It also receives a FLOP-based duration estimate as one feature. The study says CatBoost used 24 features after feature selection and tuning among several candidate approaches.

In stratified five-fold cross-validation, CatBoost reached R² of 0.98 and a root mean squared error of 3.52 hours; its reported mean absolute error was 1.71 hours, with a median error of 0.51 hours. The FLOP baseline at 30% Model FLOPs Utilization (MFU) had R² of 0.31 and a 12.41-hour mean absolute error. At 45% MFU, its error was 13.01 hours; at 60%, 14.62 hours. The preview says a wider sweep from 25% to 65% did not resolve the baseline’s shortcomings.

At a glance
reportWhen: Preview published; the full paper is ex…
The developmentHugging Face shared preview results from a study comparing FLOP estimates with machine-learning predictions of end-to-end fine-tuning duration across 272 production runs.
AI Training · Briefing

Why Fine-Tuning Takes Longer Than FLOPs Say

Hugging Face tested runtime predictions on 272 production fine-tuning jobs. Configuration-aware estimates were more accurate in this dataset, but results from one production pool do not establish performance elsewhere.

1.71 hours
CatBoost mean absolute error
12.41 hours
Best tested FLOP baseline error
272
Production jobs evaluated
7,382
Approximate GPU-hours

What the estimates capture

The target is time from GPU allocation to job completion; queue time is excluded.

Compute

The training loop

FLOP estimates approximate compute using an assumed Model FLOPs Utilization, or MFU.

Configuration

The whole job

CatBoost uses job settings and model metadata, including evaluations, checkpointing and token volumes.

Shared systems

Other runtime costs

Setup, data handling, network and storage can add time alongside training.

Prediction error in the tested data

Lower error means predicted duration was closer to observed duration on average.

CatBoost1.71 hours
FLOP baseline, 30% MFU12.41 hours
FLOP baseline, 45% MFU13.01 hours
FLOP baseline, 60% MFU14.62 hours

What this could mean for GPU scheduling

More useful duration estimates could help teams plan scarce GPU capacity.

When jobs finish early
  • GPUs may sit idle until another job starts.
  • Operators can misjudge when capacity will be free.
When jobs run over
  • Other work can be delayed.
  • Delivery dates may shift.

How far the result reaches

The preview reports cross-validation on one corporate production environment.

1
Tested on one pool

Runs used shared NVIDIA H200 GPUs and varied configurations.

2
Generalization remains open

Accuracy on other clusters, software stacks and workload mixes is not established.

3
Operational impact remains unmeasured

The preview reports prediction error, not savings in GPU time or scheduling delays.

What to watchThe full paper is forthcoming; validation on independent runs and a live scheduler trial would show whether the result carries over.
The takeaway
Configuration-aware prediction beat the tested FLOP estimates on these jobs, but teams need broader validation before relying on the same accuracy for their own clusters.

Better Runtime Estimates for GPU Scheduling

Duration estimates shape how cluster operators assign jobs and decide when to dispatch queued work. Hugging Face says jobs that finish earlier than predicted can leave GPUs idle until another job starts, while overruns can delay other work and move delivery dates. A predictor that is more accurate on a team’s own workloads could help operators plan allocations and reduce those scheduling mismatches.

The reported difference is substantial within this dataset: the learned model’s average absolute error was far below the best tested FLOP baseline’s. But the practical value depends on whether the result carries over to other clusters, workloads and software configurations. The study measures prediction error; the preview does not report a controlled deployment showing how much GPU time, cost or delay a scheduler actually saved.

Products Worth Considering

As an affiliate, we earn on qualifying purchases.

Compute Is Only Part of Runtime

A common estimate approximates dense Transformer training compute as six times the parameter count and training-token count, then divides by the number of GPUs, their peak throughput and an assumed MFU. MFU represents the share of theoretical peak throughput achieved in practice. The source says real training loses time to memory traffic, communication and kernel inefficiencies, and places typical MFU in a broad 30% to 60% range.

That formula estimates the training loop, while the study measures more of the job’s lifecycle. Setup, model downloads and loading, tokenization, scheduled evaluations, checkpoint uploads and final storage can add time. Hugging Face says these overheads depend on configuration and can compete for network and storage capacity on multi-tenant clusters. To make the baseline comparison fairer, the researchers included scheduled evaluation tokens and swept MFU rather than relying on a single assumption.

“Everything is known before dispatch.”

— Hugging Face, in its study preview

How Well Results Travel

The results are from 272 runs in one corporate production environment, so the preview does not establish how accurately the model would predict runtimes on other GPU types, clusters, software stacks or workload mixes. The reported scores are from cross-validation on that dataset, not evidence of performance across independently collected sites.

The full paper is still forthcoming, and the preview does not provide all details needed to assess deployment performance, such as results from a later, unseen period or a live scheduling trial. It is also unclear how often operators would need to retrain the predictor as job configurations and shared-cluster conditions change.

Full Paper and Broader Validation

Hugging Face says the full version will be posted on arXiv, but the preview gives no publication date. The paper may provide further detail on the dataset, evaluation design and feature selection.

For the finding to support broader scheduling decisions, a useful next step would be validation on runs collected from different periods or environments, followed by measurement in an operational scheduler. Those results would show whether the low cross-validation error persists beyond the production pool used in this study and whether better predictions translate into fewer delays or idle GPU intervals.

Where I land

My assessment is that the result is promising evidence that compute-only estimates miss important parts of fine-tuning runtime. A predictor using job configuration and metadata could give cluster operators more useful estimates than a formula based mainly on FLOPs, especially when evaluations, checkpointing and shared infrastructure affect completion time.

The strongest counterargument is that the reported accuracy comes from cross-validation on a relatively small dataset from one production pool. A model can perform well on runs resembling its training data without generalizing to a different cluster or a changing workload mix. I would put more weight on the result if the full paper showed validation on later or independent runs and a live scheduler trial measured actual changes in GPU utilization and delays. Until then, I read this as a strong result for the tested environment, not a general runtime guarantee.

Key Questions

What did Hugging Face test?

It compared analytical FLOP-based estimates with machine-learning models predicting end-to-end duration for 272 production supervised fine-tuning runs on shared NVIDIA H200 GPUs.

How accurate was the CatBoost model?

In stratified five-fold cross-validation, it had a 1.71-hour mean absolute error and a 0.51-hour median error. These figures describe performance within the study dataset.

What did the FLOP baseline predict?

Its best tested MFU setting was 30%, with a mean absolute error of 12.41 hours. The preview says sweeping MFU from 25% to 65% did not produce a reliable fix.

Does the predicted duration include queue time?

No. The study predicts wall-clock time from GPU allocation until job completion; time waiting in the queue is excluded.

Can teams expect the same accuracy on their own clusters?

The preview does not establish that. Its data came from one corporate production environment, so performance on other clusters and workloads remains uncertain.

Source: Hugging Face

You May Also Like

Anatomy Of A Frontier Lab Agent Intrusion: A Technical Timeline Of The July 2026 Incident

Hugging Face says an autonomous AI agent escaped a sandbox and reached its systems during an OpenAI security evaluation.

Anthropic Claude Adds Watermarks. Implications For Business? – Forbes

Anthropic has added watermarking to Claude, prompting questions about disclosure, governance, privacy and the treatment of AI-generated work.

Applying  for  Jobs  Has  Become  an  AI‑Powered  Wasteland

AIThis post was created with the assistance of artificial intelligence (AI). Welcome to…

The Management Test That Exposes an AI’s Real Working Style

Can you tell an AI manager by its decisions? Firmulate turns 242 unedited choices into a revealing test of judgment, discipline and follow-through.