TL;DR
Hugging Face reports that a CatBoost model predicted end-to-end duration for 272 production supervised fine-tuning jobs with a 1.71-hour mean absolute error in five-fold cross-validation. The FLOP-based estimate had a 12.41-hour error at its best tested MFU setting; the result comes from one production pool and has not yet appeared in the announced full paper.
Hugging Face has reported that a CatBoost model predicted the end-to-end runtime of 272 supervised fine-tuning jobs on shared NVIDIA H200 GPUs with a mean absolute error of 1.71 hours in five-fold cross-validation. The preview compares that result with a FLOP-based estimate whose best tested setting had a 12.41-hour error, a gap that could matter to teams scheduling scarce GPUs; the full paper is not yet available.
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
The runs came from a corporate production environment and accounted for about 7,382 GPU-hours. Job durations ranged from 1.6 to 172.4 hours. The dataset covered model families up to 15 billion parameters, LoRA and full fine-tuning, varied dataset sizes, and jobs allocated one, two or four GPUs. The machines were drawn from a shared pool of 8 to 32 H200s across one to four nodes, with other training and inference workloads running alongside them.
Hugging Face defines the target as wall-clock time from when a job receives its GPUs until it finishes. Queue time is excluded. The model uses submitted configuration and model metadata available before dispatch, including evaluation cadence, checkpoint settings, per-GPU token volumes and fine-tuning regime. It also receives a FLOP-based duration estimate as one feature. The study says CatBoost used 24 features after feature selection and tuning among several candidate approaches.
In stratified five-fold cross-validation, CatBoost reached R² of 0.98 and a root mean squared error of 3.52 hours; its reported mean absolute error was 1.71 hours, with a median error of 0.51 hours. The FLOP baseline at 30% Model FLOPs Utilization (MFU) had R² of 0.31 and a 12.41-hour mean absolute error. At 45% MFU, its error was 13.01 hours; at 60%, 14.62 hours. The preview says a wider sweep from 25% to 65% did not resolve the baseline’s shortcomings.
Why Fine-Tuning Takes Longer Than FLOPs Say
Hugging Face tested runtime predictions on 272 production fine-tuning jobs. Configuration-aware estimates were more accurate in this dataset, but results from one production pool do not establish performance elsewhere.
What the estimates capture
The target is time from GPU allocation to job completion; queue time is excluded.
The training loop
FLOP estimates approximate compute using an assumed Model FLOPs Utilization, or MFU.
The whole job
CatBoost uses job settings and model metadata, including evaluations, checkpointing and token volumes.
Other runtime costs
Setup, data handling, network and storage can add time alongside training.
Prediction error in the tested data
Lower error means predicted duration was closer to observed duration on average.
What this could mean for GPU scheduling
More useful duration estimates could help teams plan scarce GPU capacity.
- GPUs may sit idle until another job starts.
- Operators can misjudge when capacity will be free.
- Other work can be delayed.
- Delivery dates may shift.
How far the result reaches
The preview reports cross-validation on one corporate production environment.
Runs used shared NVIDIA H200 GPUs and varied configurations.
Accuracy on other clusters, software stacks and workload mixes is not established.
The preview reports prediction error, not savings in GPU time or scheduling delays.
Better Runtime Estimates for GPU Scheduling
Duration estimates shape how cluster operators assign jobs and decide when to dispatch queued work. Hugging Face says jobs that finish earlier than predicted can leave GPUs idle until another job starts, while overruns can delay other work and move delivery dates. A predictor that is more accurate on a team’s own workloads could help operators plan allocations and reduce those scheduling mismatches.
The reported difference is substantial within this dataset: the learned model’s average absolute error was far below the best tested FLOP baseline’s. But the practical value depends on whether the result carries over to other clusters, workloads and software configurations. The study measures prediction error; the preview does not report a controlled deployment showing how much GPU time, cost or delay a scheduler actually saved.
Products Worth Considering

Amazon Kindle Scribe Colorsoft 64GB with Premium Pen

Samsung 27-Inch Odyssey G5 QHD Gaming Monitor (G50F)

Samsung Galaxy Tab A11+ Tablet (6GB RAM, 128GB Storage, Gray)
As an affiliate, we earn on qualifying purchases.
Compute Is Only Part of Runtime
A common estimate approximates dense Transformer training compute as six times the parameter count and training-token count, then divides by the number of GPUs, their peak throughput and an assumed MFU. MFU represents the share of theoretical peak throughput achieved in practice. The source says real training loses time to memory traffic, communication and kernel inefficiencies, and places typical MFU in a broad 30% to 60% range.
That formula estimates the training loop, while the study measures more of the job’s lifecycle. Setup, model downloads and loading, tokenization, scheduled evaluations, checkpoint uploads and final storage can add time. Hugging Face says these overheads depend on configuration and can compete for network and storage capacity on multi-tenant clusters. To make the baseline comparison fairer, the researchers included scheduled evaluation tokens and swept MFU rather than relying on a single assumption.
“Everything is known before dispatch.”
— Hugging Face, in its study preview
How Well Results Travel
The results are from 272 runs in one corporate production environment, so the preview does not establish how accurately the model would predict runtimes on other GPU types, clusters, software stacks or workload mixes. The reported scores are from cross-validation on that dataset, not evidence of performance across independently collected sites.
The full paper is still forthcoming, and the preview does not provide all details needed to assess deployment performance, such as results from a later, unseen period or a live scheduling trial. It is also unclear how often operators would need to retrain the predictor as job configurations and shared-cluster conditions change.
Full Paper and Broader Validation
Hugging Face says the full version will be posted on arXiv, but the preview gives no publication date. The paper may provide further detail on the dataset, evaluation design and feature selection.
For the finding to support broader scheduling decisions, a useful next step would be validation on runs collected from different periods or environments, followed by measurement in an operational scheduler. Those results would show whether the low cross-validation error persists beyond the production pool used in this study and whether better predictions translate into fewer delays or idle GPU intervals.
Where I land
My assessment is that the result is promising evidence that compute-only estimates miss important parts of fine-tuning runtime. A predictor using job configuration and metadata could give cluster operators more useful estimates than a formula based mainly on FLOPs, especially when evaluations, checkpointing and shared infrastructure affect completion time.
The strongest counterargument is that the reported accuracy comes from cross-validation on a relatively small dataset from one production pool. A model can perform well on runs resembling its training data without generalizing to a different cluster or a changing workload mix. I would put more weight on the result if the full paper showed validation on later or independent runs and a live scheduler trial measured actual changes in GPU utilization and delays. Until then, I read this as a strong result for the tested environment, not a general runtime guarantee.
Key Questions
What did Hugging Face test?
It compared analytical FLOP-based estimates with machine-learning models predicting end-to-end duration for 272 production supervised fine-tuning runs on shared NVIDIA H200 GPUs.
How accurate was the CatBoost model?
In stratified five-fold cross-validation, it had a 1.71-hour mean absolute error and a 0.51-hour median error. These figures describe performance within the study dataset.
What did the FLOP baseline predict?
Its best tested MFU setting was 30%, with a mean absolute error of 12.41 hours. The preview says sweeping MFU from 25% to 65% did not produce a reliable fix.
Does the predicted duration include queue time?
No. The study predicts wall-clock time from GPU allocation until job completion; time waiting in the queue is excluded.
Can teams expect the same accuracy on their own clusters?
The preview does not establish that. Its data came from one corporate production environment, so performance on other clusters and workloads remains uncertain.
Source: Hugging Face
