AI
Machine learning predictions beat simple math for GPU fine-tuning schedules
Analysis of 272 production fine-tuning runs shows that gradient-boosting models can predict job duration with far greater accuracy than traditional FLOPs-based estimates.
Hugging Face Blog
Most teams managing shared GPU infrastructure rely on a simple formula to estimate how long a fine-tuning job will take: multiply the model's parameter count and training data volume to get floating-point operations (FLOPs), then divide by GPU capacity. But this theoretical approach is frequently wrong—and can be wildly off.
Researchers from Dharma-AI examined this problem by collecting real data from 272 supervised fine-tuning jobs across a production cluster of NVIDIA H200 GPUs. The analysis reveals a stark gap between theory and practice: the standard FLOPs-based estimate achieved only modest accuracy (R² = 0.31), with an average error of over 12 hours for jobs that typically last around 23 hours. By contrast, a machine-learning regressor trained on job metadata achieved near-perfect predictions (R² = 0.98), cutting the error to 1.71 hours on average and 0.51 hours at the median.
What the formula misses#
The traditional FLOPs calculation accounts only for the core training loop's computational cost. Real jobs involve much more: environment setup, model loading, tokenization, periodic evaluations, checkpoint saves, and final model uploads. On a multi-tenant cluster, all these operations compete for network and storage bandwidth with other workloads, introducing unpredictable delays that scale differently with job size or GPU count.
Even swapping the assumed GPU efficiency (MFU, or Model FLOPs Utilization) across a wide range—from 25% to 65%—did not substantially improve the formula's predictions. Even a more sophisticated analytical model from prior work, which explicitly accounts for compute, memory traffic, and inter-device communication, matched only 11% of observed wall-clock times.
How learning does better#
The researchers built a supervised regression model using CatBoost, a gradient-boosting framework. The model takes as input 24 derived features—all available before a job starts—including evaluation volume, checkpoint counts, model size, batch configuration, and GPU allocation. Notably, the FLOP estimate itself feeds into the model as one feature, so the regressor learns corrections on top of the theoretical baseline.
Compared to simpler approaches, the results are dramatic: linear regression reduced the mean error to 7.23 hours, but gradient boosting achieved 1.71 hours. Crucially, 80% of predictions fell within 2.20 hours of actual duration, and 90% within 4.31 hours, making the estimates reliable for practical scheduling.

