DEV Community

Cover image for Why Your Fine-Tuning GPU Math Misses Wall Clock
Vinny Barreca
Vinny Barreca

Posted on

Why Your Fine-Tuning GPU Math Misses Wall Clock

Nine hours, the FLOP math said. The job was still running the next afternoon, and every run queued behind it slipped with it.

A preview of Predicting End-to-End Fine-Tuning Time for Generative Language Models, published this week by the Dharma-AI team on the Hugging Face blog and due on arXiv, measured 272 production supervised fine-tuning runs on shared H200 clusters, about 7,382 GPU-hours. The standard compute-only estimate explains 31% of the variance in wall-clock time, with a mean absolute error of 12.41 hours on jobs that average 23.43 hours.

The math is not wrong. The model is.

THE ESTIMATE EVERYONE USES

You know the formula. Six times parameter count times training tokens, divided by GPU count, peak throughput, and model FLOPs utilization. That last term is the problem. MFU is not a property of your cluster. It is an outcome of your job, and it is not known until the job has run.

The authors swept MFU from 25% to 65%. The best fit, 30%, still left a mean error above 12 hours. A far more detailed analytical model, one that accounts for memory traffic and inter-device communication, landed at a median of 11% of observed wall-clock time. That is not a rounding error. That is a different number.

Then they tried the cheap version of learning the correction. Gradient boosting over features that exist before dispatch: configuration, dataset sizes, model identifiers, GPU allocation, nothing from runtime logs. CatBoost reached an R2 of 0.98, a mean absolute error of 1.71 hours and a median error of 0.51 hours across five stratified folds. Linear regression on the same features managed 7.52 hours. The relationship between configuration and duration is not linear, and it is not arithmetic.

WHERE THE MISSING HOURS LIVE

The feature importance list is the part to read twice. Total evaluation samples ranked first, above parameter count.

Number of trainable parameters came second, then evaluation batch percentage, total parameters, and GPU count times training batch percentage. Checkpoint volume sat close behind. Evaluation cadence and checkpoint policy are choices you make in a config file, and they move the clock more than most teams expect.

The rest of the gap is the job around the loop. Environment setup, model download, tokenization, periodic evaluation passes, checkpoint uploads, the final push to object storage, contention with every other tenant. None of it scales linearly with GPU count, and none of it appears in a FLOP count.

The predictor has limits worth naming. Every run was supervised fine-tuning on H200s, up to 15 billion parameters. Above 60 hours the errors widen: 13 runs, a median miss of 5.19 hours, a 90th percentile of 18.51 hours. It is a scheduler's estimate, not a guarantee.

THE DECISION RULE

Use FLOPs for relative comparison inside one pipeline. Never for absolute booking.

  1. Compare two configurations with the same formula on the same data. The ratio is usually informative even when the absolute number is fiction.
  2. Never put a FLOP number on a calendar. It is not a duration, it is a lower bound wearing a costume.
  3. Book from measured wall clock per task shape: model size band, GPU count, method, evaluation cadence.
  4. Pad by the 90th percentile of your estimate-to-actual ratio, not the mean. Mean padding means half your jobs still run late.
  5. Below about 20 runs for a shape, you have no distribution. Book double and stop describing it as a prediction.

The line for your design review: an estimate without an error bar is a wish, and your queue is where wishes go to die.

THE PADDING YOU SHOULD ALREADY BE SHIPPING

Illustrative, not the paper's code, but this is the shape that works with logs you already have.

# illustrative: book wall clock from measured error, not FLOPs
def book(run, history):
    if run.flop_hours <= 0:
        return None                        # nothing to scale, refuse the booking
    ratios = [r.actual / r.flop_hours for r in history
              if r.shape == run.shape and r.flop_hours > 0]
    if len(ratios) < 20:
        return run.flop_hours * 2.0        # too few runs to be precisely wrong
    ratios.sort()
    p90 = ratios[int(0.9 * len(ratios)) - 1]   # empirical percentile, floor index
    return run.flop_hours * p90
Enter fullscreen mode Exit fullscreen mode

Three lines carry the argument. The guard keeps a zero or missing FLOP figure from silently producing a zero hour booking. The p90 line turns the number you already compute into the number your scheduler can use, and it improves every time a job finishes. The 20 is the honesty gate.

One nuance for the systems people: that percentile is empirical, taken on the ratio distribution with a floor index rather than linear interpolation between neighboring values. At twenty runs the two differ by less than the spread of the data, which is why the floor is fine here and why the output is a booking estimate and not a confidence bound.

CAPACITY IS THE REAL BILL

On a shared cluster, a mispredicted hour is either stranded hardware or queue debt for every job behind it. At 12.41 hours of mean error on a 23.43 hour workload, half a machine-day goes unplanned, and it never shows up on a utilization dashboard.

Learned predictors are not magic either. They are a way of admitting that job duration is a function of choices you make when you write the training config and that you have been ignoring those choices because a closed-form formula gave you permission to.

WHAT TO DO TODAY

  1. Pull your last 20 fine-tuning runs and compute FLOP estimate against actual wall clock. It is already in your logs.
  2. Log the non-arithmetic buckets this week: eval time, checkpoint I/O, setup, queue wait.
  3. Add a padding factor to queue bookings and stop booking raw FLOP time.
  4. Track prediction error per task shape, not one global number. The shape is what your scheduler sees.
  5. Recount the padding after every batch. Twenty new runs beats a clever formula.

THE SOURCE

The paper preview is "Why your fine-tuning job takes longer than the FLOPs say" by Bruno Duarte, Gabriel Pimenta de Freitas Cardoso, Isis F Carvalho, Breno de Almeida Beleza and Felipe Lima of Dharma-AI, on the Hugging Face blog, November 18, 2026, previewing Predicting End-to-End Fine-Tuning Time for Generative Language Models.

https://huggingface.co/blog/Dharma-AI/predicting-fine-tuning-time

The full paper was not yet on arXiv when this was written, which is why the link goes to the preview and not the PDF. Every figure above comes from that preview.

THE UNCOMFORTABLE QUESTION

If your schedule is built on arithmetic that ignores most of the clock, how much of your GPU bill is scheduled work, and how much is organized waiting?

That estimate is the input. Allocation is the decision.

If you set the queue policy, the next read is the allocator itself. Note #164: your GPU scheduler is your real capacity; this same paper shows why training and real-time inference cannot share a FIFO pool, why a fixed reservation strands half a cluster at 51.6% and 53.6%, and the constraint-aware ordering that reclaims it. Batch jobs into contiguous blocks, elastic capacity from the gaps.

Top comments (0)