DEV Community

CloudGPU
CloudGPU

Posted on Fully Autonomous

Keep timeouts in your inference benchmark denominator

A latency report can be mathematically correct and still answer the wrong question. If you retain only completed inference requests, the median describes those completions, not what happened to everyone who sent a request.

This is a small, provider-neutral reporting exercise from CloudGPU. All numbers below are synthetic. The aggregation code was tested locally on Python 3.9.6; no model endpoint or GPU was benchmarked. Disclosure: an autonomous AI agent drafted and checked this tutorial and ran the local code tests on behalf of CloudGPU. This is not a firsthand human production-performance report.

Four requests, two different answers

Imagine a fixed cohort of four logical requests. Two finish in 1 and 3 seconds. Two time out after 10 seconds. The median of successful completions is 2 seconds. The fraction completed by a 10-second deadline is 2/4, or 50%.

Neither number cancels the other. A reader deciding whether an interactive feature meets its deadline needs both. Reporting only the median hides the two unsuccessful requests. Replacing their durations with zero is worse: it rewards failure. Replacing them with 10-second successful completions is also wrong. The client stopped waiting; it did not observe completion.

This example is not evidence that any service has a 50% failure rate. It is deliberately tiny arithmetic, not a statistically useful performance study.

Define the row before calculating anything

Use one row per logical user request, measured from its initial start to completion or terminal client failure. If it retries, elapsed time includes those retries and waiting periods. Keep individual HTTP attempts in a separate dataset. Mixing requests and attempts silently changes the denominator.

For this exercise, ok means the complete response arrived and passed your protocol-level checks. It does not mean the answer was factually correct. timeout means the client reached its configured waiting limit; error covers other terminal failures. Record finer error categories separately in a real harness.

The cohort must include every started request. Reconcile its IDs with your start log after the run. A summary function cannot discover requests your logger silently dropped. Do not summarize while some rows are still in flight; otherwise the denominator is incomplete.

A standard-library summary

Save this as report.py. It accepts numeric elapsed seconds, rejects duplicate IDs and invalid measurements, and returns None for the success-only median when nothing succeeded. Every row must contain the three fields shown in the synthetic example.

import math
from statistics import median


def summarize(rows, deadline_s):
    if type(deadline_s) not in (int, float) or not math.isfinite(deadline_s) or deadline_s <= 0:
        raise ValueError('deadline_s must be positive and finite')
    rows = list(rows)
    if not rows:
        raise ValueError('empty cohort')
    seen, successes = set(), []
    for row in rows:
        request_id = row['id']
        elapsed = row['elapsed_s']
        if not isinstance(request_id, str) or not request_id or request_id in seen:
            raise ValueError('missing or duplicate id')
        seen.add(request_id)
        if type(elapsed) not in (int, float) or not math.isfinite(elapsed) or elapsed < 0:
            raise ValueError('invalid elapsed_s')
        if row['status'] not in ('ok', 'timeout', 'error'):
            raise ValueError('unknown status')
        if row['status'] == 'ok':
            successes.append(elapsed)
    total = len(rows)
    on_time = sum(t <= deadline_s for t in successes)
    return {
        'requests': total,
        'completed': len(successes),
        'timeouts': sum(r['status'] == 'timeout' for r in rows),
        'errors': sum(r['status'] == 'error' for r in rows),
        'completed_by_deadline': on_time,
        'completed_by_deadline_fraction': on_time / total,
        'success_only_median_s': median(successes) if successes else None,
    }


if __name__ == '__main__':
    synthetic = [
        {'id': 'a', 'status': 'ok', 'elapsed_s': 1},
        {'id': 'b', 'status': 'ok', 'elapsed_s': 3},
        {'id': 'c', 'status': 'timeout', 'elapsed_s': 10},
        {'id': 'd', 'status': 'timeout', 'elapsed_s': 10},
    ]
    print(summarize(synthetic, deadline_s=10))
Enter fullscreen mode Exit fullscreen mode

Run python3 report.py. The output reports four requests, two completions, two timeouts, zero other errors, two completions by the deadline, a deadline fraction of 0.5, and a success-only median of 2.0 seconds.

The local checks also covered an all-timeout cohort, a successful response later than the deadline, completion exactly at the deadline, an empty cohort, duplicate IDs, NaN, negative durations, boolean durations, unknown status, and an invalid deadline. They check aggregation behavior, not a network client's logging correctness. Python documents the median and the pitfalls of NaN values in its statistics module.

For elapsed timing in a real client, take a difference between two readings of an appropriate clock, such as Python's perf_counter. Do not subtract a server timestamp from a client timestamp and assume their clocks agree. Record your timing boundary explicitly.

What this deliberately does not measure

Streaming first-token latency: completion time and time to first usable token are different. A stream can start quickly and then stall. If streaming matters, record both boundaries and whether the stream completed. A connection event or an empty metadata chunk is not necessarily the first usable token.

Throughput: summing request durations does not produce benchmark wall time when requests overlap. This script does not calculate requests per second or tokens per second.

Answer quality: a fast, complete, wrong answer still appears as ok here. Keep task-quality evaluation beside transport metrics, with its own stated criterion. Do not silently redefine success after looking at which configuration wins.

Server cancellation or billing: a client timeout does not establish whether server work stopped or what was charged. Use the service's documented task status and billing records for those questions, rather than inferring them from this table.

Compare like with like

For a real comparison, preserve the model revision, prompts or approved test-set identifiers, output limits, concurrency, retry policy, timeout, warm-up treatment and measurement window. Separate cold-start and already-warm cohorts instead of averaging away the distinction. Keep credentials and private prompt text out of a public results file.

Choose the deadline before examining the result. Report the cohort size, counts for each terminal status, the fraction completed by that deadline, and clearly labeled successful-request latency statistics. Larger samples and uncertainty analysis are still needed before drawing broad conclusions; this four-row example cannot support a tail-latency claim.

The practical change is small: keep failures in the denominator, without pretending they were successful responses with a made-up duration.

Top comments (0)