Your Timer Is a Wall Clock, Not a Stopwatch: Four Elapsed-Time Myths and a Monotonic Probe

How long did that batch job take?

Simple question. Your logs probably answer it wrong.

I kept seeing two numbers for one run. The job log said eleven minutes. The dashboard said forty-two. Neither was lying. They were reading different clocks.

So here is the myth-busting FAQ I wish I had read first. Four claims developers repeat. Evidence for each. And one probe you can run in ten minutes.

The setup: one job, two clocks

The workflow is boring and common. Take N inputs. Call a model once per input. Write a JSONL report.

I run this kind of thing on a free server, calling a free model endpoint, because it is a cheap way to test the harness before spending anything. MonkeyCode's free model access and free server option are what I use for that step.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

That is the whole product role here. The rest of the article is about clocks, and it stays true if you swap in any host and any provider.

Myth 1: "time.time() minus time.time() is a duration"

This is the big one. Everyone does it. It is wrong.

Wall clock answers "what time is it?" It does not answer "how long did that take?" Those are different questions.

Your host adjusts wall time. NTP steps it, slews it, or corrects it after a suspend. Subtract two wall readings and you can get a negative number, or a number that includes time you never spent.

Check your own host before you argue:

timedatectl                     # is NTP sync on? any last adjustment?
chronyc tracking 2>/dev/null    # offset and slew rate, if chrony is present
cat /proc/uptime                # seconds since boot, two numbers
python3 -c "import time; print(time.get_clock_info('monotonic'))"

That last line matters most. On Linux you will usually see clock_gettime(CLOCK_MONOTONIC). That clock never goes backwards. That is the clock you want for durations.

Corrected mental model: wall clock is for labels, monotonic is for durations.

Put a wall timestamp on every log line. Measure every elapsed time with time.monotonic().

Myth 2: "If my timer jumped, the host slept"

Maybe. But that is not the only explanation. And the difference changes your fix.

A gap in monotonic time means one thing: your process did not run. It does not tell you why. Throttled? Preempted? Host suspended? Blocked on I/O? All produce a gap.

So log both clocks side by side. Then compare the deltas.

# clock_probe.py — log wall/monotonic pairs, flag disagreement
import json, os, sys, time

path = sys.argv[1] if len(sys.argv) > 1 else "clock.jsonl"
interval = float(os.environ.get("PROBE_INTERVAL", "5"))

prev = None
with open(path, "a", buffering=1) as f:
    while True:
        wall, mono = time.time(), time.monotonic()
        rec = {"wall": round(wall, 3), "mono": round(mono, 3)}
        if prev:
            rec["wall_delta"] = round(wall - prev["wall"], 3)
            rec["mono_delta"] = round(mono - prev["mono"], 3)
            rec["drift"] = round(rec["wall_delta"] - rec["mono_delta"], 3)
        f.write(json.dumps(rec) + "\n")
        prev = rec
        time.sleep(interval)

Run it, leave the box alone, then read the drift:

PROBE_INTERVAL=5 python3 clock_probe.py clock.jsonl &
sleep 900 && kill %1
python3 - <<'EOF'
import json
rows = [json.loads(l) for l in open("clock.jsonl")]
for r in rows:
    if abs(r.get("drift", 0)) >= 1.0:
        print("disagreement:", r)
EOF

Now read the two columns as separate signals.

• Large drift, small mono_delta → the wall clock moved. Your host stepped time.
• Large mono_delta, normal drift → the process really was not scheduled.
• Both consistent → your host kept time fine for that window. Good, now you know.

Corrected mental model: a monotonic gap is evidence of not running, not proof of suspension.

Stop guessing which. Measure it once, then cite your own log.

Myth 3: "A timeout means the request failed"

Does it? A timeout is a local observation about your side of the socket. It says nothing about the remote side.

The remote work may have completed after you gave up. Retry blindly and you may now have two results, two writes, or two charges against a budget.

This is the myth that actually costs people data. Free endpoint, paid endpoint, same story.

Corrected mental model: a timeout means unknown, and unknown needs a dedupe key.

Derive the key from the request, not from the clock:

import hashlib, json

def request_key(payload: dict) -> str:
    canon = json.dumps(payload, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(canon.encode()).hexdigest()[:16]

Log the key before the call. Log it again after. Persist it to disk, not to a variable. When you restart, you can tell a rerun from a new unit of work.

Myth 4: "Retries are free because the model is free"

They are not. Free is a price, not a property of your runtime.

Every retry spends wall time, and wall time is your actual budget. Three retries with a 30-second timeout can turn a fast job into a slow one with no extra output.

Wrap the call so timing lives in one place:

import time

def timed(fn, *args, **kwargs):
    t0 = time.monotonic()
    result = fn(*args, **kwargs)   # plug in your provider call here
    return result, round(time.monotonic() - t0, 3)

Note what this does not do. It makes no assumption about request shape, model name, or SDK. You pass your own callable. That keeps the harness portable.

Then add a deadline and a seen-set:

def run_batch(items, call, budget_s=600):
    start = time.monotonic()
    done, seen = [], set()
    for item in items:
        key = request_key(item)
        if key in seen:
            continue
        if time.monotonic() - start > budget_s:
            print(f"budget hit after {len(done)} results; stopping cleanly")
            break
        result, elapsed = timed(call, item)
        seen.add(key)
        done.append({"key": key, "elapsed_s": elapsed, "result": result})
    return done

Two rules for this snippet.

1. seen should be backed by disk. A set in memory dies with the process.
2. time.monotonic() is per-boot. Never persist it, never compare it across hosts.

A decision table you can paste into the README

Question	Clock to use	Never use
What time did this happen?	Wall, ISO 8601, UTC	Local time, no zone
How long did the call take?	Monotonic	Wall subtraction
Is this the same request?	Content hash	Timestamp
Did the host sleep?	Wall and monotonic, compared	Either one alone
Should the job stop?	Monotonic deadline	A cron interval

That table is the whole article compressed. The rest is evidence.

Limitations, honestly

This probe proves what happened on one host, for one window. It does not prove a provider's latency, an uptime guarantee, or anything about a service level.

It also cannot see inside a hypervisor. If your host suspends, you will observe the gap, not the cause.

And a single probe run is not a benchmark. Run it across idle periods, busy periods, and a restart before you trust the numbers.

Who should not use this approach

• Anyone needing hard real-time guarantees. A monotonic clock is a measurement tool, not a scheduler.
• Billing or audit pipelines. Those need server-side timestamps and reconciliation, not your container's clock.
• Teams that cannot store state. Dedupe without persistence is theatre.
• Anyone who will not read their own probe output. The tool only helps if you look.

What to do in the next ten minutes

Run time.get_clock_info('monotonic'). Confirm your clock source. Then patch the one place in your code where you subtract two wall timestamps.

That single change is usually worth more than a week of latency dashboards.

How long has your timer been lying to you? If you want to run the probe end to end, free model access plus the free server option in MonkeyCode is enough to do it without a paid plan.

Just read the drift column before you trust the chart.