The Free Box Was Fine. The Wakeups Were Not.

The bottleneck was not CPU. It was the timer. My nightly job kept overrunning its window by minutes, and nothing in the logs said "slow".

Every span looked healthy. The mean was flat. Throughput was fine until 05:40, then it fell off a cliff.

So I stopped profiling the code. I started profiling the clock.

How do you profile a clock? You ping it. You ask the kernel for a wakeup in 10ms, then measure when it actually arrives. That gap is drift. Drift has a distribution, and on a shared host its tail decides your deadline, not your loop.

The probe: twenty lines, no root, no dependencies

Save this as drift.py. It sleeps on a fixed schedule and records the overshoot in milliseconds.

#!/usr/bin/env python3
# drift.py — measures timer wakeup drift. stdlib only, no root.
import csv, sys, time

def run(period_ms: int, n: int):
    period = period_ms / 1000.0
    next_t = time.monotonic()
    for i in range(n):
        next_t += period
        time.sleep(max(0.0, next_t - time.monotonic()))
        yield i, (time.monotonic() - next_t) * 1000.0  # overshoot in ms

if __name__ == "__main__":
    period = int(sys.argv[1]) if len(sys.argv) > 1 else 10
    n = int(sys.argv[2]) if len(sys.argv) > 2 else 6000
    w = csv.writer(sys.stdout)
    w.writerow(["i", "drift_ms"])
    for i, d in run(period, n):
        w.writerow([i, round(d, 3)])

Run it for one minute at a 10ms period:

python3 drift.py 10 6000 > drift.csv

Then summarise it without pandas:

python3 - <<'PY'
import csv, statistics as st
d = sorted(float(r["drift_ms"]) for r in csv.DictReader(open("drift.csv")))
q = lambda p: d[min(len(d) - 1, int(len(d) * p))]
print(f"n={len(d)} mean={st.mean(d):.2f} p50={q(.5):.2f} p99={q(.99):.2f} max={d[-1]:.2f}")
PY

The median will lie to you. The median always lies. Read p99 and max instead.

# illustrative shape only — run the probe, use your numbers
n=5999 mean=1.83 p50=0.94 p99=11.20 max=412.60

That last number is the article. One wakeup landed over 400ms late. Do that sixty times an hour and your six-hour job becomes a seven-hour job. Nobody sees it in an average.

Was it my code or the host?

You need a control. Run two probes at once, one fast and one slow:

python3 drift.py 10 6000 > fast.csv &
python3 drift.py 1000 60 > slow.csv &
wait

Now line up the timestamps. If the same wall-clock seconds spike in both files, the noise is host-level. If only the short-period probe spikes, your loop is amplifying it.

On Linux, ask the kernel directly:

cat /proc/pressure/cpu /proc/pressure/io /proc/pressure/memory
grep -E '^(cpu|procs_running)' /proc/stat

If some avg10 climbs for CPU while your probe is idle, you are a guest on a busy box. No code change fixes that. You move the work, or you move the deadline.

The experiment: fewer wakeups, same work

Here is the model I use now. Cost per wakeup is fixed overhead plus the drift tail. So total wall time behaves roughly like:

wall ≈ wakeups × (overhead + drift_p99) + work / parallelism

Batch size K divides the first term and leaves the second alone. Doubling K does not make each item slower. It just stops paying the tail twice as often.

So I did two things. I batched the queue drain, and I scheduled with jitter instead of on the second:

import random, time

def next_sleep(base_s: float, p99_ms: float) -> float:
    # spread wakeups so a hundred workers don't hit the host at once
    return base_s + random.uniform(0, max(0.05, p99_ms / 1000.0))

Then I re-ran the probe with the job live. Same box, same period, fewer wakeups per hour. The tail shrank because there were fewer chances to draw it.

What still belongs off a shared box

Not everything tolerates a noisy clock. Here is the table I keep next to the graph.

Workload	Survives a shared box	Why
Nightly batch, minute-scale deadline	Yes	The drift tail is small next to the slack
Cron that must fire on the second	No	Wakeup latency is not yours to control
100ms polling loop	No	Tail drift is a large fraction of the period
p99 latency SLO on user HTTP	No	You inherit host-level noise
Hour-long CPU build	Yes	One long slice beats many short wakeups
Anything billing on wall-clock precision	No	Never trust a shared clock for money

The graph I kept is one line per host: drift CDF, dated, regenerated monthly. When a deadline breaks again, I overlay today's line on last month's. If it moved, the host moved, not my code.

Where the free capacity fits

This workflow has one part that genuinely likes being slow: the classification pass at the end of the drain. It is bursty, it tolerates queue time, and it fails safely. That is exactly the shape to put behind a queue, on capacity you do not own.

MonkeyCode offers free model access, and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

So the plan is boring on purpose. Run the probe first. Then run one queued classification batch and measure its queue residence time the same way. If the tail is acceptable, keep it. If not, move it back. Free capacity is a measurement opportunity, not an architecture decision.

Treat queue time as a latency budget line, not as an error. Retries hide shared capacity problems until the deadline finds them.

What this does not measure

Drift is timer overshoot. It is not syscall latency, not disk throughput, not network jitter. Run separate probes for those.

PSI counters need a modern kernel, and cgroup v2 changes what you see. A container may report clean pressure while the host is on fire. Read the host, not just the cgroup.

And be honest about the fit. If your product promises p99 under 200ms, a shared free box is the wrong home. If you need reproducible benchmark numbers for a paper, same answer. This method is for deadline-shaped batch work, not for SLO-shaped serving.

If you want a cheap first data point, the free server option is a fine place to run drift.py before you trust it with anything real.