DEV Community

Odd_Background_328
Odd_Background_328

Posted on

Your Free-Tier Load Test Is Lying: How Shared CPU Pollutes Capacity Numbers

#observability #devops #performance 
03:47 UTC. Load test running. p95 climbing. vCPU steady. Conclusion: scale up.
Enter fullscreen mode Exit fullscreen mode

Wrong. The vCPU was never yours.

I spent a week load-testing an agent worker on a free server with free model access. The numbers looked clean. The conclusion was garbage. This article is the autopsy.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The setup that looked fine

Topology:

[load generator] -> [free server: worker + Redis] -> [free model endpoint]
Enter fullscreen mode Exit fullscreen mode

Declared test conditions:

  • One free server, 1 vCPU, 1 GB RAM.
  • One worker process, one Redis instance.
  • 500 synthetic jobs, Poisson arrival, mean 2/sec.
  • Measured: queue latency, token throughput, CPU steal.

MonkeyCode's free tier provided the model access and the free server option. Both are real. Both have limits that are not visible in a dashboard.

What the first run showed

Run 1 output, 10-minute window:

requests: 500
avg latency: 1.84s
p95 latency: 4.12s
p99 latency: 9.87s
cpu: 87%
Enter fullscreen mode Exit fullscreen mode

Looks like a saturation problem. p99 is 5x the average. The obvious move: scale the worker.

I did not scale. I checked /proc/stat first.

The metric nobody graphs

steal time. The hypervisor's bill to your VM for CPU it promised but did not deliver.

mpstat 1 5 | grep -i steal
Enter fullscreen mode Exit fullscreen mode

Run 1, same window:

03:47:01  %steal  38.5
03:47:02  %steal  71.2
03:47:03  %steal  12.0
03:47:04  %steal  89.4
03:47:05  %steal  44.1
Enter fullscreen mode Exit fullscreen mode

Your load test was not testing your worker. It was testing your neighbor on the same physical host.

The experiment that proved it

I reran the same load at 02:00 local time, when the shared host was quieter.

requests: 500
avg latency: 0.92s
p95 latency: 1.71s
p99 latency: 2.44s
cpu: 63%
cpu steal: 4%
Enter fullscreen mode Exit fullscreen mode

Same code. Same queue. Same model endpoint. Half the latency. The only variable was who else was running on the hypervisor.

The capacity number you computed at 14:00 is not the capacity number you have at 02:00. Both are real. Neither is "the" capacity.

How to detect the pollution

Three signals, in order of reliability:

  1. %steal above 10% during the test window. If steal is spiking, your latency numbers are partly someone else's workload.
  2. Latency variance without a corresponding queue-depth change. Queue depth flat, latency doubling? Suspect the host, not your code.
  3. Rerun at a different hour and get a different curve. A capacity test that is not time-of-day reproducible is not a capacity test.

The corrected methodology

For a free server, use a three-window protocol:

Window Purpose Decision rule
Peak hours (14:00-16:00) Worst-case latency If p95 > SLO, add headroom or move workload
Off-peak (02:00-04:00) Best-case latency If p95 still > SLO, the problem is your code
Mid (08:00-10:00) Typical latency Baseline for capacity math

Run the same load in all three windows. Compare the curves. If they diverge by more than 30%, your free server is the variable, and no scaling decision is valid until you accept that.

What this means for capacity planning

With a free server, you are not planning capacity. You are planning a range:

  • Lower bound: off-peak throughput. This is what your system can do when the host is quiet.
  • Upper bound: peak-hours throughput. This is what your users experience during the day.
  • The gap between them: the cost of shared infrastructure. That gap is your real constraint.

If your workload needs predictable p95, a free server cannot give it. The model is not the bottleneck. The hypervisor is.

The rollback and cleanup

After the test:

# stop the worker
kill $(pgrep -f worker.py)

# clear the queue
redis-cli flushdb

# destroy the server (per your provider's flow)
# then recreate it for the next test
Enter fullscreen mode Exit fullscreen mode

Keep the raw output from all three windows. You will need it when someone asks why the "same" test gave different numbers.

Who should not use this approach

  • Teams with a hard latency SLO. Free servers have no CPU guarantee, so your SLO has no guarantee.
  • Capacity tests that feed a budget request. Numbers from a shared host will not survive finance review.
  • Any experiment where the conclusion is "scale up". You cannot conclude that from a polluted baseline.

The takeaway

Free infrastructure is fine for learning the shape of a system. It is not fine for measuring its size.

The steal counter is your first question, not your last resort. Check it before you trust a single latency number from a shared box.

The free server and free model access from MonkeyCode are enough to reproduce this entire experiment. Run the three-window protocol once, and you will know exactly what your free tier can and cannot promise.

Then delete the server. The lesson is the deliverable.

Top comments (0)