FAQ: A GitLab Stopwatch Job Is Not an API Test

Does a green timing job prove your API is fast?
I keep seeing that claim on merge requests.
It does not. Wall-clock is not an SLO.

Why this FAQ exists

GitLab prints a duration on every job.
That figure feels like a benchmark result.
It is a kitchen timer. Nothing more.

People then ask a model for a quick probe.
They run it on some leftover host.
They paste a green check into the MR.

What did they actually measure?
Usually the wrong clock. Often the wrong network.
This FAQ names the mix-up. Then it replaces it.

The mental model I use

I split every probe into three clocks.
Mix them and your review becomes folklore.

1. Job wall-clock. Pickup, image pull, clone, script, artifacts.
2. Client RTT. DNS, TCP, TLS, first byte, total transfer.
3. Server work. Time spent behind your load balancer.

Which clock did your job record?
If you cannot name it, stop merging.
A pipeline can host a probe. It is not the probe.

Myth 1: Job duration equals request latency

The claim. The job finished in twelve seconds, so the API is fine.

The evidence. Open the log. Count steps before the first request.

You will see runner pickup.
You will see an image pull.
You will see git clone, maybe a cache restore.

None of that is your API.
GitLab's UI duration includes all of it.
Treating that sum as TTFB is a category error.

Corrected model. Time the probe inside the script.
Print job duration as a separate field.
Never compare those two numbers in one sentence.

#!/bin/sh
set -eu
: "${TARGET_URL:?set TARGET_URL}"
echo "probe_label=${PROBE_LABEL:-unlabeled}"
echo "target=${TARGET_URL}"
curl -sS -o /tmp/body \
  -w "dns=%{time_namelookup}\nconnect=%{time_connect}\ntls=%{time_appconnect}\nttfb=%{time_starttransfer}\ntotal=%{time_total}\nhttp=%{http_code}\nsize=%{size_download}\n" \
  "$TARGET_URL"
wc -c /tmp/body

time_total is still client RTT from that runner.
It is not p95. It is not capacity.
It is one labeled sample. Keep it labeled.

Myth 2: A rehearsal host matches a GitLab runner

The claim. It was fast on my extra server, so GitLab will agree.

The evidence. Paths differ. Resolvers differ. Egress differs.

Your laptop sits on cafe Wi-Fi.
A spare host sits in some other region.
A GitLab runner sits wherever the executor lives.

Same curl binary. Three RTTs. Three stories.
Averaging them does not create truth.
It creates a number that survives copy-paste.

Corrected model. Label every probe with where.
Refuse unlabeled milliseconds in review.

I sometimes draft that labeling script with MonkeyCode.
The product offers free model access and a free server option.
Those two things help me rehearse a script, not certify a service.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A free server is another unlabeled region until I say so.
I write the region into the log file.
Then I refuse to treat that file as GitLab evidence.

Myth 3: A generated loop is a realistic API test

The claim. The model wrote fifty concurrent curls. Green means we scale.

The evidence. Concurrency without a workload model is theater.

Did it warm TLS sessions?
Did it send production headers?
Did it hit a cache that production will miss?
Did it authenticate like a real client?

If you skipped those questions, you ran noise.
Forked curl is not a traffic model.
A green exit code is not a capacity story.

Corrected model. Write the workload on paper first.
Then translate the paper into Python or curl.
The paper is the test. The script is only syntax.

Proposal, not a measured run:

• Actor: one health client, no auth
• Target: GET /health on staging only
• Arrival: five serial repeats, no think time
• Success: HTTP 200 and a non-empty body
• Out of scope: writes, JWT, pagination, fan-out

# proposal: serial health probe, not a load test
import os
import sys
import time
import urllib.request

url = os.environ["TARGET_URL"]
timeout = float(os.environ.get("PROBE_TIMEOUT", "5"))
samples = []

for i in range(5):
    t0 = time.perf_counter()
    with urllib.request.urlopen(url, timeout=timeout) as res:
        body = res.read()
        code = res.status
    dt = time.perf_counter() - t0
    samples.append((code, dt, len(body)))
    print(f"i={i} http={code} seconds={dt:.4f} bytes={len(body)}")

codes = {row[0] for row in samples}
if codes != {200}:
    sys.exit("non-200 in serial probe")
print("serial_probe_ok")

I call this a smoke probe in review comments.
I do not call it capacity planning.
Names matter when the MR title says perf.

Myth 4: Faster jobs mean a faster API

The claim. We cached the image. Latency improved.

The evidence. You improved job wall-clock.
The API never saw the change.

Smaller images help.
Sparse checkout helps.
Cache restore helps.

They cut CI minutes. They do not move TTFB.
If the diff only touches .gitlab-ci.yml, believe the pipeline dashboard.
Do not update the service SLO board.

Corrected model. Keep two conversations.
One is pipeline cost. One is service latency.
A stopwatch job can feed the first. Not the second.

Myth 5: One staging URL represents every caller

The claim. Staging answered in 80ms. Ship it.

The evidence. Who called, from where, with which payload?

Runners often sit near cloud APIs.
Your users may not.
A nearby 80ms can hide a distant 800ms.

/health is not /search.
/search is not /checkout.
One URL class cannot carry the rest.

Corrected model. Record PROBE_LABEL, region hint, and URL class.
If the label is missing, the number is trivia.
Trivia can stay in the log. It cannot gate a release.

Artifact: promote or reject table

I paste this table into reviews.
It is the whole method.

Signal you collected	You may conclude	You may not conclude
Job duration in the GitLab UI	The job is cheap or expensive	The API met an SLO
curl -w time_total on a runner	That runner's RTT to that URL	Customer RTT, p95, or capacity
Same script on a free server	The script runs without crashing	GitLab will see the same RTT
Model-drafted concurrent loop	Syntax might be valid	The workload matches production
HTTP 200 five times in a row	The smoke path is alive	The error budget is healthy
Image cache made the job faster	CI minutes probably dropped	Users got a snappier API

If the job name contains perf or sla, require the table.
Names lie. Columns do not.

A GitLab job that refuses to lie

This job records clocks. It does not gate an SLO.
Pin your own image digest before production use.
The YAML below is a proposal, not a benchmark run.

# proposal: labeled smoke probe, not an SLO gate
stages: [verify]

labeled_probe:
  stage: verify
  image: python:3.12-alpine
  variables:
    PROBE_LABEL: "gitlab-runner:unspecified-region"
    TARGET_URL: "$STAGING_HEALTH_URL"
  rules:
    - if: $STAGING_HEALTH_URL
  script:
    - python probe.py | tee probe.txt
  artifacts:
    when: always
    expire_in: 7 days
    paths:
      - probe.txt

Keep probe.py in the same commit as the job.
Reviewers should read the script, not the job name.
If STAGING_HEALTH_URL is unset, the job should no-op via rules.
Do not point a public runner at an internal URL.

Reproducible test plan

This is a plan you can rerun.
It is not a published latency study.

1. Freeze the URL in a CI variable. No hardcoded production host.
2. Freeze probe.py in git. No unreviewed paste.
3. Run once on a rehearsal host. Save rehearsal.txt.
4. Run once on GitLab. Save probe.txt.
5. Diff labels and clocks. Do not average RTTs.
6. If the labels disagree, discuss the network, not the API.
7. Only then decide whether you need a real load tool.

Commands I type locally, against a non-routable example:

export TARGET_URL="https://staging.example.invalid/health"
export PROBE_LABEL="rehearsal-host:unknown-region"
python3 probe.py | tee rehearsal.txt

git add probe.py .gitlab-ci.yml
git commit -m "Add labeled smoke probe, not an SLO gate"

Use a dedicated staging variable in GitLab.
Do not reuse CI_JOB_TOKEN for this probe.
The probe needs a public health URL or a protected variable you already own.

Limitations

This workflow will not save a latency incident.
It will not replace k6, Gatling, or your APM.

Shared runners are noisy neighbors.
One sample is not a distribution.
urllib will not replay production cookies.

A free model draft can invent flags you never asked for.
Read every line before it hits main.
A free server is not a GitLab runner and not your VPC.

I do not use this job as an SLO merge blocker.
I use it to stop lying in the MR description.
If you need percentiles, collect them outside this FAQ.

Who should skip this

Skip it if you already have a dedicated perf pipeline.
Skip it if legal needs signed latency evidence.
Skip it if the API cannot accept GitLab.com egress.

Do not paste tokens onto a rehearsal host.
Do not let a model invent TARGET_URL.
Do not rename a smoke probe to load_test for the badge.

What I want in the next review

Ask three questions. Then stop.

1. Which clock is this number?
2. Which network did the probe use?
3. Which workload did we write on paper?

If the author shrugs, the job is a stopwatch.
A stopwatch is useful. It is not an API test.
Export the probe into the repo before you quote a number.