FAQ: Your Agent Did Not Replay That GitLab Job

Did the chat just declare your pipeline fixed?
I keep seeing that claim on red merge requests.
An agent called an API, then sounded very certain.

Was that a GitLab job, or a story about one?
I keep asking that before I trust the transcript.

The pattern I keep hitting

You paste a failed job log into a coding agent.
Maybe you also clone the repo onto remote compute.
The model reruns one unit test and prints green.

Does that merge request widget care about chat?
GitLab still owns the graph, the variables, and the trace.

Why this FAQ exists

Tool-calling demos are loud again this week.
Agents fetch JSON, open shells, and look extremely busy.
Looking busy is not the same as a pipeline replay.

Can a Jobs API response explain a flake by itself?
Only the fields GitLab actually returned can do that.
A confident paragraph is still not a job trace.

I use agents as scratch pads for failing tests.
I refuse to treat those pads as GitLab runners.

Myth 1: The Jobs API replayed the failed job

The claim sounds reasonable during a red incident.
Developers call the Jobs API and feel finished.
They read failure_reason and then they stop thinking.

Did that HTTP request execute your script block again?
It only read metadata about a past GitLab execution.

Here is the corrected mental model I use.
Keep these three objects in separate buckets.

• An API read returns status, duration, and failure_reason.
• A trace read returns log text GitLab stored.
• A replay is a new job on a real runner.

Do not collapse those three objects inside a chat summary.
Chat summaries hide which object you actually have.

# This is a read. It is not a replay.
# Use a read-only token. Never paste it into the model.
curl --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  "$CI_API_V4_URL/projects/$PROJECT_ID/jobs/$JOB_ID"

That JSON never ran before_script on your runner.
It never mounted cache, services, or runner tag selection.

If the agent called the API, ask one sharper question.
Which response fields came back, quoted verbatim from JSON?

Myth 2: A pasted job log is the full environment

The log is only a text artifact GitLab kept.
It is not /proc, and it is not the process environment.

What usually never appears in the paste you hand over?
I keep this gap list beside the paste.

• Protected variable values stay hidden by design always.
• Service container health shows up only as late errors.
• Cache hit keys appear only if a script echoed them.
• Runner tags and executor details often get truncated.
• The exact image digest appears only if you printed it.

So the model fills gaps with plausible sounding guesses.
That fill is fiction until a command proves it.

Corrected model: quote a log line, or drop the claim.
I want the first failing command and its exit code.
I want the image name I actually echoed from the job.

Everything else remains a hypothesis about the runner.
Hypotheses do not belong in the merge request description.

Myth 3: Green tests on a scratch box equal a green MR

This myth bites people using free remote compute.
A remote box is useful for a fast bisect.
That remote box is still not gitlab-runner.

Did you run the same language runtime version there?
Did you export the same CI_* variables into the shell?
Did you start the same services: containers beside it?

If any answer is no, you only ran a cousin environment.
A cousin passing still does not update the MR widget.

Corrected model: scratch compute is only a lab bench.
The merge pipeline remains GitLab's machine, every time.

I still use a scratch box for unit test bisection.
Then I push a job that prints a runner fingerprint.

Myth 4: Rewritten YAML means GitLab expanded the graph

Agents love to fix a messy .gitlab-ci.yml file.
They run a local YAML linter and then they smile.

Did GitLab expand include, rules, and workflow for you?
A local parse cannot see project CI inputs at all.

It cannot see downstream trigger status on other projects.
It cannot see protected-branch rule evaluation either.

Corrected model: lint checks syntax, not pipeline semantics.
The pipeline editor and a real pipeline remain different.

# Syntax only. GitLab may still build a different graph.
python -c "import yaml,sys; yaml.safe_load(sys.stdin)" < .gitlab-ci.yml

Need the real job graph after a YAML edit?
Create a pipeline on the branch under review.
Read that pipeline's jobs list, not the chat.

Artifact: fingerprint the runner, then diff the scratch box

I want a reproducible check, not a confident vibe.
Add a job that prints facts the agent must quote.

# proposed snippet — not a production template
fingerprint:
  stage: test
  image: python:3.12-slim
  script:
    - echo "EXPECTED_IMAGE=python:3.12-slim"
    - |
      python - <<'PY'
      import os, platform
      keys = [
          "CI_JOB_NAME",
          "CI_JOB_STAGE",
          "CI_RUNNER_DESCRIPTION",
          "CI_RUNNER_TAGS",
          "CI_PIPELINE_SOURCE",
          "CI_COMMIT_SHA",
      ]
      print("python", platform.python_version())
      print("platform", platform.platform())
      for k in keys:
          print(f"{k}={os.environ.get(k, '')}")
      PY
    - echo "PWD=$(pwd)"

Run the same probe on your scratch server next.
Save both outputs before you claim a reproduction.
Diff them before you write the merge request comment.

# Example commands. Adjust paths for your machine.
python probe.py > /tmp/scratch_fingerprint.txt
# Save the GitLab job section that printed the same keys.
diff -u /tmp/runner_fingerprint.txt /tmp/scratch_fingerprint.txt

No diff on SHA, platform, and Python version?
Then a unit-test rerun is worth talking about.

A diff on runner tags or missing CI_* values?
You did not replay the job, so stop claiming that.

Decision table I keep beside the terminal

I keep this decision table beside the terminal.
It stops the chat from laundering missing evidence.

Claim in the chat	Evidence I require	If that evidence is missing
Job failed inside pytest	Log line with node id and exit	Re-run only that node on GitLab
Wrong Python on the runner	Fingerprint python version line	Pin the image in .gitlab-ci.yml
API says the job failed	Job JSON status plus failure_reason	Open the trace and quote one line
I reproduced it remotely	Scratch fingerprint matches the runner	Call it a cousin run and move on
The YAML is valid now	A real pipeline created on the branch	Do not merge on local lint alone

That table is the whole method, start to finish.
The agent may draft the left column quickly.
You still fill the middle column every single time.

A tiny parser for citable log lines

Do not hand the model a twelve-thousand-line paste.
Extract facts first, then ask questions against facts.

# extract_trace_facts.py — example parser, not GitLab itself
import re
import sys

FAIL = re.compile(r"(ERROR|FAILED|exit code \d+)", re.I)
IMAGE = re.compile(r"(Using Docker image|EXPECTED_IMAGE=)", re.I)

def main(path: str) -> None:
    facts = []
    with open(path, encoding="utf-8", errors="replace") as fh:
        for i, line in enumerate(fh, 1):
            if IMAGE.search(line) or FAIL.search(line):
                facts.append(f"L{i}: {line.rstrip()}")
    print("CITABLE_FACTS")
    for row in facts[-40:]:
        print(row)
    print("UNPROVEN_UNLESS_FINGERPRINTED")
    print("- service containers were healthy")
    print("- cache keys matched the failed job")
    print("- protected variables were identical")
    print("- runner tags selected the same executor")

if __name__ == "__main__":
    main(sys.argv[1])

Feed only that output into the coding model.
Refuse any cause that lacks an L line prefix.

Where does a free coding environment actually help here?
On the scratch side of that decision table.
Not on the GitLab side, and not in the MR widget.

MonkeyCode is one lab bench I will allow for probe.py.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The project is open source, with free model access available.
A free server option exists for that same lab-bench role.
If you try that loop, keep secrets off the box.

I still refuse to treat that box as gitlab-runner.
GitLab remains the only machine that can paint the widget.

What I will not claim in this FAQ

I will not name models I have not verified here.
I will not quote token quotas I cannot show you.
I will not publish duration, hardware, or benchmark theater.

Job JSON fields above match GitLab's public Jobs API.
If your self-managed instance differs, trust your instance.

Who should skip this scratch-server loop

Skip the remote bisect in the cases listed below.
Those flows need GitLab's machine, not a lab bench.

• Protected-branch pipelines that inject production secrets should skip this.
• Compliance jobs that need attested GitLab runners should skip this.
• Release flows that require artifact provenance should skip this too.
• Failures caused by network policy around the runner will not reproduce.
• Anyone about to paste CI_JOB_TOKEN into chat should stop.

That last item is not a cute myth to debate.
That is a credential leak with extra conversational steps.

The mental model I keep after the chat ends

Ask four questions before you believe the agent.
Write the answers in the merge request, not the chat.

1. Did GitLab execute this, or did we only read metadata?
2. Which log line supports the cause you just named?
3. Does the scratch fingerprint match the runner fingerprint?
4. Did a new pipeline confirm the YAML graph you edited?

If you cannot answer, the pipeline is still red.
The chat can wait for a real GitLab job.
A green cousin run is still not a merge.