FAQ: Five Myths About Agent Tests Replacing GitLab CI

CI is still red on the merge request. The agent says every test already passed. Which report should you actually trust tonight?

I keep hearing the same swap this month. A chat window looks greener than GitLab. People treat that window as a pipeline.

Want the blunt version up front? Agent tests are a scratch rehearsal. Your .gitlab-ci.yml is the merge contract. Rehearsal is useful. It does not sign the merge.

Why this claim spreads so fast

Agents print pytest in a tidy block. Humans read tidy blocks as truth. Did you capture the image tag though?

Did you capture GitLab services:? Did you capture pytest markers? Chat logs hide those knobs on purpose.

I wrote this as a myth FAQ. Five repeated claims. Five checks. One inventory you can rerun.

No timings. No pass-rate theater. Just questions your pipeline already answers.

Myth 1: "pytest passed, so the GitLab job would pass"

The claim I hear

The agent ran the test command. The exit code was zero. Therefore the test job is green.

What I actually look at

Which interpreter ran on that box? Which extras got installed? Which markers were selected?

A throwaway shell often uses whatever Python is around. Your GitLab job pins an image. Those are not the same proof.

# example only — labeled, not a real project file
test:
  image: python:3.12-slim
  services:
    - name: postgres:16
      alias: postgres
  script:
    - pip install -e ".[test]"
    - pytest -m "not slow" --cov=src --cov-fail-under=80

See the gaps in one glance? Image. Service. Extras. Markers. Coverage gate.

Did the agent start Postgres? Did it load pytest-cov? Did it honor -m?

Corrected mental model

A green agent run proves one argv. It does not prove the job definition.

Ask this every time, out loud. What exact command ran? What exact image?

Myth 2: "One green job equals the whole pipeline"

The claim I hear

Tests are the real gate. Lint is noise. Typecheck is ceremony. Audits can wait.

What I actually look at

Open .gitlab-ci.yml. Count jobs. Count stages. Count needs: edges.

A typical merge request pipeline is not one pytest call:

• lint
• typecheck
• unit
• integration
• build
• maybe a dependency audit job

Did the agent run ruff? Did it run mypy? Did it run the Node job too?

GitLab rules: also matter. Branch pipelines are not merge-request pipelines. Did the agent simulate CI_PIPELINE_SOURCE?

# example only — rules the agent will not invent for you
unit:
  stage: test
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
  needs: ["lint"]
  script:
    - pytest tests/unit -q

If needs: skipped lint, you did not run the graph. You ran a souvenir cell.

Corrected mental model

Pipelines are a matrix of promises. One green cell is not the matrix.

I do not merge on a single cell. Should you?

Myth 3: "Green means the coverage gate passed"

The claim I hear

Tests executed. Assertions held. Coverage must be fine then.

What I actually look at

Coverage is a policy, not a vibe. --cov-fail-under lives in CI script text.

GitLab also parses a coverage regex from the job log. Settings and the coverage keyword both matter. Did the agent print a line GitLab would parse?

Omit paths matter. Branch coverage matters. Was pytest-cov even installed?

# inventory — run where the agent worked
python -V
python -c "import pytest; print(pytest.__file__)"
python -c "import pytest_cov; print('cov-plugin', pytest_cov.__file__)" || echo "no pytest-cov"
pytest --collect-only -q
# labeled example; do not treat this output as a gate

If coverage never loaded, the gate never ran. Zero is not eighty. Chat said "all passed." Passed what?

Corrected mental model

Passing tests without the gate is incomplete evidence. Copy the flag from YAML. Then rerun it.

Also copy the coverage regex. If GitLab cannot parse the log, the badge lies later.

Myth 4: "The agent installed packages, so the lockfile is honest"

The claim I hear

The install succeeded. Imports worked. Dependencies are therefore good.

What I actually look at

Did the command honor uv.lock? poetry.lock? package-lock.json?

Or did it run a loose pip install pytest requests? Those are different universes. One is reproducible. One is a souvenir freeze.

GitLab jobs usually pin the installer. poetry install --sync is not pip install -U.

# example comparison — checklist, not a vendor tool
test -f uv.lock && echo "uv.lock present"
test -f poetry.lock && echo "poetry.lock present"
test -f package-lock.json && echo "package-lock present"
pip freeze > /tmp/agent-freeze.txt || true
# diff freeze against the lock on your branch, not in chat

Did the agent mutate poetry.lock and forget to say so? Open git status. Then open git diff --lock.

Corrected mental model

Install success is not lockfile fidelity. Always diff freeze against the lock.

If the lock changed, the pipeline must install from that lock. Not from memory.

Myth 5: "Required checks are optional if the agent already tested"

The claim I hear

Waiting on GitLab slows people down. The agent already proved safety. Uncheck the boxes.

What I actually look at

Required checks are not etiquette. They pin the runner tag. They pin the image. They pin the job name reviewers expect.

They also pin who can skip. Merge trains care about that. Protected branches care too. Approval rules care about pipeline success.

Would you disable them because a laptop was green? Then do not disable them for a chat transcript.

GitLab "pipelines must succeed" is a contract with the branch. The agent is not a party to that contract.

Corrected mental model

Treat agent output as a draft signal. Treat CI as the merge contract.

Scratch runs can save you a cycle. They cannot sign the merge request.

Where a free session still helps

I still want a throwaway shell sometimes. I want a model to draft the inventory, not to bless the MR.

That is where MonkeyCode's free model access and free server fit for me. I paste the YAML. I ask for a gap list. I run the inventory commands. Then I throw the box away.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I do not use that box as a GitLab runner. I use it to list missing flags. Real proof still comes from GitLab runners.

Remove the product and this checklist still works. That is the point.

Artifact: the CI-contract inventory

Do this on the same tree the agent used. Then compare it to GitLab. Do not skip the remote run.

Step 1 — freeze the runtime

#!/usr/bin/env bash
# ci_inventory.sh — example workflow, not a vendor tool
set -euo pipefail
mkdir -p /tmp/ci-inventory
{
  echo "pwd=$(pwd)"
  echo "user=$(id -un)"
  echo "python=$(command -v python3 || command -v python || true)"
  python3 -V 2>/dev/null || python -V || true
  echo "node=$(command -v node || true)"
  node -v 2>/dev/null || true
  echo "--- installer hints ---"
  command -v poetry && poetry --version || true
  command -v uv && uv --version || true
  echo "--- pip freeze ---"
  pip freeze 2>/dev/null || true
} > /tmp/ci-inventory/runtime.txt
pytest --collect-only -q > /tmp/ci-inventory/collect.txt 2>&1 || true
echo "wrote /tmp/ci-inventory"

Label it again. This is an example. It talks to no vendor API. It proves nothing until you compare files.

Step 2 — extract job names from GitLab CI

# example only: list jobs from .gitlab-ci.yml
# requires PyYAML in *your* environment
python3 - <<'PY'
# proposal / unexecuted example
from pathlib import Path
import sys
try:
    import yaml
except ImportError:
    sys.exit("install pyyaml locally; do not improvise on prod")

raw = Path(".gitlab-ci.yml").read_text()
data = yaml.safe_load(raw) or {}
reserved = {"stages", "variables", "include", "workflow", "default", "image"}
for name, body in data.items():
    if name in reserved or str(name).startswith("."):
        continue
    if not isinstance(body, dict):
        continue
    print(
        name,
        "image=", body.get("image"),
        "services=", body.get("services"),
        "rules=", bool(body.get("rules")),
        "needs=", body.get("needs"),
    )
PY

Did the agent name those jobs in order? Usually no. That silence is the finding.

Hidden includes will fool this parser. Anchors will fool it too. Read the rendered pipeline in GitLab. Then trust that view more than this sketch.

Step 3 — decision table

Agent evidence	CI contract question	Merge on it?
pytest exit 0	Same image and Python minor?	No, not yet
"installed deps"	Same lockfile command as YAML?	No, not yet
collected N tests	Same markers, paths, and CI_* vars?	No, not yet
no coverage line	--cov-fail-under and GitLab regex?	No
one job mentioned	All stages and needs: satisfied?	No
inventory matches YAML	Required MR pipeline jobs green?	Then maybe

Read the last row twice. Matching inventory is necessary. It is not sufficient.

You still need the actual pipeline. On the actual runners. Against the actual merge request ref.

A twenty-minute drill

1. Paste the exact test argv from the agent log.
2. Run ci_inventory.sh in that same tree.
3. Diff /tmp/ci-inventory/runtime.txt against the job image docs.
4. Tick every job printed from .gitlab-ci.yml. A missing tick means missing proof.
5. Push the branch. Wait for GitLab. Do not skip the wait.

What failed for you last time? Image drift? Missing Postgres? A marker the agent dropped?

Write that failure on the MR. Not "the agent said it was fine." Reviewers can replay a job. They cannot replay a vibe.

Limitations

This workflow does not prove production behavior. It does not prove load.

It does not prove secret scanning. It does not prove runner tags. It does not prove cache keys.

I am not publishing timings. I am not publishing pass rates. I am not claiming a free box matches shared runners.

Free model replies can omit flags. Free servers are not your fleet. Do not paste customer data into either.

The YAML parser above is a sketch. Real GitLab config includes include:, anchors, and rules:.

Hidden jobs will fool a naive parse. workflow: rules: can drop the whole pipeline. Read the rendered config.

Local tools can help you lint YAML. I still want the remote run on GitLab. Local green is still rehearsal.

Who should not use this approach

Do not use agent tests as a bypass. Compliance teams should refuse that swap.

Do not use it if you cannot read the CI file. The file is the whole point.

Do not use a free server for customer data. Inventory the toolchain only. Keep fixtures synthetic.

If your merge requires signed provenance, skip the shortcut entirely. Wait on the real jobs.

If you cannot tell pytest from the pipeline graph, stop. Learn the YAML first. Then invite the agent back.

What I want you to remember

The agent can draft a gap list. GitLab remains the contract.

Green chat is a hint. Green required jobs are the receipt you merge on. Which one are you showing reviewers?

Draft the inventory. Throw the scratch box away. Merge on GitLab, not on a transcript.