FAQ: Five Myths About Treating a Scratch Box as CI

Did the agent just prove your GitLab job?

I keep seeing that claim on merge requests.

A green scratch shell is not a runner.

Why this keeps happening

Pipelines feel slow. Chat feels fast.

So people replay script: on a throwaway box.

Then they paste that output into the MR.

Does that output share identity with gitlab-runner?

No. It shares a prompt. That is different.

GitLab still evaluates .gitlab-ci.yml on the server.

It still injects CI variables for that job.

It still picks a runner, an image, and a graph.

A free model can help you read that file.

A free server can give you a disposable shell.

That pair is a lab. It is not CI.

The lab, not the runner

I clone the repo at a known SHA.

I pick one job name from the YAML.

I replay the script: lines by hand.

Then I write down every mismatch I find.

MonkeyCode is one option with free model access and a free server.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I use that box to interrogate the job spec.

I do not ask it to replace the runner.

Ready for the myths people still repeat?

Myth 1: "Same language runtime means same image"

The claim. I ran Python 3.12 on the scratch box. The job uses Python. Ship it.

What the YAML actually says. Look at image:. Look at the tag. Look at a digest if you pinned one.

Corrected model. The runner starts the job image. Not your PATH. Not the agent's package cache.

Ask the boring questions:

• Is the tag identical, including the digest?
• Did you inherit an entrypoint: you never ran?
• Are there services: the scratch box never started?

Inspect the file at the same commit:

git rev-parse HEAD
git show HEAD:.gitlab-ci.yml

If the job pins python:3.12.6-slim@sha256:..., laptop Python is irrelevant.

So is a random slim image the agent pulled today.

Myth 2: "The agent installed deps, so the runner will"

The claim. pip succeeded in chat. CI will have the same site-packages.

What actually happens. A runner job starts clean, unless you defined cache:.

before_script: is the contract. Chat history is not.

Corrected model. Dependencies exist only if the job installs them, caches them, or bakes them into the image.

Checklist:

1. Copy before_script and script verbatim.
2. Do not add extra pip install "just to help."
3. Record every extra package the agent sneaked in.
4. Check cache:paths and cache:key in the job.

If the agent "fixed" ImportError by installing a library, you learned a gap.

You did not learn that GitLab will install it.

Myth 3: "curl worked here, so the job can reach that host"

The claim. The scratch server reached the API. The runner will too.

What actually happens. Runners often sit on isolated networks.

Some cannot hit the public internet at all.

Some cannot hit production. Some reach only GitLab and a registry.

Corrected model. Network policy is part of the job identity.

Ask:

• Does the job need per-build networking flags?
• Do services resolve only via Docker DNS names?
• Is the hostname only in your laptop /etc/hosts?

A free server with open egress is a different planet.

Treat a successful curl as a hint. Never as proof.

# example only: record where you are, not where CI is
curl -sS -o /dev/null -w "%{http_code} %{url_effective}\n" https://example.invalid || true
env | grep -E '^(http|HTTP|NO)_PROXY=' || true

Did that hostname even exist in the job YAML?

If not, you tested your curiosity. Not the job.

Myth 4: "I exported the token, so CI variables are tested"

The claim. I set DATABASE_URL in the scratch shell. Variables work.

What GitLab actually injects. CI variables have scopes you cannot hand-wave.

Protected. Masked. File type. Environment scoped. Branch scoped.

CI_JOB_TOKEN is minted per job. You cannot honestly fake it.

Corrected model. Pasting a secret into a scratch session tests leakage.

It does not test GitLab's variable expansion rules.

Never paste production secrets into a free box.

Use throwaway values. Mark them as fakes in the log.

Then verify the YAML references, names only:

# example only — not a real pipeline
test:unit:
  image: python:3.12-slim
  variables:
    PYTEST_ADDOPTS: "-q"
  script:
    - pytest

Ask GitLab, not the chat:

• Is the variable protected, and is this branch protected?
• Is it limited to environment:production?
• Is it a file variable mounted at a path?

If you needed a real token to "prove" the job, stop.

That is an access problem, not a CI proof.

Myth 5: "The shell passed, so the pipeline YAML is valid"

The claim. The commands ran. Therefore rules: are fine.

What GitLab still evaluates. rules, workflow, needs, artifacts, retry.

Also interruptible, resource_group, and child pipelines.

A bash replay does not compile that graph.

Corrected model. Two different machines sit in your head.

Machine A: does this command exit 0?

Machine B: would GitLab even schedule this job?

You can pass A and skip B entirely.

rules:if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH will not run on your scratch SHA.

needs: ["build"] will not fetch artifacts you never produced.

Green shell. Missing graph. That is the myth.

Artifact: job replay sheet

Do not argue from screenshots.

Fill this table for one job, one SHA.

Check	Scratch box	.gitlab-ci.yml job	Match?
git SHA
job name
image name + digest
entrypoint
services
before_script lines
script lines
extra packages the agent added
cache key / paths
variables (names only)
protected / masked / file flags		n/a on scratch
network egress
artifacts:paths
needs / rules	not evaluated		no

If any row is "no", the scratch run did not cover the job.

It covered a neighboring command. Write that down.

Example: extract one job (template)

This script is a template. Label it unexecuted against your repo.

It prints image, services, scripts, and variable names.

It does not talk to GitLab. It does not mint CI_JOB_TOKEN.

#!/usr/bin/env python3
"""Extract one GitLab CI job. Example only. Requires PyYAML."""
import sys
import yaml

HIDDEN = {
    "include",
    "stages",
    "variables",
    "workflow",
    "default",
    "image",
}


def load_ci(path):
    with open(path, encoding="utf-8") as fh:
        data = yaml.safe_load(fh)
    if not isinstance(data, dict):
        raise SystemExit("expected a mapping in .gitlab-ci.yml")
    return data


def dump_job(ci, name):
    job = ci.get(name)
    if not isinstance(job, dict):
        raise SystemExit(f"no job named {name}")
    default = ci.get("default") or {}
    image = job.get("image", ci.get("image") or default.get("image"))
    print(f"job: {name}")
    print(f"image: {image}")
    print(f"services: {job.get('services')}")
    print(f"needs: {job.get('needs')}")
    print(f"rules: {job.get('rules')}")
    print(f"cache: {job.get('cache')}")
    print(f"artifacts: {job.get('artifacts')}")
    vars_ = {}
    vars_.update(ci.get("variables") or {})
    vars_.update(job.get("variables") or {})
    print("variable_names:", sorted(vars_))
    for key in ("before_script", "script", "after_script"):
        print(f"{key}:")
        for line in job.get(key) or []:
            print(f"  {line}")


if __name__ == "__main__":
    if len(sys.argv) != 3:
        raise SystemExit("usage: extract_job.py .gitlab-ci.yml JOB")
    dump_job(load_ci(sys.argv[1]), sys.argv[2])

Run it like this:

python3 -m pip install --user pyyaml
python3 extract_job.py .gitlab-ci.yml test:unit

Then replay only the printed before_script and script.

No extra flags. No helpful upgrades from the model.

Log mismatches in the table. That is the artifact.

Commands I run next to the table

Keep the evidence next to the SHA.

git status --short
git rev-parse HEAD
git branch --show-current

If the agent edited files, stop immediately.

You are no longer replaying the job.

You are replaying a cousin of the job.

Restore and start over:

git diff
git checkout -- .

Still tempted to trust the chat transcript?

Read git diff. Then read the filled table.

The transcript is not the job spec.

Limitations

This workflow does not start a real runner.

It does not expand include: from other projects.

It does not evaluate rules:if against GitLab predefined variables.

A simple YAML load also misses !reference tags unless you add a loader.

Includes from a CI/CD catalog will look missing locally.

That is a limitation. Write it down. Do not hide it.

A free model may misread YAML anchors and hidden keys.

A free server may lack Docker, Kubernetes, or your private registry.

I do not claim hardware, quotas, or model names here.

Those change. The job contract does not care.

Who should not use this approach

Do not use a public scratch box for production secrets.

Do not use it as a self-hosted runner replacement.

Do not use it if compliance requires isolated runners only.

Do not skip the real pipeline on protected branches.

If you need GitLab-issued artifacts, wait for GitLab.

The lab is for finding mismatches early.

The pipeline is still the contract.

What I want in the MR

Paste the table. Paste the SHA.

Paste the job name. Paste the mismatches.

Skip the "it worked on the free box" slogan.

If you spin up a disposable lab, keep that mismatch table in the MR.