Keep the Regex Writer Until Shadow Receipts Match

A regex parser should keep the production write path. An LLM extractor belongs in shadow until frozen fixtures agree. This article shows a local receipt harness for that cutover.

The harness compares structured fields instead of fluent prose. It writes a receipt file a reviewer can diff. Promotion waits for a clean receipt, not a demo.

The silent break regex never caused

Regex extractors fail loudly when lines do not match. LLM extractors fail by inventing keys or dropping ids. They also smooth timestamps that later break joins.

Downstream tickets then look complete while identifiers vanish. That failure hides inside pipelines that still exit zero. The JSON still parses, so CI stays green.

Only a field-level diff against a frozen baseline catches it. Reviewers need that diff in a file, not a chat. A missing run must show up as not_run, never as silence.

Freeze the logs before changing the parser

Do not start with a remote model call. Start with logs that already produced known tickets. Copy those lines into a fixture pack and freeze them.

Each fixture needs one input blob and one expected object. The expected object comes from the current regex path. That path is the baseline, not the enemy.

Guessed fields belong in comments, not in expected. A guessed baseline poisons every later receipt. Edit fixtures in a separate pull request from parser changes.

What a receipt must contain

A useful receipt stays small and hostile to vibes. It records fixture id, baseline fields, candidate fields, and mismatches. It also records whether the candidate call was skipped.

Skipped calls matter during every review. A missing run is not a pass. Reviewers should see not_run instead of an empty mismatch list.

Store receipts as CI artifacts beside the parser change. Chat paste is not an audit trail. A later revert needs the same bytes.

Numbered cutover workflow

1. Export twenty production-like log lines that already created tickets.
2. Store each line beside the regex JSON it produced last week.
3. Run the regex extractor again and confirm the baseline still matches.
4. Point a candidate extractor at the same fixtures in read-only mode.
5. Write receipt.json with field diffs and skip reasons.
6. Fail CI when any required field drifts or any fixture is skipped.
7. Keep the regex writer enabled until three consecutive receipts are clean.
8. Send candidate output to a staging queue only after that, never to prod.

The sequence stays in git on purpose. Screenshots of a lucky sample are not the record. Owners of the fixture pack should be named in the pull request.

Fixture layout

Save fixtures under fixtures/log_extract/. The layout below is a proposed template. It is not evidence from a named production fleet.

# fixtures/log_extract/inc-1042.yaml
id: inc-1042
source: syslog
input: |
  2026-09-02T04:11:08Z host=api-3 level=error
  request_id=r-9f2c timeout after 5000ms route=/checkout
expected:
  incident_id: r-9f2c
  host: api-3
  route: /checkout
  kind: timeout
  severity: error
required_fields:
  - incident_id
  - host
  - route
  - kind

Keep one incident per file. Mixed blobs hide which field drifted. Name files after ticket ids so receipts stay readable.

Baseline regex, kept boring

The baseline stays local and deterministic. The pattern below is labeled as a proposed example. Teams should replace the pattern with their real parser.

# baseline_extract.py — proposed example, not a live production parser
import re
from typing import Any

LINE = re.compile(
    r"(?P<ts>\S+)\s+host=(?P<host>\S+)\s+level=(?P<level>\S+)\s+"
    r"request_id=(?P<rid>\S+)\s+(?P<rest>.*)$",
    re.I,
)

ROUTE = re.compile(r"route=(?P<route>\S+)")


def extract_regex(text: str) -> dict[str, Any]:
    line = text.strip().splitlines()[-1]
    m = LINE.search(line)
    if not m:
        return {"kind": "unparsed"}
    rest = m.group("rest")
    route = ROUTE.search(rest)
    kind = "timeout" if "timeout" in rest.lower() else "error"
    return {
        "incident_id": m.group("rid"),
        "host": m.group("host"),
        "route": route.group("route") if route else "",
        "kind": kind,
        "severity": m.group("level").lower(),
    }

Run a sanity check before any candidate path is added:

python3 -c "from baseline_extract import extract_regex; print(extract_regex(open('fixtures/log_extract/inc-1042.txt').read()))"

If this print disagrees with expected, fix the fixture pack first. A broken baseline makes every candidate look worse than it is. Do not tune the model to match a stale expected object.

Candidate extractor as a narrow adapter

The candidate is an adapter, not a product tour. The function takes text and must return a dict. Network details stay outside the comparison core.

MonkeyCode can sit behind that adapter as one shadow lane. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The project offers free models for repeated fixture calls. It also offers a free server for bulk overnight shadow runs. This article does not name models, quotas, or hardware. Those details change and belong in current project docs. Read those docs before any shadow job is scheduled.

A proposed adapter shape looks like this. It is unexecuted here. Real endpoints may expect a different JSON envelope.

# candidate_extract.py — proposed adapter, not a vendor SDK
from __future__ import annotations

import json
import urllib.request
from typing import Any

PROMPT = """Return JSON only with keys incident_id, host, route, kind, severity.
Use kind=timeout when the log shows a timeout. Do not invent ids.
LOG:
"""


def extract_llm(text: str, url: str, timeout: float = 30.0) -> dict[str, Any]:
    body = json.dumps({"prompt": PROMPT + text, "format": "json"}).encode()
    req = urllib.request.Request(url, data=body, method="POST")
    req.add_header("Content-Type", "application/json")
    with urllib.request.urlopen(req, timeout=timeout) as resp:
        raw = json.loads(resp.read().decode())
    if isinstance(raw, dict) and "incident_id" in raw:
        return raw
    if isinstance(raw, dict) and isinstance(raw.get("output"), dict):
        return raw["output"]
    raise ValueError("candidate did not return a JSON object")

Point url at a local stub during unit tests. Point it at a free-model endpoint during shadow CI. Point it at a free-server job URL for the nightly bulk pack. The receipt script does not care which lane answered.

Local stub so the harness runs offline

A stub keeps the first pull request honest. It proves receipts fail closed without a network. The stub below echoes a parsed object and is labeled as a proposed example.

# stub_server.py — proposed local stand-in, not a production gateway
from http.server import BaseHTTPRequestHandler, HTTPServer
import json

from baseline_extract import extract_regex


class Handler(BaseHTTPRequestHandler):
    def do_POST(self) -> None:
        length = int(self.headers.get("Content-Length", "0"))
        payload = json.loads(self.rfile.read(length) or b"{}")
        text = str(payload.get("prompt") or "")
        marker = "LOG:"
        log = text.split(marker, 1)[-1] if marker in text else text
        body = json.dumps(extract_regex(log)).encode()
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)


if __name__ == "__main__":
    HTTPServer(("127.0.0.1", 8765), Handler).serve_forever()

Start it in one terminal, then aim the receipt script at it:

python3 stub_server.py
python3 shadow_receipt.py fixtures/log_extract --candidate-url http://127.0.0.1:8765

The stub will match the regex baseline on purpose. That is a wiring test, not a model test. Replace the URL only after this path already fails closed on a broken fixture.

Receipt compiler

The compiler is the artifact. It loads YAML fixtures, runs both extractors, and writes receipt.json. Install PyYAML, then run it from the repo root.

python3 -m pip install pyyaml
python3 shadow_receipt.py fixtures/log_extract --candidate-url "$SHADOW_URL"

#!/usr/bin/env python3
"""Shadow receipt for log extractors. Proposed local tool, not a benchmark."""
from __future__ import annotations

import argparse
import json
import sys
from pathlib import Path

try:
    import yaml
except ImportError:
    raise SystemExit("pip install pyyaml")

from baseline_extract import extract_regex
from candidate_extract import extract_llm

REQUIRED_DEFAULT = ["incident_id", "host", "route", "kind"]


def load_fixtures(folder: Path) -> list[dict]:
    files = sorted(folder.glob("*.yaml")) + sorted(folder.glob("*.yml"))
    out = []
    for path in files:
        data = yaml.safe_load(path.read_text())
        if not isinstance(data, dict) or "id" not in data:
            raise SystemExit(f"bad fixture: {path}")
        data["_path"] = str(path)
        out.append(data)
    return out


def field_diff(expected: dict, actual: dict, required: list[str]) -> list[dict]:
    mismatches = []
    for key in required:
        left = expected.get(key)
        right = actual.get(key)
        if left != right:
            mismatches.append({"field": key, "expected": left, "actual": right})
    return mismatches


def run_one(fix: dict, candidate_url: str | None) -> dict:
    text = str(fix.get("input") or "")
    expected = dict(fix.get("expected") or {})
    required = list(fix.get("required_fields") or REQUIRED_DEFAULT)
    baseline = extract_regex(text)
    base_mis = field_diff(expected, baseline, required)
    record = {
        "id": fix["id"],
        "fixture": fix["_path"],
        "baseline": baseline,
        "baseline_mismatches": base_mis,
        "candidate": None,
        "candidate_status": "not_run",
        "candidate_mismatches": [],
    }
    if base_mis:
        record["candidate_status"] = "blocked_by_baseline"
        return record
    if not candidate_url:
        return record
    try:
        cand = extract_llm(text, candidate_url)
        record["candidate"] = cand
        record["candidate_status"] = "ok"
        record["candidate_mismatches"] = field_diff(expected, cand, required)
    except Exception as exc:
        record["candidate_status"] = "error"
        record["error"] = str(exc)
    return record


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("folder")
    parser.add_argument("--candidate-url", default="")
    parser.add_argument("--out", default="receipt.json")
    args = parser.parse_args()
    fixtures = load_fixtures(Path(args.folder))
    if not fixtures:
        raise SystemExit("no fixtures found")
    url = args.candidate_url or None
    rows = [run_one(f, url) for f in fixtures]
    failed = [
        r
        for r in rows
        if r["baseline_mismatches"]
        or r["candidate_status"] != "ok"
        or r["candidate_mismatches"]
    ]
    receipt = {
        "fixture_count": len(rows),
        "failed_count": len(failed),
        "rows": rows,
    }
    Path(args.out).write_text(json.dumps(receipt, indent=2) + "\n")
    print(f"wrote {args.out} failed={len(failed)}")
    sys.exit(1 if failed else 0)


if __name__ == "__main__":
    main()

The exit code is the policy. A skipped candidate is a failure. A baseline drift is also a failure. That second case means the regex changed, or the fixture aged, and the pack needs an honest edit.

How to read receipt.json

Open the file in the pull request. Read failed_count before any model output. Then open the first failed row and look at field.

A drifted incident_id is a stop. A drifted kind can be a prompt bug. A candidate_status of error is infrastructure, not linguistics. Those three classes need different owners.

Do not average the mismatches into a single score. One bad identifier outweighs nine matching routes. The write path should stay on regex until identifiers hold.

Where free models and a free server belong

Short fixture packs belong on free models. Each call is small, repeated, and easy to retry. The point is field stability across many cheap runs, not a single impressive summary.

Larger shadow packs belong on a free server. Overnight logs are longer and should not sit in a laptop sleep cycle. The same receipt schema still applies. Only the adapter URL changes.

Do not mix those lanes inside one undocumented shell script. Record the URL source in the CI job name. Reviewers should see shadow-free-models or shadow-free-server in the workflow file.

A proposed GitHub Actions step looks like this. Secrets stay in the host, not in the receipt.

# .github/workflows/shadow-extract.yml — proposed workflow
name: shadow-extract
on:
  pull_request:
  schedule:
    - cron: "20 3 * * *"
jobs:
  receipt:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: python3 -m pip install pyyaml
      - name: regex baseline only
        run: python3 shadow_receipt.py fixtures/log_extract --out receipt-baseline.json
      - name: candidate shadow
        if: github.event_name == 'schedule'
        env:
          SHADOW_URL: ${{ secrets.SHADOW_URL }}
        run: python3 shadow_receipt.py fixtures/log_extract --candidate-url "$SHADOW_URL" --out receipt-nightly.json
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: extract-receipts
          path: receipt-*.json

The pull request job can skip the candidate URL. That still proves the baseline pack is healthy. The scheduled job is the actual shadow. A red nightly receipt should block promotion, not page the whole company.

Decision table for promotion

Promotion is a table, not a vibe. Values are qualitative on purpose. This table is not a latency benchmark and not a cost study.

Gate	Stay on regex writer	Keep shadow only	Allow staging writes
Fixture pack	Missing or hand-waved	Twenty frozen cases with owners	Pack still frozen, edits in a separate PR
Baseline receipt	Regex disagrees with expected	failed_count is zero on regex	Same, plus regex still in the write path
Candidate fields	Any incident_id drift	kind or route still noisy	required fields match on the whole pack
Run lane	No URL, local stub only	Free models on the small pack	Free server on the nightly bulk pack
Writes	Prod tickets from regex only	Receipt artifacts only	Staging queue, with regex still shadowing

If two columns still feel close, keep the regex writer. The cost of a delayed LLM cutover is extra CI minutes. The cost of a wrong incident_id is a missing page.

Limitations

This harness does not score fluency, tone, or summary quality. It only diffs required fields against a frozen expected object. A candidate can match every key and still omit a useful clause.

It does not measure tokens, dollars, or latency. Constants such as the thirty-second timeout are tripwires, not vendor SLOs. Changing them to force a pass is not a test.

It does not redact secrets. Logs with tokens or customer payloads need a separate scrubber before any remote URL is used. A receipt that stores raw input can become a leak.

The adapter body is a proposed JSON envelope. Real free-model and free-server endpoints may expect a different schema. Confirm that schema from current docs. Do not copy the envelope blindly.

The stub server always mirrors regex output. Agreement against the stub proves plumbing, not extraction quality. Nightly receipts against a live lane are the actual signal.

Who should not use this

Skip this workflow when the parser never writes anywhere. A one-off notebook does not need receipts. Skip it when a security team already forbids hosted inference for these logs.

Skip it when there is no regex baseline at all. The method needs a frozen expected object. A blank page is not a baseline.

Skip it when the output is free prose. Release notes and chat replies need different evaluation. This receipt format will not help there.

Skip it when nobody owns the fixture pack. An unowned YAML folder will rot, then CI will fail forever, then someone will delete the job.

What to do with a clean receipt

A clean receipt is permission to stage, not to delete regex. Keep both extractors for a full night of real logs. Compare ticket ids the next morning with the same required-field list.

If the nightly bulk pack needs a longer window, run that pack on the free server lane. Keep the small pull-request pack on free models so reviewers still get a fast signal. The useful comment in a pull request is the first mismatched field.

Run the receipt script on an existing fixture pack before any parser swap, and keep the JSON beside the change.