Half my eval regressions never touched the prompt file

I keep a tally. Over the last two quarters, a little more than half of the output regressions I traced in one RAG service came from a change that never went near the prompt directory. The eval gate was green the whole time. It was green because it never ran, or it ran against a dataset that did not exercise the thing that moved.

That is the bug I want to talk about. Not a specific incident. The shape of the coverage.

The trigger

Most eval gates are wired to fire on two things: a prompt change or a model change. Someone edits prompts/answer.md, CI runs the eval suite, the suite scores the new prompt against a dataset, and the merge is blocked if the score drops. That is a good gate. It catches the failure mode it was built for.

Look at the trigger, though. paths: ["prompts/**"]. The gate's entire theory of risk is "output changes when the prompt changes." That theory is incomplete, and the gap is not small.

The hole

Your model's output is a function of a lot of inputs. The prompt is one. Here are others, none of which touch the prompt file:

• The tokenizer. A dependency bump can swap it. The same text then becomes a different number of tokens, your context budget shifts, and long inputs get truncated at a different boundary. Anthropic's own model docs note that the tokenizer introduced with a recent model generation produces "roughly 30% more tokens" for the same text than earlier models (Anthropic models overview). A 30% swing in token accounting is not a rounding error. It moves what fits in the window.
• The retrieval index. Rebuild it, re-embed with a new model, or change the chunker, and the same query returns the same documents in a different order, or returns different documents. The prompt template is byte-for-byte identical. The context inside it is not.
• The tool schema. Rename a parameter, change an enum, tighten a JSON schema, and the model's tool calls change. The prompt that describes the tool may not have changed at all.
• A config default. Temperature, top_p, max_tokens, a response_format flag, a system-vs-developer role default in an SDK. Bump the SDK minor version and a default can flip under you. Nothing in your repo shows a diff.
• The provider model behind an alias. You pinned a moving alias (a classic gpt-4o-style pointer) instead of a dated snapshot. The provider points that alias at a newer build. Your code did not change. Your outputs did.

Every one of those can regress the output. Not one of them trips a paths: ["prompts/**"] filter.

Two failure modes, not one

Separate them, because the fix differs.

Mode one: the gate does not run. The change lives in requirements.txt or services/retrieval/index.py, the path filter does not match, CI skips the eval job entirely. Silent.

Mode two: the gate runs and passes anyway. This one is sneakier. Say the tool schema changed. The eval suite runs, but the eval dataset is thirty prompt-quality cases that never invoke that tool. Full marks. Green check. The regression ships behind a passing gate, which is worse than no gate, because now you trust it.

Both modes have the same root cause. Gate coverage was defined by "what changed in the prompt directory" instead of "what can move the output."

The input list

Here is the check I actually run. It is boring and it works.

List every input that can affect the output. Prompt, model id, model version behind the alias, tokenizer, decoding params, retrieval index, embedding model, chunker, tool schemas, SDK version, and any upstream service that shapes context. For each one, answer a single question: if this changes, does an eval run get triggered, and does the eval dataset exercise it?

Two columns. Triggered yes/no. Exercised yes/no. Anything that is no in either column is a hole. You will find holes. The first time I did this for the RAG service, seven of the eleven rows were "no" on at least one column. The prompt row was the only one that was solidly "yes" on both, which is exactly why the prompt regressions were the ones we always caught and the rest were the ones that bit us.

You do not need tooling to make this list. A text file is fine. The value is in forcing yourself to name the inputs, because the ones you forget to name are the ones with no gate.

The canary

The lightweight fix is a canary eval that runs on any merge to the service, not just prompt merges. Widen the trigger from the prompt directory to the whole service path. Keep the eval small and fast so it is cheap enough to run every time.

# .github/workflows/canary-eval.yml
name: canary-eval
on:
  push:
    branches: [main]
    paths:
      - "services/answer/**"   # the whole service, not just prompts/**
jobs:
  canary:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - run: python -m eval.canary --n 40 --fail-under 0.9

The canary itself is deliberately small. Forty fixed cases, chosen to exercise the surfaces that the big prompt-quality suite ignores: at least a handful that force a retrieval hop, a handful that force a tool call, and a couple with long inputs that live near the context boundary. Pass rate in, threshold out.

# eval/canary.py
import sys, argparse
from statistics import mean
from myservice import answer      # the real service entrypoint
from eval.dataset import CANARY   # 40 fixed cases: retrieval + tools + long inputs

def score(case) -> float:
    out = answer(case.query)
    return float(case.check(out))  # 1.0 pass, 0.0 fail

def main() -> int:
    p = argparse.ArgumentParser()
    p.add_argument("--n", type=int, default=40)
    p.add_argument("--fail-under", type=float, default=0.9)
    args = p.parse_args()
    rate = mean(score(c) for c in CANARY[: args.n])
    print(f"canary pass rate: {rate:.2f}")
    return 0 if rate >= args.fail_under else 1

if __name__ == "__main__":
    sys.exit(main())

This is not your full eval suite. It is the smoke test that runs when someone bumps a dependency or rebuilds the index and does not think of it as a model change. Forty cases run in well under a minute for most services, which is the whole point: cheap enough that you never argue about whether to run it.

Pinning

The canary catches the changes that live in your repo. Two changes do not live in your repo, and you have to handle them separately.

First, pin the provider model to a dated snapshot rather than a moving alias, where the provider offers one. Anthropic's model docs are explicit that for models before their 4.6 generation, the alias entries are "convenience pointers that resolve to a dated model ID" (models overview), and OpenAI's classic aliases have historically moved to newer builds under the same name. A moving pointer is convenient right up until it moves during a release you did not tag as a model change. Pin the dated id in config, and treat bumping it as a change that runs the full suite.

Second, alert when the alias moves. If you must run against an alias, record the resolved model version (most providers return it in the response metadata) and diff it on a schedule. When the alias starts resolving to a new build, that is a model change that happened with zero commits on your side. You want a page, not a surprise in next week's quality numbers.

Neither of these is heavy. A pinned string in a config file and a cron job that compares a recorded version field. The cost is minutes. The thing it protects against is a class of regression you otherwise cannot see, because there is no diff to review.

What I'd check first

1. Read your eval workflow's trigger. If it says paths: prompts/** and nothing wider, every non-prompt input to your model is ungated. That is the hole, in one line.
2. Grep your service for the model string. If it is a moving alias and not a dated snapshot, you are trusting the provider not to move it between your releases. Log the resolved version so at least you would notice.
3. Open your eval dataset and count how many cases force a tool call and a retrieval hop. If the answer is near zero, your gate can pass while the tool and retrieval surfaces regress. Green does not mean covered.