Prompt Registry & CI Eval Gates for Safe LLMs

Treat prompts like code: why a prompt registry matters

In 2026 the most common production incidents I see start with prompt changes, not model quality. Prompts drift fast: product teams tweak phrasing, marketing asks for tone changes, and a small single-word edit can quietly drop factuality or groundedness. If you manage prompts as text blobs in code or in an ad-hoc admin UI, regressions often surface hours later when users complain.

A prompt registry plus CI-style evaluation and a shadow rollout gives you the engineering controls to make prompt edits auditable, testable, and reversible — fast.

Quick checklist you can implement this week

1) Prompt registry: single source of truth

• Store prompt_id, immutable version (content hash or semantic version), author, changelog, stored golden-dataset pointer, model hint, and parameters (temperature, max_tokens).
• Keep immutable version content separate from a small mutable pointer table that maps environment aliases (prod, staging, canary) to a concrete version and rollout_bps.

2) CI-style eval gate

• Every PR that touches prompts runs an automated eval against a golden set measuring groundedness, factuality (or hallucination proxy), latency, and utility.
• Compare candidate vs. current production version on the same cases (paired per-case delta) and fail the build when deltas exceed thresholds (e.g., groundedness drop > 3%).

3) Shadow rollout + monitor

• Deploy changes to a shadow cohort (1–5% or mirrored shadow traffic) and compare live metrics before promoting or rolling back.
• Make rollbacks a single pointer update, not a redeploy.

Minimal schema and registry fragment

A tiny immutable version row plus a pointer table is all you need to start. Example version fragment (stored in your registry or git):

{
  "prompt_id": "search_rewrite_v2",
  "version": "1.4.0",
  "content_hash": "sha256:6a1b...",
  "author": "[e-mail]",
  "changelog": "Reduce verbosity; prefer explicit source citations",
  "params": { "temperature": 0.0, "max_tokens": 512 },
  "check": "groundedness>=0.88",
  "golden_dataset": "goldens/search_rewrite/500.jsonl"
}

A pointer (mutable) row decides what the environment serves:

{
  "prompt_id": "search_rewrite_v2",
  "environment": "prod",
  "version": "1.4.0",
  "rollout_bps": 500,    // 5% traffic
  "previous": "1.3.9",
  "updated_by": "ci-bot",
  "updated_at": "2026-09-11T16:03:00Z"
}

When the pointer flips, rollback is a single write to previous.

CI eval: pair candidate vs baseline (example)

The important detail is pairing: run the same golden cases with candidate and baseline, compute per-case deltas, and block promotion when the paired delta CI crosses your threshold.

A minimal GitHub Actions-style CI job:

name: Prompt CI
on:
  pull_request:
    paths:
      - 'prompts/**'

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.11'
      - name: Install deps
        run: pip install -r evals/requirements.txt
      - name: Run prompt eval
        run: python evals/run_eval.py --prompt-path prompts/search_rewrite_v2.yaml --golden evals/goldens/search_rewrite_500.jsonl --baseline-version 1.3.9 --metric groundedness --threshold 0.03

A compact Python sketch (pseudocode) for the eval runner's core logic:

# load candidate prompt and baseline prompt
# run N golden cases through model or offline judge
# compute groundedness per case (e.g. entailment/embedding overlap or token-based lexical grounding)
# compute per-case deltas: delta_i = candidate_score_i - baseline_score_i
# bootstrap 95% CI on delta vector; fail if upper bound < -threshold

if paired_ci_upper < -THRESHOLD:
    print('Regression detected: blocking merge')
    sys.exit(1)
else:
    print('Pass: no meaningful regression')
    sys.exit(0)

Notes:

• Use a hermetic capture mode where the CI compares candidate responses to a frozen baseline capture when possible (avoids rate limits and flakiness).
• If you must call live model endpoints in CI, run small suites and use deterministic settings (temperature=0).

Shadow rollout and automated rollback

Shadow rollouts (mirroring or hashing a small percent of traffic) let you test the candidate against real distribution without exposing users to it. Two useful patterns:

• Shadow mirror: duplicate requests to candidate and baseline; only baseline response is returned to the user, candidate is scored asynchronously.
• Canary hash: deterministic per-tenant hash modulo 10,000 compared to rollout_bps so assignments are sticky.

JavaScript selection pseudo-code:

function selectVersion(promptPointer, tenantId) {
  const bucket = hash(tenantId + ':' + promptPointer.prompt_id) % 10000;
  return bucket < promptPointer.rollout_bps ? promptPointer.version : promptPointer.previous;
}

If live metrics (groundedness, task-success, error-rate) degrade beyond configured gates at the canary stage, automated orchestration flips the pointer back to previous and triggers cache invalidation and alerting. Because the pointer change is a single write, rollback can be instantaneous.

Best practices and pitfalls

• Treat a prompt change as a release: require a PR, an author, a changelog, and an automated eval before pointing staging/production.
• Store the resolved version id on every request and trace/span. Without that join key you cannot attribute regressions to a prompt version.
• Bake compiled defaults and a last-known-good cache into your binary so registry unavailability does not break production.
• Keep metrics per-version (cost, latency, groundedness, task success) and surface head-to-head comparisons.
• Use paired-per-case statistics (bootstrap CI) rather than absolute floors alone to catch slow drifting regressions.

Closing: make rollbacks fast, intentional, and auditable

A tiny registry and a CI eval gate stops the class of silent regressions caused by prompt edits. It gives product, safety, and infra a shared signal, and makes rollbacks a single, auditable action instead of a frantic deploy. Start small: move five high-impact prompts into a registry, add an eval-on-PR job, and wire a 1–5% canary with monitoring. You’ll sleep better.

If you already have a prompt registry, what's the one eval you wish you had in your CI gate?