Catch Prompt Regressions Before They Ship: Testing an Invoice-Extraction Prompt with Promptfoo and GitHub Actions

Zack Thompson — Sun, 25 Oct 2026 04:06:13 +0000

This article was written with help from an AI assistant; all code was tested and the text reviewed. Example repo: https://github.com/zackthompson317/promptfoo-invoice-extraction-ci (Promptfoo 0.125.0, Node 22.22+)

The problem: a prompt that breaks quietly

Say your accounts-payable tool sends every incoming invoice to an LLM and asks for JSON: invoice number, vendor, date, currency, line items, subtotal, tax and total. It works. Then someone "improves" the prompt so it handles a new vendor's layout, and three weeks later finance notices that every euro invoice has been booked in dollars.

Nothing crashed. The output was still valid JSON, and nobody's unit tests touched the prompt, because prompts usually live outside the test suite. That's the gap this tutorial closes. We'll put a prompt under the same kind of regression testing as code, using Promptfoo's declarative test cases, and run it on every pull request with GitHub Actions. The build fails if quality drops.

By the end you'll have:

a promptfooconfig.yaml that compares two prompt versions side by side
20 synthetic invoices as test cases, including edge cases
layered assertions: valid JSON, a JSON schema, exact field checks, and a custom JavaScript check that the numbers actually add up
a GitHub Actions workflow that blocks merges when the pass rate falls below 95%
a setup that runs with no API key and no cost by default, so you can try the whole pipeline before you spend anything

Project layout

prompts/v1.txt              # the original, terse prompt
prompts/v2.txt              # the "improved" prompt under review
invoices/inv01.txt..inv20.txt
tests.yaml                  # one test case per invoice
schema.json                 # JSON schema for the extraction output
asserts/totals-add-up.js    # custom arithmetic check
providers/local-extractor.js# offline stand-in model for $0 CI
promptfooconfig.yaml
.github/workflows/prompt-eval.yml

Step 1: The prompts

Version 1 is what most teams write first:

Extract the invoice below as JSON with keys invoice_number, vendor, date (YYYY-MM-DD),
currency (ISO code), line_items [{description, quantity, unit_price}], subtotal, tax, total.
Return JSON only.

{{invoice}}

Version 2 adds the constraints we learned the hard way: no code fences, currency inferred from the symbol, plain numbers, and tolerance for a TOTAL label as well as Total Due.

Promptfoo treats {{invoice}} as a Nunjucks variable, and each test case fills it in. Listing both files under prompts: makes every test run against both versions, so the results grid becomes a before/after comparison.

Step 2: Test data you can share

Real invoices contain names, addresses and bank details, so they shouldn't go into a public repo or a CI log. The example repo uses 20 generated invoices with fake vendors and random line items. Tax is 7%, and the totals are computed, so they're guaranteed to be correct. Two invoices are in euros, and two are deliberate edge cases:

inv19 labels the total TOTAL instead of Total Due
inv20 starts with a chatty note before the invoice header

Edge cases are where prompt changes usually break, so give them their own tests rather than hoping the happy path covers them. As you find production failures, add a sanitized copy of each one to invoices/ and it becomes a permanent regression test.

Step 3: Assertions in layers

A single "looks right" check isn't enough. The suite stacks four assertions on each test, from cheap and general to strict and specific.

1. Valid JSON matching a schema. Promptfoo's is-json assertion takes an optional JSON schema:

- type: is-json
  value: file://schema.json

The schema requires every field, restricts currency to USD or EUR, requires a YYYY-MM-DD date, and needs at least one line item. That catches code fences, prose before the JSON, missing keys, and "$1,234.00" strings where numbers belong.

2. Exact field checks. For each invoice we know the right answer, so we assert it:

- type: javascript
  value: |
    const o = JSON.parse(output);
    return o.invoice_number === 'INV-1007' && o.currency === 'EUR' && Math.abs(o.total - 1043.29) < 0.01;

This is the assertion that would have caught the euro bug in the opening story. Compare floats with a tolerance, never ===.

3. Internal consistency. A model can copy the total correctly and still invent a line item. asserts/totals-add-up.js recomputes the math:

module.exports = (output) => {
  const o = JSON.parse(output);
  const sum = o.line_items.reduce((s, l) => s + l.quantity * l.unit_price, 0);
  const ok = Math.abs(sum - o.subtotal) < 0.01 && Math.abs(o.subtotal + o.tax - o.total) < 0.01;
  return { pass: ok, score: ok ? 1 : 0,
           reason: ok ? 'totals reconcile' : `sum ${sum.toFixed(2)} vs subtotal ${o.subtotal}` };
};

Returning {pass, score, reason} instead of a bare boolean means the Promptfoo viewer shows why a case failed, which saves a lot of time when you're debugging twenty red cells.

4. Rubric grading for fuzzy fields (optional). Some fields don't have one right string. "Acme Supplies" and "Acme Supplies Inc." are both acceptable vendor names. Promptfoo's llm-rubric assertion asks a grading model to judge:

- type: llm-rubric
  value: The vendor field names the same company as the "From:" line, ignoring suffixes like Inc or LLC.

llm-rubric needs a real grading model and an API key, so the example repo leaves it off by default. Turn it on once you add a provider key (see Step 6).

Step 4: A $0 provider for CI

Here's a practical problem: you want the pipeline in place on day one, but you may not want to put an API key in CI yet, or pay for evals on every documentation PR. Promptfoo supports custom providers, which are plain JavaScript classes with a callApi method, so the repo ships a deterministic stand-in:

class LocalExtractor {
  id() { return 'local-extractor'; }
  async callApi(prompt) {
    const t = prompt.slice(prompt.indexOf('INVOICE'));
    // ...regex extraction of number, vendor, date, line items, totals...
    return { output: JSON.stringify(out) };
  }
}
module.exports = LocalExtractor;

It's wired in with id: file://providers/local-extractor.js. To be clear, this doesn't measure model quality. It parses the invoice text out of the rendered prompt. What it does show is that your config, tests, schema, custom assertions and CI gate all work, for free, before any model is involved. It's also a useful baseline: if a real model scores below a regex parser on clean synthetic data, you'll want to know.

Running it locally:

npx promptfoo@0.125.0 eval -c promptfooconfig.yaml --no-cache
npx promptfoo@0.125.0 view

The output is a grid with one row per invoice and one column per prompt. With the local provider, all 40 cells (20 tests Ã— 2 prompts) pass.

Step 5: Gate pull requests in GitHub Actions

The workflow runs whenever a PR touches prompts, tests, invoices, assertions or providers:

on:
  pull_request:
    paths: ['prompts/**', 'promptfooconfig.yaml', 'tests.yaml', 'invoices/**', 'asserts/**', 'providers/**']
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 22.22 }
      - run: npx --yes promptfoo@0.125.0 eval -c promptfooconfig.yaml --no-cache -o results.json
        env: { PROMPTFOO_DISABLE_TELEMETRY: 1 }
      - name: Fail if pass rate < 95%
        run: |
          node -e "const r=require('./results.json').results.stats;
          const rate=r.successes/(r.successes+r.failures);
          console.log('pass rate',rate); if(rate<0.95) process.exit(1)"
      - uses: actions/upload-artifact@v4
        if: always()
        with: { name: promptfoo-results, path: results.json }

A few design choices are worth calling out:

Pin versions. Both the Promptfoo version and the Node version are pinned. Recent Promptfoo releases need Node 22.22 or newer, and an unpinned npx promptfoo will eventually change under you. Pin the model version too, once you add one.
Use a threshold, not perfection. LLM output varies. With a real model, a strict 100% gate turns into a flaky build that people learn to ignore. Start at 95% and tighten it as the suite settles. Keep temperature: 0 for extraction tasks.
Keep the artifact. Uploading results.json even on failure means reviewers can download the exact outputs that failed without re-running anything.
Turn off caching in CI (--no-cache), so a passing cached result can't hide a regression after a provider-side model update.

Step 6: Switching to a real model

When you're ready, uncomment a provider in promptfooconfig.yaml:

providers:
  - id: file://providers/local-extractor.js
    label: local-extractor
  - id: openai:gpt-4.1-mini
    config: { temperature: 0 }

Add the key as a repository secret (OPENAI_API_KEY) and pass it through the workflow's env. Keeping the local extractor alongside the real model gives you a stable baseline column in every run. Now llm-rubric can be turned on too.

On cost: 20 short invoices Ã— 2 prompts Ã— 1 model is 40 small requests per run, so with a mini-sized model a run costs well under a cent. If that adds up across many PRs, the paths: filter already skips runs for changes that can't affect prompt behavior.

What to do when a run fails

Open the uploaded results.json artifact, or run promptfoo view locally, and filter to failures.
Read the reason string from the custom assertion. "sum 812.40 vs subtotal 912.40" points straight at a dropped line item.
Decide whether the prompt regressed or the test is wrong. Fixed synthetic data makes that easy, because the expected values come from the generator and aren't hand-typed.
If a production invoice exposed a new failure mode, add a sanitized version as a new test before fixing the prompt. Watch it fail, then fix the prompt and watch it pass.

Lessons

Most of the value is in tests 2 and 3. Schema validation catches formatting problems, but the field-level and arithmetic checks catch the silent, expensive failures.
Edge cases earn their keep. In practice, the label-variant and noisy-preamble invoices are the ones that flip when prompts change.
Make it cheap, then make it real. A no-key provider gets the pipeline merged on day one. Adding the model is a three-line change.
Twenty tests is a starting point, not a target. Grow the suite from real failures rather than inventing hypothetical ones.

Clone the repo, run npx promptfoo@0.125.0 eval, then try breaking prompts/v2.txt, for example by deleting the currency instruction, and watch the gate catch it.

DEV Community: Zack Thompson