DEV Community

MohammadReza Shabani
MohammadReza Shabani

Posted on

I built a safety net for prompt changes — PromptSeal, regression testing for LLM apps

  1. The problem: silent behavioral drift (story: one-word system prompt change broke refund tone)
  2. Why unit tests don't work for non-deterministic outputs
  3. The core loop: describe → seal → change → diff (GIF)
  4. 5-minute quickstart with the mock provider (no API key)
  5. Recording real traffic with the local proxy + PII redaction
  6. Model matrix: comparing gpt-4o vs llama3.1 on the same suite, cost included
  7. CI gate with the official GitHub Action (code sample + PR screenshot)
  8. What's next: pytest plugin, agent-trace assertions, suite registry
  9. Links: repo, action repo, tutorial (EN + فارسی)

Call to action: star the repo, try the 30-second quickstart, tell us which
assertion you'd add.

Top comments (0)