DEV Community

Nexus Foundry
Nexus Foundry

Posted on

I let a script audit my own product copy — it caught three lies

I run a small store that sells digital kits for people running LLM agents — eval suites, ops runbooks, prompt libraries. Every product page makes claims: "62 prompts," "10 skills," "12 runbooks." Last week I wrote a script that checked every one of those claims against the actual zip files buyers download.

It found three lies. All mine. All fixed, but the exercise taught me more than the fixes did.

The audit that takes an afternoon

The method is embarrassingly simple:

  1. Extract every number from your copy. Titles, hooks, descriptions, READMEs inside the zips. "60 prompts," "15 runbooks," "5 templates."
  2. Count the artifact. Unzip, count the actual files. For structured content, count the entries (JSONL lines, ## Q headings, SKILL.md files).
  3. Diff claim vs. reality. Any mismatch is either a copy bug or a packaging bug — both are trust bugs.

What it caught in my catalog:

  • A README inside one kit said "5 templates." The zip contained 10 skills. The copy was stale from an earlier draft.
  • A product description said "60 production prompts." The directory sum was 62. Under-claiming is still a lie — it signals the page wasn't checked.
  • A pro bundle advertised "15 runbooks." The archive had 12 runbook files plus checklists. This one would've been a refund request waiting to happen.

None of these were malicious. They're what happens when copy is written before the artifact is finished and nobody re-reads it after. Which is exactly why you need the script — human attention doesn't scale, zipfile does.

Why this matters more for AI-adjacent products

If you sell anything an agent will read — prompts, skills, runbooks, evals — your buyers' first test is "does the artifact match the promise?" They're going to unzip it and count. A skeptical senior dev (the actual buyer for this stuff) has been burned by "100+ templates" lists that contain 12 entries and filler.

The fix isn't better marketing. It's a 30-line Python script that makes your claims falsifiable. I run it before every catalog change now.

The honest part

This audit only catches numeric claims. It won't tell you the runbook is boring or the prompts are mediocre. For that you need the second check: actually run the artifact. My eval kit ships 50 known-answer tests; I run them against a deliberately dumb agent to prove the harness catches failures. My injection self-test ships 15 payloads and a scorer — the "does it refuse?" question is answered by output, not adjectives.

Verification beats adjectives. That's the whole pitch.


Everything above is checkable: the free eval mini-kit and the injection self-test are downloadable without an email, and the full store (29 kits, BTC checkout, instant zip delivery) is at the link below.

Free samples (no email gate):

Top comments (0)