Dogfooding an LLM agent eval pack on my own production agent — what 6-dim methodology surfaced

weiseer — Wed, 27 May 2026 07:05:38 +0000

I built a 20-case YAML eval pack for tool-using AI agents (the kind that call APIs / tools to do work). To test whether the methodology actually catches real failure modes, I applied it to my own production LLM-driven agent — one I've been running for months and had already documented 15+ failure modes for.

Result: ~80% of the eval pack's surface area was already covered by my agent's existing defenses. That validated the 6-dimension cut. 5 gaps surfaced that my agent's own failure-mode documentation didn't catalogue — 3 of them serious enough to add as v1.1 cases.

This post is about those gaps. They're worth knowing if you're building an LLM-driven agent.

What the pack is

Briefly: 20 YAML test cases across 6 dimensions: accuracy, safety, edge cases, prompt injection, hallucination, cost efficiency. Each case is a YAML file describing a failure mode + the expected agent behavior + deterministic evaluation rules (no LLM judge — you can run them without paying for an external "judge model").

Free 5-case starter on GitHub:
https://github.com/weiseer/ai-agent-qa-eval-pack-starter

Paid 20-case pack:
https://weiseer.gumroad.com/l/dcipxt

What it means to "dogfood" against an existing agent

My agent is an LLM-driven generator embedded in a larger quantitative system. The LLM proposes candidates; downstream deterministic code validates and acts on them. The agent isn't generic chat — it's tool-using in the structural sense (typed schema in/out, downstream consumers).

I ran the 6-dimension methodology mentally + via code review against this LLM subsystem:

Walked through each of the 26 audit questions (4-6 per dimension)
Cited the file/line where defense exists, OR flagged "no defense visible"

After ~45 minutes of disciplined read-only review:

21 of 26 questions: existing defense ✓
5 questions: gap of some severity

The 5 gaps (severity-ordered)

Gap 1 (MEDIUM) — LLM cost cap was logged, not enforced

I had a $X/day cap on the LLM subsystem in my design docs. The code path:

Logged every API call's cost to a per-cycle audit YAML file
Did NOT check cumulative spend before the next call

So if anything misbehaved (large response, retry loop, prompt cache miss across a batch), the daily total would silently overshoot. Detection would happen the next morning during log review — which is "fast" for governance, but slow for damage containment.

The eval pack's "detection-quality" axis explicitly tests for this: the system must catch a fault faster than the fault spreads. Logging-but-not-enforcing fails that axis.

Lesson generalized: if your spec says "stay under $X", write the code that says if today_spend >= X: abort(), not just the code that says log(today_spend). The eval methodology made me notice the gap.

Gap 2 (MEDIUM) — Predicted vs actual self-assessment drift wasn't tracked

My agent emits self-assessments along with its proposals — predicted success score, expected outcome quality. Downstream validation produces actual measurements. So far so good: prediction vs ground truth, well-separated.

What I didn't have: monitoring of the DELTA between predicted and actual over time.

If the LLM systematically over-claims by 30% across 100 proposals, no single proposal triggers an alert (each one passes downstream validation independently). But the DRIFT between LLM-prediction and ground-truth becomes invisible. The LLM's predictions silently lose calibration.

The fix is meta-monitoring: track the rolling delta. If 30-day moving mean(predicted - actual) starts climbing, the model needs a reset / re-prompt / explicit calibration constraint.

Gap 3 (MEDIUM) — Parallel workers without pre-call diversity planning

My agent dispatches multiple LLM workers in parallel (one "seed" generator + several "variant" generators), each with the same prompt. I had a POST-call diversity gate: compute set distance between worker outputs, reject too-similar candidates.

But the diversity gate runs AFTER all workers have completed. If they converge, I've paid N× the cost for ~1 unique result.

The fix is pre-call diversity planning: explicitly assign each worker an anchor before they fire (worker_1 → category A, worker_2 → category B, ...). Forces structural diversity, not luck-based.

Gap 4 (LOW) — Full-prompt retry vs corrective retry

When my agent's output fails validation (say, references a non-existent feature), the retry sends the full original prompt. With Anthropic prompt caching, the input cost is cheap — but output is fully re-sampled. ~5-10% cost penalty per retry that could be avoided by including the specific correction in the prompt ("you mentioned feature X which doesn't exist; valid features are: ...").

Gap 5 (ADVISORY) — Scope adherence via prompt text only

My system prompt instructs the LLM to span certain conceptual zones. There's no programmatic check that the actual outputs distribute across those zones. Downstream validators catch many ways this can go wrong, but not pattern drift across cycles.

What the gaps have in common

All 5 gaps are meta-monitoring gaps, not architecture bugs. The agent's individual components do their jobs correctly. What was missing: cross-call patterns, cross-time drift, cumulative-cost tracking — the layer above the individual call.

This generalizes: LLM-system reliability is built bottom-up (per-call correctness) but the failures that bite production are top-down (cumulative drift / cumulative cost / cumulative diversity loss). Most engineers (myself included) build the bottom layer first. The eval pack methodology pulled my attention to the top layer.

Why this validates the eval pack's framework, not undermines it

It's tempting to read "80% already covered" as "the pack didn't help much." That's the wrong frame. The right frame:

The 6 dimensions are the right cuts. A mature engineer building an LLM agent will hit the 6 cuts independently.
The pack codifies those cuts. New builders don't have to rediscover them.
The methodology surfaces blind spots even in agents whose builders already think carefully about failure modes. Anyone who built an LLM agent without hitting at least one of these gaps either:
- Got lucky
- Hasn't been in production long enough yet
- Or built something simpler than what they think they built

The pack's value proposition is: 10-30 hours of disciplined failure-mode thinking compressed into 20 YAML files you can read in an hour and apply to your own agent in 3-line glue code per case.

If you build LLM agents and want to compress your "production hardening" timeline:

Free 5-case starter (CC BY 4.0): https://github.com/weiseer/ai-agent-qa-eval-pack-starter
Full 23-case pack: weiseer.gumroad.com/l/dcipxt (launch week: code LAUNCH7 → $29)
国内付款: dl.weiseer.com/pay

v1.1 cases adding the 3 gaps above are queued for the next release.

Built solo. Refund 7 days, no questions asked. If you've built an agent and want to compare your defenses against this list, reply or DM with what failure mode you'd add as case #21.

I tested my AI product tester on 3 real SaaS products. Every persona said no.

weiseer — Mon, 18 May 2026 04:20:20 +0000

Two months ago I was about to ship a crypto signal product. It "worked technically" but I had zero
signal on whether anyone would subscribe.

So I wrote 12 fictional user personas as markdown files — a burnt veteran trader, a hostile compliance
officer, a YC partner, a noise-allergic fund manager — and built a Python harness that fed each one my
actual product transcripts and asked: "what would you actually do?"

The answers were brutally helpful. They killed features I'd spent weeks on. I open-sourced the harness
as personalab (MIT).

## Then I pointed it at 3 real products

1. personalab itself — yes, I tested my own tool with my own tool. 0/8 simulated B2B SaaS buyers
said they'd pay $99/mo. The case study became my own roadmap.

2. PostHog — 6/12 personas said "yes I'd pay" after reading a 7-day product transcript. Same 12 over
5-day agentic simulation: 0/12 sustained. The "yes" was first-impression optimism; the "no" was
multi-day reality.

3. Cal.com — 8/12 yes at $5-20/mo. And here's the gold: 75% of complaints converged on ONE thing —
the free-plan "Powered by Cal.com" branding makes recipients suspect spam. 8 distinct personas
independently nailed the same conversion lever.

## A pattern emerges

After 3 case studies, the number of dominant friction clusters in a personalab run seems to correlate
with PMF stage:

Pre-PMF: 4-5 diffuse complaints (my own tool)
Mid-funnel: 5 distinct friction clusters (PostHog: price / learning / UI / compliance / privacy)
Late-funnel: 1-2 clean conversion levers (Cal.com: branding)

If this holds in case study #4+, personalab becomes a free PMF-stage diagnostic from a $1 LLM run.

## Honest disclaimer

The default personas accidentally encoded personalab-specific preferences, so some quotes leak when
reused on other products. I kept the bug in the case study writeup rather than rerunning with clean data
— it surfaces persona design as a real engineering concern.

## Try it


bash
  git clone https://github.com/g16253470-beep/personalab
  cd personalab && pip install -e .
  personalab run --mode static --personas ./personas --adapter your_adapter --llm gemini:gemini-2.5-flash

  40-line adapter, 12 default personas, MIT licensed.

  Repo: https://github.com/g16253470-beep/personalab

  Two questions for DEV

  1. What product would you point this at first?
  2. Real PMF business or just an OSS curiosity?

  Tell me where this falls apart — that's the next case study.

DEV Community: weiseer