How we validate 12 agent config kits (and what validation can't do)

Our validator checks every kit before release: files, budgets,
frontmatter, sections, placeholders. Here's exactly what it checks,
why those checks exist, and — more interestingly — what it cannot
check, because pretending otherwise is how trust dies.

---

Agent config has a trust problem, and it's deserved. The market is
full of markdown with confident READMEs and no contract. "Validated
against real projects!" is the new "military-grade encryption." So
instead of adjectives, this post is the actual spec of our validation
harness — the same script that ships in the pack — including the
failure it caught, and the honest list of what it does not check.

The contract

A kit is a set of files with structure. Structure is checkable;
therefore the contract is structural. Every kit must pass all of:

1. Required files exist.
AGENTS.md, CLAUDE.md, copilot-instructions.md, and a
.cursor/rules/ directory with 3–5 .mdc files. Trivial? Every
missing-file bug in our own builds was "trivial" until it shipped.

2. The CLAUDE.md line budget.
Fewer than 300 lines, enforced, not suggested. The number is a design
decision (see the compliance-curve argument in our longer piece), but
the enforcement is the point: budgets without gates become
suggestions, and suggestions become 900-line files.

3. Cursor frontmatter parses, with a declared activation mode.
Each .mdc must open with parseable YAML frontmatter containing a
non-empty description, and either alwaysApply: true|false or a
globs entry. This check exists because of the nastiest silent
failure in the format: a malformed frontmatter block doesn't error —
Cursor treats the file as broken or ignores it, and a rule you
believe is protecting your test files simply never loads. The
validator makes "the rule never loads" a build failure.

4. The baseline has its sections.
AGENTS.md must contain: Project context, Commands, Code style,
Testing expectations, Commit conventions, Safety rules, Definition of
done. These aren't bureaucratic — they're the sections agents act on
most (the command table above all). A kit missing "Safety rules" is
missing the file's most important content, so it fails.

5. No placeholder text, anywhere.
TODO, TBD, FIXME, lorem ipsum, XXX — case-insensitive,
across every kit file. Config rot starts as a placeholder that
outlived its author.

6. No unrendered templates.
The kits are generated, so generation bugs are a risk class: the
validator scans for unrendered variables ({lint_cmd, and friends)
in output. This check exists because it caught exactly that bug —
a double-formatting trap left the literal text lint_cmd_placeholder
inside every kit's definition-of-done. The validator flagged it on
its first real run; the fix took one line; the check stays forever.

7. Skills exist and are non-trivial.
All eight workflow skills must be present and above a minimum size.
A stub skill file is a broken promise.

Output: human-readable by default, --json for CI. Exit code is the
release gate: non-zero, no release.

The proof it works: we break things on purpose

A validator that has never failed is a validator that has never been
tested. Before trusting ours, we negative-tested it — deliberately
corrupting a kit and watching:

• Appended TODO: fix this to a kit's AGENTS.md → failed, named the file, named the check.
• Restored the file → all green again.

That loop — break, catch, restore — is the entire basis on which we
ask you to trust the green checkmark. The check that caught the
unrendered-template bug was added the same way: a real bug, then a
check that makes it permanent history.

What validation cannot do

Now the honest part, because this is where most tool marketing lies by
implication.

It cannot check whether the advice is good. The harness knows
"Commands" exists; it cannot know your test command is right for your
repo. That's why kits mark the command table as the first thing to
edit — and why the validator refuses to pretend otherwise.

It cannot check semantics. Whether the baseline contradicts a
Cursor rule, whether your anti-pattern list matches your actual codebase
— these need human review or much cleverer tooling. We structure the
generation so contradictions are hard (tool files are projections of
one source), but hard is not impossible, and we won't claim the
validator catches what it doesn't.

It cannot check real-repo firing. We don't claim globs were
"tested against N production repos," because they weren't. The glob
syntax and mode declarations are validated; whether your repo's
layout matches the glob is checked by you, in your repo, in one
minute, using the harness we ship. Any product claiming automatic
firing-verification across arbitrary repos is claiming to have run
your build — ask them what that means, exactly.

Why structural validation is still the right contract

Because everything downstream depends on it. A rule that never loads
is zero percent as good as written, no matter how wise. A budget that
silently creeps degrades compliance in ways nobody attributes to the
cause. Structural checks are the difference between "config" and
"markdown with aspirations" — and unlike vibe-based claims, they're
runnable on your machine, against our kits, and against your own
edits, before you spend a cent.

That's the offer, verified: the validator ships in the pack, the
release output (12/12 kits, 8/8 skills, green) is in the changelog,
and the free MIT sample kit runs through the same harness. Trust the
check, not the adjective — and run it yourself.

AgentConfig Studio's twelve kits and the harness they ship with are on
AgentConfig Studio on Gumroad — all 12 kits.