Most CI pipelines gate what code does. Nothing gates what an agent is allowed to do, or whether a config change silently made it worse at picking the right tool for a task. We built that gate for sharekit and it's simpler than it sounds.
The problem
Agent config (skills, hooks, routing logic) changes constantly, and none of it is covered by a type checker or a unit test. A regex tweak in a routing hook can silently make the agent pick the wrong skill for a task, and the first sign of it is a user complaint weeks later.
The gate
evals/routing/README.md in sharekit runs 40 frozen routing-task files against a pinned OpenRouter model on every PR. Offline validation always runs (--validate-only, no key needed); the full regression check runs when OPENROUTER_API_KEY is configured, and fails CI on >5pp accuracy drop against a fingerprinted baseline.
Why 40 frozen tasks and not a live curriculum? Because the fixed-catalog approach is deliberate: reproducibility over recency. A frozen set means a regression is always a regression, not "the eval also changed."
The part that isn't visible from the gate
The gate's shape (small-n statistical gating, a fixed task catalog, a pinned baseline) isn't arbitrary. It's downstream of an actual embedding/RAG research program: bake-offs across e5/bge/jina embedding models with Matryoshka dimension-truncation trials, a distilled-reranker-vs-RRF ablation that measured RRF winning at the label scale this system runs at, and a deliberately reproduced circularity-trap failure (a ranker validating itself against its own outputs) characterized before guarding against it.
None of that shows up in the README. Full writeup: docs/ml-rigor.md in the repo, if you want the methodology instead of the summary.
Try it
npx ＠lucassantana/sharekit install <your-repo>
Gate fires on the next PR that touches routing config. Repo: https://github.com/LucasSantana-Dev/sharekit-profile
Top comments (1)
Gating behavior in CI is the right direction. If agents are part of the delivery system, their actions need tests too: tools used, files touched, approvals required, and whether they respected the boundaries under pressure.