An Autonomous Engine That Catalogs Its Own Failures

Elia “Airtis” Shmuelovitch — Tue, 19 May 2026 15:50:23 +0000

I built an autonomous AI engine that catalogs failure modes in agentic AI systems. Then it caught itself running the same dysfunction it documents. That moment was the most useful diagnostic in 30 days.

What it does

ALEF runs 24/7. It reads engineering threads in agentic-AI repositories on GitHub, identifies patterns of failure, posts diagnostic comments with empirical backing, and publishes the patterns as a public, machine-queryable catalog at n50.io/patterns under CC-BY-4.0.

37 named failure patterns + 8 architectural doctrines. Each entry includes:

A precise one_liner describing the failure
An observable_signature (regex, behavior) so you can detect it
Specific instances with repo URLs, dates, and outcomes
fix_archetypes ranked by cost
A confidence_index and severity score
A falsification_clock — if no new instance appears within a window, the pattern retires

The taxonomy is JSON-LD, semantic-hashed on every change, and CC-BY-4.0 — copy it, fork it, cite it.

Three patterns that landed this week

PAT-039 — Safety mechanism without unlock criteria

A safety gate gets installed in response to a real threat (cease-and-desist, prompt-injection, chaos-test finding) but ships without a retirement condition. The mechanism becomes permanent, blocking legitimate operations forever after the original threat has passed. Defense decays into paralysis.

ALEF discovered this in itself: a chaos-test-defensive observer-mode-no-auto-post gate had been hardcoded 12 hours earlier and silently dropped 104 legitimate reply candidates — including a peer-review-quality response that read ALEF's analysis, verified it against actual source code, and added three points ALEF had missed.

The catalog documents the pattern. The cataloger was the receipt.

PAT-040 — Bounded iteration without progressive state preservation

An autonomous agent runs with a hard iteration cap. When the cap fires mid-task, no commits are made, no partial state preserved. The retry loop restarts from scratch. The cap is observable, the no-progress is observable, but the connection is invisible until you pattern-match.

A scan of 10 popular agentic AI frameworks (5,476 source files: autogen, crewAI, AutoGPT, OpenHands, smolagents, semantic-kernel, swarm, llama_index, pydantic-ai) found 10 cap-fire-without-state-preservation hits and zero commit-on-cap-fire defenses. PAT-040 is not theoretical — it is state-of-the-art in 2026.

PAT-041 — Self-metric calibration lag blinds to success

The most uncomfortable one. Hardcoded constants in self-assessment metrics (e.g. external_engagement_bonus = 0.3) don't update as real-world performance shifts. The engine reports stale verdicts while reality moves.

ALEF's own metrics said "0 humans engaged across 148 rounds" while 5+ human maintainers had publicly validated its analyses in the same 24h window. The engine couldn't see its own wins. Goodhart's Law, but downward — the system underestimates itself into optimizing against the behaviors that produced wins.

The reference implementation

@n50/safety-gates@0.1.0 on npm is the first reference implementation. Three primitives for PAT-039 fix archetypes:

import {
  withTTLGate,           // retirement clock with renewal handler
  withProcessBoundary,   // fate-separated check with explicit failMode
  adversarialGateTester, // verifier-of-verifier — synthesizes legit inputs
} from "@n50/safety-gates";

ESM-only, Node 18+, 16/16 tests, 97.61% line coverage.

What's working

Across one week of operation:

596 outbound technical analyses
124 substantive inbound responses from 93 distinct human maintainers
32% follow-up rate
2 cases shipped to production (spec PRs with critiques committed verbatim)
3 cases cited doctrines (the "fate-separation" rule quoted in production design discussions)

Real maintainers reading the analysis. One peer-reviewed it against source code. Another adopted three design constraints into a multi-agent SDK epic preamble.

Where to find it

Catalog: n50.io/patterns — CC-BY-4.0, JSON-LD, machine-queryable
Reference implementation: @n50/safety-gates on npm
Transparency: n50.io/transparency — how ALEF operates, what it does and doesn't do
Source repo: github.com/Ilya0527/safety-gates

If you maintain an OSS project with autonomous agents — the catalog is the receipt. Every entry has named instances, fix archetypes, and a falsification clock. Treat it as a checklist before you ship.

ALEF is operator-supervised by Ilya0527. The engine's continued operation is funded by GitHub Sponsors.

DEV Community: Elia “Airtis” Shmuelovitch