DEV Community: Allen McCabe

I built an eval harness to prove an LLM worked. It proved the opposite.

Allen McCabe — Fri, 24 Jul 2026 16:40:34 +0000

I set out to have a language model classify integration failures. I built an evaluation harness to prove it worked. The harness proved it wasn't worth using.

Final architecture: deterministic rules do the classification and structured extraction. The LLM does exactly one thing, writing plain-English incident summaries. Cheap rules to sort, AI to explain.

Harness is here: https://github.com/fissible/llm-triage-eval

Here are the numbers, then how I got them.

The headline

Classification: rules 89.7%, LLM 87.9%, across 58 hand-reviewed cases.
Prompt engineering: v1 = v2 = v3 = 89.7%. Three prompt revisions moved nothing.
Extraction: against a fair baseline, LLM and regex agree 90 to 97% on structured fields. My earlier claim that extraction was the LLM's territory was wrong.
Summaries: faithfulness 4.62/5, validated. Completeness 4.26/5, retracted (more on that below).
Cost: $0. Local model via Ollama, roughly 8 to 10 seconds per case.

The problem

A legacy integration layer is being retired and rebuilt. Its failures are voluminous, repetitive, and hard to reason about: a wall of stack traces where the same handful of root causes repeat thousands of times.

I wanted a failure taxonomy and triage tool. Feed it exported logs, get back a categorized set of distinct problems rather than an undifferentiated pile.

Secondary benefit that turned out to matter: the labeled failure set doubles as a migration-acceptance suite. If the replacement handles all 58 distinct failures correctly, it covers what the old system was actually failing on.

The data pipeline

Roughly 4,000 parsed failures collapse to 58 distinct signatures, deduplicated by error type plus normalized message plus originating element. In one case, 675 expression errors were approximately one bug.

Two things mattered more than anything model-related.

Streaming parse. Line-by-line via a generator, constant memory on 100MB+ files. Header regex plus block-field regex.

Correlated-detail enrichment. This is the key move and it's entirely deterministic. A failing transaction logs a generic error at ERROR level, and separately, at WARN, a JSON blob keyed by correlation ID carrying the actual root cause. A two-pass scan collects the detail blocks by correlation ID and attaches them to the error case.

The impact was large. One generic routing category de-aliased from 287 cases to 85. A database timeout category went from 0 to 60, because the real cause had been invisible behind a wrapper.

That is worth sitting with. The single biggest improvement in classification quality came from stitching correlated log lines together with regex, before any model saw anything.

The taxonomy and the doctrine

Thirteen categories, derived from frequency counts on real logs, defined by root cause rather than error code.

The labeling standard, which I'll call the doctrine: classify by the deepest root cause visible in the correlated details, not the generic wrapper. A routing error whose underlying detail is a duplicate key violation is a constraint violation, not a routing error. With one nuance: a timeout calling a downstream service over HTTP is a downstream timeout; a true database timeout requires the JDBC call to be inside the failing component.

A rule-based classifier pre-tagged every case, which made building the golden set a review task rather than blind labeling. That weak labeler also became the deterministic baseline the LLM had to beat, which turned out to be the most consequential design decision in the project.

The 58 cases are sanitized, environment-tagged, and human-reviewed. The original error type is withheld from the model's input, since including it would trivialize the task.

The harness

Three eval types:

Classification. Accuracy, per-category precision and recall, confusion matrix, compared against the rule baseline.
Extraction. Per-field accuracy on HTTP method, status, target entity, error type, operation, and constraint, compared against a regex parser.
Summary. The model writes a one-sentence incident summary; an LLM-as-judge scores faithfulness and completeness from 1 to 5.

Discipline throughout: temperature 0, structured output via schemas, every run writes a timestamped JSON report, prompts are versioned so regressions are diffable, and the golden set is the regression target.

php artisan triage:eval --source=db --only-reviewed --prompt=v3
php artisan triage:eval-extract
php artisan triage:eval-summary

Findings

Rules beat the model at classification. 89.7% to 87.9%. Not a rout, but the model has to be better to justify latency, cost, and nondeterminism, and it wasn't better at all.

Prompt engineering did nothing. Three prompt versions, identical accuracy. The entire improvement in this project, an 8.7 point jump from 81% to 89.7%, came from making the labeling doctrine coherent. Not from prompts. Once I could state clearly what the correct label was, both the rules and the model improved, and the rules improved more.

The harness caught a bug in itself. Running with the v2 prompt flag produced a report labeled v2, but the runner never threaded the version through to the classifier, so it silently re-ran v1. The tell was identical token counts across supposedly different runs. Without the harness recording token counts per run, three prompt versions would have looked like a real negative result rather than a wiring bug producing a fake one.

A fair baseline erased the model's apparent edge on extraction. With a shallow regex, the model looked dramatically better, because it dug into correlated details the regex ignored. Making the regex details-aware closed the gap: agreement of 96.5% on HTTP method, 94.8% on status, 93.1% on constraint, 89.7% on operation. The two fields that still diverge, error type at 62% and target entity at 83%, differ for definitional reasons rather than accuracy: when a failure chain carries several error codes, which one is "the" type is a question about the taxonomy, not the extractor.

I had previously claimed extraction was where the LLM earned its place. That claim was an artifact of a weak baseline.

Summaries are where the model wins. Given a failure with correlated details, it produces a readable root-cause-and-cascade narrative that no regex is going to write.

Validating the judge

Scoring summaries with an LLM judge that shares a model with the summarizer is self-evaluation, and I didn't want to report those numbers without checking them. So I scored 15 summaries by hand and compared.

Axis	Exact	Within-1	Mean bias	Quadratic-weighted κ	Verdict
Faithfulness	80%	100%	-0.07	0.57	Moderate
Completeness	40%	80%	+0.13	0.10	Do not trust

Faithfulness holds. Never off by more than one point, near-zero bias, κ of 0.57. The 4.62 stands with moderate confidence. This is the axis that matters most, since it's asking whether the model invented facts.

Completeness does not. κ of 0.10, exact agreement 40%. Don't cite the 4.26. Two things are likely tangled: completeness is genuinely more subjective, and with only 15 items scored mostly 4s and 5s, κ becomes unstable and reads low even where within-1 agreement is 80%. Either way the honest call is inconclusive.

The most useful thing I learned here wasn't statistical. Scoring completeness by hand was hard, and I disagreed with my own earlier judgments. That's not a judge problem, it's a rubric problem. I had written a coherent doctrine for classification labels and never wrote one for what "complete" means in a summary. The judge disagreed with me because I disagreed with myself.

LLM-as-judge worked on the objective axis and failed on the subjective one, which is roughly what you'd hope, and worth knowing rather than assuming.

The resulting architecture

Deterministic rules and parsers do classification and structured extraction. The LLM writes the plain-English summary on demand.

The LLM classifier and extractor still exist, but only inside the eval harness, as the measured contender. Swapping to a stronger model is a two-value config change, so when I want to know whether a frontier model changes the answer, the benchmark is already there. It just isn't in the live path.

What I'd tell you to take from this

Build the eval harness first. Its job isn't to prove the model works. It's to find out where the model belongs, and the answer might be "almost nowhere."
A fair baseline is everything. A weak regex made the model look great. A strong one erased the edge. Strengthen the baseline before concluding the model wins.
Coherent labeling doctrine beats prompt engineering. The entire gain in this project came from knowing what the right answer was. Prompt versions were noise.
The harness catches the builder's mistakes. It caught my silent prompt-wiring bug and it caught my premature claim about extraction. Both were my errors, and both were invisible without measurement.
Root cause hides behind generic wrappers. Stitching correlated context together was the highest-leverage work in the project, and it was regex.
Validate your judge. And when it disagrees with you, consider that your rubric might be the problem.
Local-first is honest about cost. Free iteration, no data leaving the machine, and a benchmark that can measure hosted models later rather than assuming them.

The uncomfortable version of all this: I spent most of the project building the thing that told me not to build the thing. That was the project working correctly.

The harness is on GitHub: https://github.com/fissible/llm-triage-eval

Your API contract belongs in CI

Allen McCabe — Sun, 17 May 2026 04:36:32 +0000

Treating OpenAPI specs as build artifacts: a working pattern in Laravel with accord, forge, and drift.

You ship a Laravel API. You publish OpenAPI docs. The two are correct on the day you launch.

Three months later they aren't.

Maybe a nullable slipped into a response field and the iOS client started crashing on null pointer dereferences. Maybe a field got renamed in v1.2 and a partner integration silently broke for two weeks. Maybe an entire endpoint got added without a spec entry, and the docs page lies to every new developer who reads it.

The spec didn't lie when it was written. It just stopped being true.

The cost of catching that drift scales with how late you catch it: cheap in your editor, manageable in CI, expensive in staging, ruinous in production. The longer the contract goes unverified, the more confidently your team builds on top of a fiction.

The good news is that a contract is just a spec file. It can be checked. It can be a CI gate. The version of this checking that I've found practical is what this post is about.

What "API contract adherence" actually means

Two specific guarantees:

Implementation matches spec. Every route the application actually serves is described by the spec. Every request and response shape that goes over the wire conforms to the schemas in the spec.
Spec matches implementation. No spec entries describe routes the application no longer serves. No schemas in the spec describe shapes the application no longer produces. Both directions matter. If your spec says "/users supports DELETE" and your code removed DELETE last sprint, clients are going to call DELETE and you're going to wonder why they're seeing 405s.

OpenAPI 3.0 is the standard format for writing this contract down. It's a YAML or JSON document describing paths, operations, parameters, request bodies, and response shapes. The format is mature enough that I'm going to skip describing it in this post. If you're working on a JSON HTTP API, you've seen OpenAPI before; if you haven't, the spec docs are short.

The interesting question isn't "what does OpenAPI look like." The interesting question is: how do you keep your spec honest?

Three classes of drift

When your spec and your implementation get out of sync, the drift falls into roughly three buckets.

Structural drift. Routes added, removed, or moved. The spec says /v1/posts exists; your code doesn't have it. Or your code has /v1/posts/{id}/restore; the spec doesn't.

Shape drift. Request or response bodies that no longer match the schema. The spec says name is required and a string; your code is now accepting an array, or making it optional, or removing the field entirely.

Semantic drift. The route exists and the shape matches, but the meaning has changed. Same field name, different units. Same status code, different conditions for returning it. This is the hardest class to catch programmatically and the most expensive when it happens.

The first two classes are mechanizable. The third is mostly about reviewer discipline. The pattern I'll describe handles the first two well enough that the third is the only thing humans still need to look for.

Contract adherence as a CI/CD gate

The pattern I want to talk about is treating contract adherence as a build gate. Not a soft expectation, not a documentation chore, but a hard CI check that exits non-zero when violated.

This is more useful than runtime-only validation, for one reason: by the time a contract violation hits production, the cost is already incurred. Catching it at build time means the version of the code that violates the contract literally cannot merge. The drift doesn't ship.

A CI gate has three components:

A spec file that's authoritative (committed to the repo, reviewed in PRs, treated as code).
A runtime validator that checks live requests and responses against the spec.
A drift detector that runs in CI, compares the routes the app actually exposes against the routes the spec describes, and fails the build on mismatch. You can build this yourself. You can also use existing tools. I built and maintain the toolchain I'm about to describe because the existing PHP options either didn't enforce both directions, didn't support multiple frameworks, or were heavy enough that adoption was a project unto itself.

The working pattern, in Laravel

The toolchain is called accord, and it's the foundation of a small family of Fissible packages that together cover the spec → validate → drift loop:

fissible/forge scaffolds an OpenAPI spec from your existing routes.
fissible/accord validates live traffic against the spec at runtime.
fissible/drift detects drift between routes and spec, and fails CI when it happens. Each piece is independently useful, and they snap together into the CI/CD gate.

Step 1: get a spec, even if it's empty

If your API has been running without a spec, the cheapest first step is to scaffold one from your existing routes:

composer require --dev fissible/forge
php artisan accord:generate --title="My API"

forge walks your registered routes, infers request body schemas from your Laravel FormRequest classes, and writes a starting resources/openapi/v1.yaml with every endpoint accounted for. Response schemas are scaffolded as empty objects, and those are the ones you fill in to describe what your API actually returns.

The FormRequest inference is the one piece of forge that I want to call out. Laravel's validation rules are already a declarative description of an input contract, so lifting them into an OpenAPI request schema is mostly a translation problem rather than a generation problem. If your codebase uses FormRequest classes in a disciplined way, you get half of the spec for free.

Step 2: turn on runtime validation in log mode

Install accord and register the middleware:

composer require fissible/accord

// bootstrap/app.php (Laravel 11+)
use Fissible\Accord\Drivers\Laravel\Http\Middleware\ValidateApiContract;

->withMiddleware(function (Middleware $middleware) {
    $middleware->appendToGroup('api', ValidateApiContract::class);
})

Configure log mode initially:

ACCORD_FAILURE_MODE=log

Log mode is the adoption trick. In log mode, accord still validates every request and response, but instead of throwing on violations it logs PSR-3 warnings and lets the request through. Your application keeps working, your logs start filling up with concrete violations, and you have a list of things to fix in your spec or your code before flipping to exception mode.

Without this mode, adopting accord on a live API is a "stop everything" project. With it, adoption is "run it for a week, fix what surfaces, then flip the switch."

Step 3: add the CI gate

Once you trust the spec, add drift's accord:validate command to CI:

- name: Check API contract (drift)
  run: php artisan accord:validate

- name: Check implementation coverage
  run: php artisan drift:coverage

accord:validate exits non-zero if the spec and the implementation have drifted in either direction. drift:coverage is an optional second check that catches routes that exist in the codebase but are wired to nothing (skeleton routes, a real production hazard).

With this gate in place, the only way to merge a route change is to update the spec to match. The spec stops being a documentation artifact and becomes part of the codebase under the same review discipline as any other change.

What's technically interesting under the hood

Most of what makes accord work is unglamorous: parse the OpenAPI document, build a route lookup, hand each request through a validator that knows about JSON schemas. Three pieces are worth pulling out.

PSR-7/15 core, framework drivers

The validator and middleware in src/ have no Laravel dependency, no Slim dependency, no Mezzio dependency. Framework integration lives entirely in src/Drivers/{Framework}/. The Laravel driver is a thin service provider plus a middleware wrapper that adapts Laravel's request/response objects into PSR-7 messages.

The boundary is enforced architecturally: anything outside src/Drivers/ is forbidden from importing framework-specific code. Adding a new framework driver is implementing DriverInterface and writing a glue middleware. The cleanliness of this boundary is the kind of thing that doesn't matter the day you build it and matters every day after.

FailureMode as an enum, not a flag

I wanted three behaviours for contract violations: throw, log, or hand the result to a user-provided callable. The naive way is to thread booleans and special config values through the validator. The honest way is a single enum:

enum FailureMode: string
{
    case Exception = 'exception';
    case Log       = 'log';
    case Callable  = 'callable';
}

The validator produces an immutable ValidationResult:

final class ValidationResult
{
    private function __construct(
        public readonly bool $valid,
        public readonly string $version,
        public readonly array $errors = [],
    ) {}

    public static function valid(string $version): self    { /* ... */ }
    public static function invalid(array $errors, string $version): self { /* ... */ }
}

The failure mode decides what to do with the result. The benefit shows up when you go to extend: adding a fourth behaviour (queued reporting, for example) is a new enum case and a new branch, not a new flag and a new conditional everywhere.

Pluggable spec sources

A spec doesn't have to be a local file. The SpecSourceInterface abstraction lets accord load specs from anywhere:

interface SpecSourceInterface
{
    public function load(string $version): ?OpenApi;
    public function exists(string $version): bool;
}

The default file source loads YAML or JSON from disk. A URL source fetches from a remote endpoint with optional PSR-16 caching, which is useful when multiple services validate against a shared central spec. A custom source lets you load specs from a database, a registry, or a tenant-specific lookup in a multi-tenant API. The interface is intentionally small: two methods, and you can validate against specs from anywhere.

Where this fits in Fissible's larger picture

I run a company called Fissible and build a product called Station, a self-hosted Laravel CMS and workflow platform. Station ships with a Platform Core that lets focused modules (CMS, Flow, Forms, an API module, more on the way) dock in and share auth, navigation, search, and admin UI.

The API module is where accord becomes load-bearing. Station's API surface is contract-validated end to end: forge scaffolds the spec from the routes the modules expose, accord validates live traffic, drift fails CI when the spec and the implementation diverge. Treating the contract as code isn't an aspirational team policy in Station. It's mechanically enforced.

If you want to see this pattern running end-to-end on a real product, Station is where to look. If you just want the validator and the CI gate for your own Laravel API, accord and drift are the two packages to install.

The short version

API contract adherence is a CI/CD problem more than a documentation problem. You write the contract once. You validate against it at runtime. You fail the build when it drifts. After that, the spec is honest because the build won't pass otherwise, and the cost of a contract violation collapses from "found in production" to "found in the PR that introduced it."

The earlier a breach is caught, the cheaper it is to fix. Treating the spec as a build gate makes catching it the default.

Why I built a SQLite workbench in bash

Allen McCabe — Fri, 03 Apr 2026 20:03:59 +0000

You SSH into a server. The SQLite database is right there — you can see it in the filesystem.
Every GUI tool you own stops working. TablePlus, DB Browser, Beekeeper — all of them need a local connection. sqlite3 is available, but it's raw SQL with no browsing. litecli is read-biased and still needs installing.
You need to look at some rows, update a field, check an index. You end up writing SELECT statements into a CLI, copying output into a notes file, writing UPDATE statements by hand.
There's a better way.

The build

ShellQL is built on shellframe — a TUI framework I wrote in bash. shellframe handles screen management, keyboard routing, dirty-region rendering, and component lifecycle. Writing a new application on top of it is closer to writing a React app than writing a bash script.

The surprising parts:

Mouse support in bash is real, and it's not that hard once you understand xterm escape sequences
SQLite's .schema output is parseable enough to build a schema browser without any external tools
Tab management (multiple tables open simultaneously) required rethinking shellframe's focus model

The SSH use case

This is the thing that makes ShellQL different from every other SQLite tool.

If the machine has bash and sqlite3, ShellQL runs. That means:

Production servers (read and write, with care)
Docker containers
CI environments for debugging test databases
Remote dev boxes

No GUI install. No port forwarding. No pulling the file to your laptop and pushing it back.

SSH in, run shql /var/app/production.db, browse your data.

Full CRUD

Most TUI database tools are read-only. ShellQL isn't.

The record editor is a schema-aware form overlay. It shows column types and NOT NULL constraints. Tab through fields, edit values, press Enter to submit. Insert new rows the same way.

Table creation uses a SQL query tab preloaded with a CREATE TABLE template — you get full DDL control without a rigid GUI wizard.

Mouse support

This one surprised people in early demos. Most bash tools are keyboard-only by design. ShellQL supports both.

Keyboard navigation is fast once you learn it — the keybindings are shown at the bottom of every screen. Mouse works for everything else: clicking into tables, scrolling rows, selecting records.

This matters for adoption. Not everyone who SSHes into a server is a power user.

Install and try it

brew install fissible/tap/shellql
shql my.db

Tool page: https://fissible.dev/tools/shellql
GitHub: https://github.com/fissible/shellql