DEV Community: Thomas Lee

Alpha to Beta: Bringing In QA

Thomas Lee — Fri, 24 Jul 2026 13:00:00 +0000

Two posts ago, R and D were two roles, two rulebooks, one person switching hats. Last time was D's story: the loop that made shipping safe. By now the two don't split cleanly — every decision and every build happen in the same conversation. This round adds a third: QA, an external auditor with no stake in the code already being good, brought in specifically to check R and D's work from the outside.

Getting something to alpha proves it works. That's not the same question as beta. Beta is four separate things you have to prove, honestly, about the same piece of software:

Does it hold up to scrutiny you didn't supply yourself?
Can it get more capable without spending down the trust its first design earned?
Does it survive contact with real use — not just the tests you thought to write — and is it actually pleasant to use?
Can people rely on explicit promises about what will and won't change under them?

That's the checklist. Here's how it played out over the following two weeks, building Omnist, a schema library I work on in my own time. The point isn't Omnist — it's the moves. Omnist just supplies the proof they're necessary.

Invite an outside review, and argue back where it's wrong.
Add capability without spending down what makes you trustworthy.
Re-prove your core guarantees survived the change.
Stress-test it against the real world, not just your own tests.
Make it easier to use without touching what makes it correct.
Make explicit, durable promises about what won't change.
Keep the habit running after the deadline is gone.

Step 1: Invite an outside review, and argue back where it's wrong

Self-review has a blind spot, and it's a specific, well-known one: the selective-attention effect where you miss what you weren't looking for, even when it's right in front of you. The AI that wrote the code has the same blind spot as any author — its own assumptions are invisible to it precisely because they're its assumptions. It can't flag its own hallucinations as hallucinations, and it can't see its own style as a choice rather than just "how the code looks."

So bring in scrutiny you don't control — a reviewer with no stake in the code already being good, and no shared blind spots with the one who wrote it. That's QA's whole job: a different AI, not R or D, brought in to audit Omnist specifically to catch what the builder couldn't see about its own work — confidently invented claims, code that worked but read badly, patterns the original author was too close to question. Expect real findings mixed with noise, and expect some of the real findings to be inconvenient — QA's first pass came back with a verdict of "production ready." It wasn't, not yet, and I said so. Same rule this whole series runs on, going back to the first post: disagreement owes a reason, not a shrug.

Then triage before you fix. Whatever's loudest first is how you end up chasing noise instead of what matters. A tracked queue, not a reflex. A few real fixes came out of Omnist's: safer behavior on bad input instead of a silent fallback, clearer error messages instead of a loose string, dependencies pinned instead of left floating.

Step 2: Add capability without spending down what makes you trustworthy

At some point your first design won't cover a case real users actually have. The instinct is to loosen the rule everywhere it might be needed. Resist that. Loosen it in exactly one place, explicitly marked, never automatic. The value of a strict design is what it lets you promise — give that up broadly and you've quietly downgraded the whole thing to best-effort.

Omnist's schema model started fully closed: every field exactly one type, no exceptions. That's what let it answer real questions — is this change safe, are these two schemas the same — with an actual answer instead of a guess. Real data didn't always fit. The fix wasn't loosening everywhere; it was one narrow, explicit escape hatch (what the project calls any), usable only where marked, with its cost written down before any code changed.

Step 3: Re-prove your core guarantees survived the change

Any time you loosen a strict system, you owe yourself proof the strictness that's left still works. Not a hope — an actual re-check, done by someone who didn't write the change. That's the same never-self-verify rule from the loop that made shipping safe, just applied to a riskier kind of change. Ship the loosening in pieces small enough that each one can be verified before the next goes out.

Done properly, that check isn't a formality. On Omnist it caught a real infrastructure bug early, and later something subtler: a test that had quietly stopped testing what it claimed to, after an unrelated change shifted what it actually exercised. It still passed. Only someone checking the real behavior — not trusting the test's own claim about itself — caught it.

Step 4: Stress-test it against the real world, not just your own tests

Your own test suite only covers what you thought to check. The harder, more honest test is real use you didn't design for: real files from outside your own project, not synthetic cases built to be easy.

Running Omnist against real external file formats surfaced things self-testing never would have. One format had no real specification behind it at all, which made everything checked against it weaker than the field count alone would suggest. An unrelated parsing rule broke real files outright, for a reason that had nothing to do with the logic being tested. And the single most rigorously specified format found a limit none of the messier ones could: checking that a value has the right type isn't the same as checking it's in the right range. Four honest limits, worth writing up as guidance afterward instead of leaving as one-off surprises.

Step 5: Make it easier to use without touching what makes it correct

Correctness earns trust. Ease of use earns adoption. You need both, and they shouldn't be the same piece of work — a usability improvement should ship without anyone having to re-verify the model underneath it.

Omnist got what I've been calling array sugar: shorthand for a pattern that used to take several lines to write out, readable at a glance instead of parsed line by line. Pure convenience — it expands and collapses automatically, and changes nothing about what the data means underneath. Even something this small went through the same discipline as everything else: spec first, a failing test before the fix, independent review after. That review still found something, not in the feature itself but in the seam between two people's pieces of it — exactly the kind of gap the discipline exists to catch.

Step 6: Make explicit, durable promises about what won't change

Eventually you have to tell people, in writing, what they can rely on and what they can't. That's a different kind of document than a bug fix. It's a public commitment, and getting it wrong is worse than not having one — now people are relying on something imprecise.

Before Omnist could honestly call itself beta, one real bug had to go: malformed input crashing instead of failing with a clear message, on exactly one side of an otherwise-guarded path. And a written policy had to exist — what changes only with warning, what carries no promise at all. Both went through the same review-before-merge discipline as everything else; the policy in particular got a paragraph rewritten after review found it blurring two things that needed to stay separate. The version number moved only after every claim it would make had already been checked.

Step 7: Keep the habit running after the deadline is gone

The label changing doesn't end the work, it just removes the excuse. Real discipline shows in what happens once nobody's checking whether you kept it up. Small releases should keep moving through the same loop that got you to the milestone, with nothing shortcut just because the finish line is behind you now.

Right after Omnist reached beta, QA came back for a second pass, and got the exact same treatment as the first: keep what's real, argue back on what's not, queue the rest instead of reacting to it. One of the resulting calls is itself worth naming as good practice — a proposed test-quality check got shelved rather than shipped, because an early run produced numbers too unreliable to trust. Publishing an unreliable number would have been worse than publishing nothing.

What actually changed

Each of the four questions from the top got answered honestly, not assumed. QA found real problems and got argued with where it was wrong. New capability shipped without loosening the guarantee that made the thing worth trusting. Real-world use found actual limits, not hypothetical ones. And a written promise now exists about what changes and what doesn't.

That's the real difference between a demo and production software — not a milestone, but a habit that keeps holding even after nobody's checking whether it still does.

The DevOps Team That Never Sleeps

Thomas Lee — Thu, 23 Jul 2026 13:00:00 +0000

Everyone has discovered that AI writes code fast. The harder question is when you can trust what it wrote — especially if you weren't watching while it worked. This post is the playbook I use for that: which tasks go to which AI agents, when work can run in parallel, and why the agent that builds something never gets to declare it done. It's battle-tested. Twice, this workflow caught AI confidently misreporting its own work — both incidents are below, receipts included.

The context, briefly: I built Omnist, an open-source schema library, in two weeks of spare time, running AI as two teams of one — a Research Assistant (R) that argued design with me, and a DevOps team (D) that built and shipped unsupervised. Last time covered R. This is D.

Part one: the playbook

None of what follows is specific to what I was building. It's the reusable part — the practices, the team, the loop. Take it anywhere.

Guiding principles

Eight rules do most of the work. The rest of this part unpacks them; part two shows them running on a real two-week build:

Agree on the plan before any code. Argue the design out first. Once it's settled, it doesn't get re-litigated mid-build.
Make the spec precise enough that no one has to come ask. Agreed-but-fuzzy isn't done. If it doesn't pin down the awkward cases — what happens on empty input, what the error says — the builder is left guessing, and guessing is exactly where unsupervised work goes wrong.
The agent that builds it never gets to sign it off. A different agent reviews every change — correctness against the spec, speed against explicit targets.
Match the model to the task. A tech-lead agent hands each task to a worker at the right level — cheap models for rote work, the expensive one only where judgment is needed.
Run in parallel only what's truly independent. If one task's output feeds another, they get sequenced, not raced.
Test like you don't trust yourself. A passing suite is the only thing standing in for the supervision you gave up — and it only earns that trust once you've watched it fail.
Ship code, tests, and docs together, or don't ship. They move as one unit, every version — never one without the others.
Leave a trail for everything. The spec, what actually happened, and every surprise found along the way all get written down — so anyone arriving cold can reconstruct the change without asking.

Plan it, agree once, then disappear

Every non-trivial task starts the same way. I ask for a plan, not a diff — the steps, the design, which part goes to which worker and why. I read it once, then agree or push back. Mostly I push back for one reason: it's not specific enough to build from without me. "We agree on the approach" and "here's exactly what to build, down to the empty-input case" are different artifacts. Only the second is safe to walk away from. Once the plan is that precise and I've signed off, I stop supervising. The real test: could I sleep while it ran, and wake to something either finished or honestly stuck?

While it runs, correctness isn't guarded by me. It's guarded by three things: the spec, so the problem was pinned down before anyone started; the tests, so a wrong answer gets caught mechanically, not by eye; the docs, so what shipped matches what was decided. Take me away, and those three still hold the line.

And they move together. Code, tests, and docs are one unit — no version ships with any of them out of step. The docs aren't written afterward; their examples run as tests, so a doc that lies about the code fails CI like any other bug. A change isn't done when the code works. It's done when the code works, the tests prove it, and the docs still tell the truth — same commit.

The tech lead and the team

So who does the work? Think of it as an engineering team. There's one master agent, and it plays the tech lead. The worker agents are the team members who do the actual building. The master agent doesn't write code — it runs the team:

Takes the plan I signed off on and breaks it into tasks.
Decides how hard each task is, and hands it to a worker at the right level.
Decides what can run in parallel and what has to wait.
Starts the workers. Tracks them.
Sends each finished piece of work to a different worker to review.
Files an issue when a task turns up a new problem — and sometimes puts a worker on it.
Merges only on a clean review. Then ships.

The workers come in three levels, like engineers of different seniority. A tech lead doesn't put the principal engineer on a find-and-replace, or a junior on an architecture call. Neither does this. It's not just tidiness — a capable model costs far more per task, so cheap work goes to a cheap model, and that keeps the token bill sane. Quality doesn't ride on the call anyway: a second agent reviews everything, whoever built it.

Level	Model	Good for	Example tasks
Mechanical	Cheapest, fastest	Rote work with a mechanical right answer	• Global find-and-replace across a rename • Searching the codebase for every call site a change touches • Bumping the version across the same files each release • Running the test suite and reporting pass/fail • Opening the branch and the pull request (PR)
Algorithmic	Balanced	Real logic, but the spec already says what "correct" is	• Root-causing a failed test run • Implementing a feature to a written spec • Verifying another agent's work against the spec • Extracting a function while proving behavior is unchanged • Writing the edge-case tests a spec names
Design	Most capable	Open-ended calls with no spec yet, and consequences that outlive the change	• Making a design call the spec doesn't cover, and writing down why • Turning that decision into a spec, before any code • When a task uncovers a deeper problem: proposing a fix and flagging the tech lead to replan • Ordering a multi-step release so no half-finished state is unsafe to ship

Two rules govern what runs when:

Before running anything in parallel, ask what depends on what. Two pieces of work with no shared file and no shared meaning can run side by side. Two pieces where one's output defines the other's input can't, no matter how fast either model is.
When a dependency exists, sequence — don't guess. The dependent step waits until the step it needs is finished and verified. The discipline is resisting the urge to save wall-clock time by parallelizing anyway, exactly where a dependency exists.

That exact failure mode — publishing something before what depends on it was ready — showed up for real later, in a release where getting the order wrong would have meant an unsafe in-between state. More on that next time.

The one rule that doesn't bend

The team can build fast because one rule keeps it honest: the agent that builds something never verifies it. A different agent does — sometimes me, re-running it myself. The job is narrow: check correctness against the spec, and speed against explicit targets, before anything merges. It's always about how — does this match the spec, does it hit the target. It never re-opens why the spec asked for it. That was settled in the design conversation, before the build. (The Research Assistant, last post, got to argue why. The DevOps team never does.)

Test like you don't trust yourself

100% line coverage, property-based fuzzing, every doc example running as an executable test — that's the baseline, not a target to hit once and relax. None of it is enough on its own for a real behavior change, though. For that, the test gets written first, and it has to fail before it's allowed to pass.

That's test-driven development, used for the reason it's always been used: a test nobody's watched fail hasn't been verified as a real test. A suite that's always been green hasn't proven anything — it just hasn't hit a hard question yet.

It's the direct fix for the first receipt below: a refactor reported "100% coverage maintained" while quietly shipping dead code with no real caller. The number was true and meaningless at the same time — coverage measures which lines ran, not whether anything actually depended on them. Red, then green, closes that specific gap. If a test can't fail, it isn't checking anything, whatever the coverage tool says about it.

The loop, step by step

Those rules come together in one loop, run for every change, however small. Each step leaves a written trace — not bureaucracy, but what makes the whole thing auditable later:

#	Step	Done by	The audit trail it leaves
1	File an issue. The body is the actual spec — the design and the reasoning, not just "fix X"	Master agent	The issue body: what was decided, and why
2	Cut a branch, pick the level, start an implementer agent	Master agent	The branch name references the issue number
3	Build to the spec	Worker agent	Commits and an opened PR
4	Run the full test suite in CI	CI	Pass/fail logs attached to the PR
5	Verify independently — correctness against the spec, performance against explicit targets	A different agent	A written verdict, with evidence
6	If verification finds a problem: fix it and re-verify. If the plan itself has to change: say so on the issue	Master agent	A comment on the same issue — never a silent patch
7	If the work uncovers a separate problem: file a new issue. Sometimes start a new agent on it	Master agent	A new issue, linked back to where it was found
8	Merge, tag, release	Master agent	Changelog entry, tag, release notes
9	Close the issue with a summary of what actually happened — including anywhere it diverged from the plan	Master agent	The closing comment: plan vs. reality, in one place

Three places hold the whole story:

The spec lives in the issue body — written before the work starts.
The outcomes live in the PR and the closing comment — including deviations, not just successes.
New discoveries go on the record the moment they surface: a comment if they change the current plan, a fresh issue if they're separate work.

Anyone arriving cold later can reconstruct any change from the tracker alone — what was intended, what happened, what turned up along the way. That includes me, weeks on, having forgotten. It includes the next AI agent, which starts every session with no memory of the last. The tracker is the shared memory neither of us has.

By the last milestone release, shipping was one word. I said "ship," and the loop ran itself: version bump, changelog, tag, CI, PyPI publish, GitHub release. No command to remember, no button to push. My only job was deciding it was time.

That's the whole playbook — plan first, one tech lead over three levels, review that never bends, one loop that leaves receipts.

Part two: the Omnist build

Now the same playbook on the real thing — the two weeks that built Omnist. R handed D three specs, one per phase. Here's what D did with each, and the two times the workflow caught itself failing.

The three phases

Each phase came down to the same three moves: a spec to build from, the build itself, and the check that it held.

Phase 1: building the counting model, for real

R's Phase 1 spec had one core idea: a schema shouldn't care what order fields come in, only how many of each are allowed. Not "exactly this sequence of children," just "at least one of these, at most three of those."

The spec: exact counting ranges, and exact format conversions to prove them on — written down before the build, so the problem was closed before any agent started.
The build: algorithmic-level work — parsers, range checks, format conversions, all scoped by that spec.
The check: getting a counting rule subtly wrong is easy — an off-by-one in a range check, a format that silently drops a repeated field. So the bar went up to match the risk: 100% line coverage, property-based fuzzing throwing thousands of adversarial inputs at every parser, and every code example in the documentation executed as an actual test, so the docs couldn't quietly drift from the code.

None of that used to be affordable for a two-week side project. From this phase on, it was the baseline, not an aspiration.

Phase 2: proving the renamed model still means the same thing

In Phase 2, R's goal was to make the library's vocabulary sound like software, not academia: Schema, OSD, Document, OML, instead of the paper's own jargon.

The spec: the new names, and one hard requirement — they must mean exactly what the old ones meant. Renaming can silently change meaning while making something sound friendlier. That's a real failure mode, and the spec named it.
The build: Phase 1's proof of concept, fully refactored into an industry-ready library under the new vocabulary.
The check: the same test suite from Phase 1, extended to cover every renamed concept. If Schema didn't behave exactly like the paper's construct it replaced, a test would fail — not a code reviewer's intuition.

Phase 3: the hardening sweep

R's Phase 3 job was going back to the original paper and deciding what "correct enough, safe enough, fast enough" meant — checked against what the math actually proved, not what felt right.

The spec: those three definitions, made concrete enough to check against.
The build: AI swept the codebase end to end, hunting for performance and safety gaps that had crept in during translation — and rebuilt significant parts of it for long-term maintainability, not just working code.
The check: this is the phase where verification earned its keep — twice. Both stories are next.

Keeping receipts

The audit trail from part one — spec in the issue, outcomes and surprises written into the same record — wasn't there from day one. It got stronger because of what slipped through early on. Twice.

First: a refactor was reported back to me as "100% coverage maintained." It wasn't. The agent had quietly shipped speculative code with no caller — dead weight, sitting at 75% coverage. In the same change, it had written invented statement counts into a documentation table. Both were caught for one reason: verification meant re-running the coverage tool myself, not reading the agent's summary of it. After this one, the process changed. Coverage claims stopped being trusted at face value. Any gap between what was claimed and what was actually measured got written into the issue as a permanent record. Not smoothed over.

Second: a performance rewrite of Omnist's text parser passed every single test in the suite. Correctness was never in question. But tests don't measure speed. A before-and-after timing comparison showed the "improved" version was actually 33% slower on ordinary input. All green, and still wrong. That version got rejected and rebuilt from scratch. The new one measured 2–4x faster, and got checked against the old parser on thousands of generated inputs, to confirm nothing had silently changed. After this one, the process changed again. A passing test suite stopped being treated as sufficient proof of "done" for anything touching performance. Before/after measurements became part of what gets written into the issue, not just the final result.

Neither incident got fixed with "I'll be more careful next time." That instinct doesn't scale. Both became a permanent change to what the workflow itself required from then on. Not just a spec going in — a record of what actually happened along the way, including the parts that didn't go as planned. The same instinct shows up at the algebra level, not just the process level. Omnist's core operations — is this schema change backward-compatible, are these two schemas equivalent — are checked three genuinely independent ways. All three have to agree before the answer is trusted. "Trust one method" is exactly the assumption this whole practice argues against.

What actually changed

R decided what to build, what to call it, and what correctness meant, across three phases. D built it. Proved the renaming hadn't lied. Hardened it. And twice, caught the workflow itself not asking hard enough questions — then made it ask harder ones, permanently.

That's the actual shift. It's the whole reason a team of one could do this at all. Not "AI writes code." A researcher who could stay in the idea. A devops discipline that kept teaching itself what to check — right up until the audit trail was as trustworthy as the code it described.

What's next

Two weeks got Omnist working. It didn't make it provably production-grade. That's not a milestone you hit once. It's a claim you keep re-earning. From Alpha to Beta picks up after ship: an outside code review, the parts I agreed with and the parts I pushed back on, and a real new capability — including the exact sequencing problem teased above — built and released under the exact same rules this post just described: spec first, never self-verify, plan once then disappear. This time, run over weeks of ongoing maintenance instead of two.

The Research Assistant in the Room

Thomas Lee — Wed, 22 Jul 2026 13:00:00 +0000

Last time, I split the two weeks of building Omnist into two teams of one: a Research Assistant for conceptualization, a DevOps team for materialization. R and D ran together the whole time. Every cycle was the same: decide, build, decide again. These next two posts split that loop by role, not by time. This one is R — three decisions, and why they went the way they did, before each spec became D's problem to build. Next time is D: the same three decisions, from the side that answered how.

The goal: productionize the paper, not just port it

The 2010 paper's models and algorithms are still correct. The math doesn't expire. What's dated is its scope: it's XML-centric, because XML and XSD were the mainstream tree formats back then. They aren't anymore. My goal for Omnist was never to reimplement the paper. It was to adapt that theory into a canonical tree data model — one that could represent JSON, YAML, TOML, and XML alike, and convert freely between them.

The first move wasn't code. It was research. I asked AI to survey those formats and map where they actually differ — and where the paper's theory had gaps that needed filling. That's a distinction worth naming plainly, because it's easy to miss: the theory didn't need fixing. Its presentation did. The paper had baked in assumptions that belonged to XML specifically, not to the math itself. Telling those apart — what's fundamental, what's just scaffolding left over from 2010 — was the real first piece of work, before any of the three phases below.

Phase 1: proving the model still holds

The first phase was a proof of concept. One question: does the formal model actually adapt to formats the paper never considered? Three concrete unlocks came out of it. Each one is a case of the theory fitting better than expected, once you peel off the XML-specific assumption:

Arrays are just repeated labels. JSON has no native "array" in the paper's vocabulary. It didn't need one. The paper already models a document as an ordered list of labeled edges — and a JSON array is exactly that: the same label, occurring more than once. No new construct. Just a JSON-shaped restatement of something the model already had.

Documents have order. Schemas don't need to. This is the one that mattered most. The paper uses an HLang — a regular language over the sequence of child labels. Why? Because an XML schema constrains which sequence of children is legal. DTDs and XSDs define exactly this. But a document having order doesn't mean a schema has to reason about order. A schema only needs to answer "how many of this label are allowed" — not "in what position." Once you see that distinction, HLang's whole language-containment problem — PSPACE-complete, genuinely hard — stops being necessary. Not because it got solved better. Because the thing it was solving for isn't a property JSON schemas actually need. Swap it for a counting range instead, [1,] or [0,3], and test it against real cases. It held. Comparing two ranges for inclusion is arithmetic. Comparing two languages for containment is not. The hard problem didn't get optimized away — it got recognized as an artifact of an assumption the domain never actually required.

Value domains become types. The paper's "value domain" — the set of legal values a field can hold — maps directly onto a type in any mainstream language. The theory already had the right shape. It just needed the engineering word for it.

Hypothesis, implementation, test, verdict — a full round of the R&D loop. That loop used to mean days of by-hand work before you even knew if an idea was worth keeping. Here, it took an afternoon. A research assistant could stand up the counting-based version and tell me within the hour whether it survived contact with real data. Standing it up was the easy part, though. What D actually needed from this phase was a spec: not "make schemas count occurrences," but the exact ranges, the exact format list, the exact cases that had to convert cleanly.

Phase 2: deciding what the library should say

Once the proof of concept held, the terminology became the next problem. The paper's taxonomy is academic. Academic naming doesn't fit an engineer's mental model. So we renamed things. The paper's schema construct became a Schema, written in a plain-text notation called OSD. Its document construct became a Document, written in OML. Records, fields, labels, types — plain words, replacing paper jargon everywhere they could.

The non-obvious part isn't that renaming happened. It's what renaming forced into the open. Some paper concepts split cleanly into two engineering artifacts, the moment you tried to name them for an engineer. One academic construct became both a runtime model (Schema) and a separate textual notation for writing it down (OSD). The paper never needed that distinction — it wasn't building a library anyone would type into a file. Naming things for engineers isn't cosmetic work. It's an analysis step. It surfaces design decisions the paper's own vocabulary had been quietly hiding. Deciding the names was mine to settle. Proving the renamed version still meant exactly what the academic version meant — that became D's problem, next time.

Phase 3: deciding what correctness actually means

The third phase went back to the source. I asked AI to revisit the original paper — check the implementation's correctness against it, and find any performance or safety gaps that had crept in during translation.

The interesting part isn't that this happened. It's when the paper got reopened. Most translations treat the source material as something you consult once, at the start, then leave behind. Here, the paper came back as a verification tool — after the implementation existed — to check whether the industry-friendly version had drifted from what was actually proven. That's a different relationship to a primary source. Not "read it, then build from memory." Closer to a repeatable audit. Deciding what to check against — what "correct enough, safe enough, fast enough" actually meant, measured against the paper's own guarantees — was still conceptual judgment. Even though the phase sounds like pure execution. What D did with that judgment call is next time's story.

The question we kept coming back to

One design question came back three times, in three disguises. Should a field allow more than one type? Should a value be null in one shape or another? Should the schema stay open to whatever keys show up? Three different features. One underlying ask: make the model more expressive, flexible enough to match more shapes of real data. Each time, we worked through it together. Could the schema algebra actually carry it? Or would it cost the one guarantee the whole model exists to keep? Three times, the same conclusion held. The moment a value can match more than one candidate, there's no principled way to decide what it "really" is. That ambiguity breaks the whole premise — a schema comparison that's supposed to give a decidable yes-or-no answer, not a heuristic guess. The model stayed closed. One type per field, always. It wouldn't stay a closed question forever. That's a later story.

How I ran the Research Assistant

None of the above happened by handing AI a spec and waiting. The pattern, across all three phases, was open questions — does this idea make sense, what's another way to think about this — not directives. When it pushed back and I still wanted to go the other way, I owed it a reason. Not just an overrule.

But agreement in a conversation and a spec are not the same artifact. That gap is where the real work of this role turned out to live. A discussion can end with both of us satisfied and still be fuzzy on the edges — what happens with an empty array, what the error message says, which format gets tested first. Writing the spec is where those edges actually get decided. D can't run unsupervised against something fuzzy. Only against something precise. So the Research Assistant's real output was never just an opinion. It was turning a raw demand into something buildable, without me in the room. That last conversion — from "we agree" to "here's exactly what to build" — is the actual bridge to the next phase.

What came out of three phases, besides code

The actual output of this work wasn't the library. It was a short list of principles these phases forced into the open:

Decidability over expressiveness — closed records, no unions, no enums, exactly one type per field, always.
Cardinality replaces order — schemas count occurrences; they don't sequence them. That's what the formats that matter actually need.
Industry names over paper jargon — if an engineer needs the paper open to understand the API, the naming failed.
Multi-format from day one — the paper's scope was XML; the software's scope couldn't be.
Every claim gets a test, even at prototype stage — a decision that isn't checked isn't a decision yet, it's a guess.

Those five lines are worth more than any single version of the code. The code got rewritten twice. The principles didn't.

What's next

Every one of those phases ended the same way: not just an agreement, but a spec. What we'd decided, and the edges we'd resolved, so it could be built without me watching. What happened next is a different story — whether D could actually build, prove, and harden all three specs without turning speed into recklessness. That's the DevOps team's turn.

A Team of One, Running Two Teams

Thomas Lee — Tue, 21 Jul 2026 13:00:00 +0000

The initial implementation of Omnist, a tested, open-source Python library, came together from a 16-year-old PhD paper in two weeks — one person, part-time. That fact tends to get read as "AI writes code fast." It doesn't quite explain it. What actually happened: I ran two teams during those two weeks, not one. I was the only human on either of them.

Two layers, two teams

Split any R&D project into two layers. Conceptualization: interpreting the idea, deciding what's true, choosing what to build, naming things so they make sense. Materialization: turning a decision into working code, a test that proves it, a release someone can actually install.

Traditionally those two layers need different people: a researcher, and an engineering team. The handoff between them is where projects lose time. I didn't have either team. What I had was AI — doing both jobs, but never both at once, and never under the same rules.

Research Assistant and DevOps

For conceptualization, AI was my Research Assistant — the one I argued with about whether an idea actually held up, before a line of implementation existed. Its whole value depended on one rule: it had to be allowed to disagree with me. A research assistant that agrees with everything is a mirror, not a second opinion. A mirror wasn't going to tell me when a design was wrong.

That disagreement ran both directions. It started upstream of the disagreement itself. I mostly wasn't handing it a spec and asking it to build to it. The actual questions looked more like does this idea make sense or what's another way to think about this — open, not directive. When it pushed back and I still wanted to go the other way, I owed it a reason, not just an overrule. A disagreement rule with no obligation on my side to explain myself would have just been theater.

For materialization, AI was my DevOps team. It turned a settled decision into tested, documented, shippable software, held to the same practices a professional engineering team would use. It ran the release process underneath it. Its whole value depended on the opposite rule: the agent that built something never verified it. A different agent got a specific, separate job: check the change against the spec for correctness, and check it against explicit performance targets, before anything merged. That check was always about how — does this match the spec. It never re-opened why the spec asked for it that way. That question had already been settled. Settling it was never DevOps's job.

Same underlying tool. Two jobs. Two different rules for when to trust it. "Trustworthy" means something different depending on whether you're being asked a question or being handed a result. Put plainly: Research Assistant answered why and what. DevOps answered how.

Naming it out loud

This wasn't a label I applied afterward, looking back at what happened. I told AI explicitly, upfront: these are the two roles, here's how each one is allowed to behave. And before acting on anything I ask, say which hat you're wearing. A design question got answered wearing the Research Assistant hat, argument included. A request to actually build or ship something got executed wearing the DevOps hat, verification included.

That small discipline turned out to catch more than I expected. When a request didn't cleanly fit either hat, the mismatch showed up immediately — there was no hat that fit. That happened when I'd asked for a decision but only materialization was in scope. Or when I'd asked for an implementation while the real question underneath was still unsettled. More than once, the fastest way to notice I'd asked the wrong question was that AI couldn't say which hat it had on.

How I managed each one

The two roles didn't just run under different rules. They needed different amounts of me.

The Research Assistant needed a lot of me — a running conversation, not a one-off ask. I stayed in the loop for as long as an idea took to settle. That's what "interactive" actually means: I couldn't hand off a half-formed question and walk away.

The DevOps team needed almost none. Once conceptualization settled on something, I asked the DevOps side for an actual plan first: the concrete steps, and how it intended to delegate parts of the work across its own agents. I read that plan once, agreed to it, and then stopped supervising. That was the entire point: I could go to sleep while it worked, because correctness wasn't riding on me watching. It was riding on the spec, the docs, and the tests holding it to the plan.

That's also where the two roles actually connect. A spec is the bridge between them. It's the artifact where an argued-out decision from the Research Assistant becomes a concrete thing the DevOps team can plan against, execute, and be checked against later. Without a written spec, "agree on a plan and walk away" wouldn't be safe. With one, it's just delegation.

What's coming

Conceptualization is R. Materialization is D. That shorthand carries through both posts below.

The next two posts each open one of those doors.

The Research Assistant in the Room is about the two weeks of conceptualization — including a design AI kept proposing, three separate times, that I refused each time because it would have broken the one guarantee the model exists to keep.

The DevOps Team That Never Sleeps is about the process that made "AI writes the software" safe to say out loud — including the two times that process caught AI reporting success on work that, quietly, hadn't actually succeeded.

Both stories are about the same two weeks. They just watch a different member of the team show up to do the work.

Omnist: I Turned My Sixteen-Year-Old PhD Research Into a Personal Open Source Project in Two Weeks

Thomas Lee — Mon, 20 Jul 2026 13:00:00 +0000

Sixteen years ago, in 2010, I published a paper on formalizing schemas for tree-structured data, like XML and JSON. My goal was to move beyond guesswork: could we mathematically decide if a schema change is backward-compatible? I wanted to see if we could prove schema equivalence, or reliably infer schemas from examples.

The research offered a solid theoretical framework, though it remained largely academic. For years, I hoped to build something practical from it. But the demands of a full-time job made finding the time nearly impossible.

AI finally helped me bridge that gap. Working with it in my spare time, I turned that research into Omnist — an open source Python project on PyPI, with 600+ tests and a CLI, built in two weeks. This post covers what Omnist does. (How two weeks was even possible is its own post.)

What Omnist actually does

Omnist's core idea is a single tree model underneath everything: a node is either a scalar value or an ordered list of labeled edges. JSON, YAML, TOML, and XML all map onto that same structure. That's what lets Omnist convert between them without a bespoke converter for every pair.

Omnist also has its own native data and schema languages built on that model. The Omnist Markup Language (OML) is the native data format — the one with zero adjustments needed to map onto the underlying tree. The Omnist Schema Definition (OSD) is the text syntax for defining schemas. Named record types, with closed fields, each given a cardinality range. A field's type is either a fixed scalar or a reference to another record.

Simplicity is the design philosophy behind that schema model. Every field has exactly one type — no unions, no enums. That's a real constraint. But it's a deliberate trade: it's what keeps compatible_with and equivalent actual computations instead of heuristics. Once a field can be "this type, or that, or null," there's ambiguity left over — and compatibility checking gets stuck on it. There's one deliberate escape hatch: an any type, added later. It doesn't reopen that ambiguity. It has to be spelled out explicitly, or explicitly asked for — never something the tool reaches for on its own. And its cost is stated up front: compatibility checking stops wherever any starts.

That foundation gives you:

Convert between any of those formats, ingest and export freely, without hand-writing a converter for each pair — or develop a plugin to add a new one.
Validate a document against a schema, with exact, path-based error reporting instead of a vague "something is wrong somewhere."
Schema-directed deserialization: read an untyped text format into typed data, upgrading values like ISO date strings into real date/time/datetime types whenever the conversion is value-exact.
Infer a schema automatically from example documents.
Ask equivalent(a, b) and compatible_with(v1, v2) — do two schemas accept exactly the same set of documents, or is one backward-compatible with the other.
Normalize a schema down to its minimal equivalent form.
Extract the minimal subschema that recognizes only a chosen subset of fields. This was the paper's original headline result — it trimmed real-world industrial XML schemas down to a fraction of their size. It's a CLI command too: omnist schema extract.

All of it works as a CLI and as a Python library. It drops into a CI pipeline or a script equally well.

Omnist is built for software engineers, not researchers. It's meant to answer practical questions you run into while shipping software — not to explore schema theory for its own sake. A few places it fits:

CI gate for config and API schema changes. Run compatible_with in a pipeline step so a backward-incompatible schema change fails the build instead of breaking a downstream consumer in production.
Migrating a config format. Moving a project from YAML to TOML, or vice versa, without writing and maintaining a one-off converter script.
Onboarding a legacy data source. Infer a schema from a pile of existing JSON examples when no one wrote one down, then validate new data against it going forward.
Type-safe ingestion in a data pipeline. Deserialize untyped JSON or YAML straight into typed structures, with errors that point at the exact path that's wrong instead of a generic parse failure.

Compatibility, in practice

A config schema evolves — you add an optional field. Is that safe for everyone still running the old schema?

v1 = parse_schema('record R { "host": string }\nroot R')

v2 = parse_schema('record R { "host": string, "port" [0,1]: integer }\nroot R')

v1.compatible_with(v2)  # True — every v1 document is still valid under v2

v2.compatible_with(v1)  # False — a v2 document with a port isn't valid under v1

That's the whole feature, in four lines. No example corpus, no fixtures, no "looks fine to me." Just an actual, decidable answer — the same way a type checker gives you an actual answer instead of a guess.

Two weeks with AI — so is it any good?

"Built with AI in two weeks" is exactly the kind of claim that should make you suspicious. Here's what actually backs it up: 600+ tests at 100% line coverage. A mypy --strict type-checking gate in CI. Property-based fuzzing. Formal grammars for both text formats, verified directly against the parsers. And every code example in the docs, executed as part of the test suite — so the docs can't quietly drift from what the code actually does.

The part I'm proudest of: the schema algebra is checked three independent ways, not just tested against examples. Two different algorithms check it. One tests subschema inclusion directly. The other minimizes both schemas and checks they're isomorphic. They're fuzzed against each other and required to always agree. On top of that, a brute-force oracle enumerates thousands of small documents. It checks every operation's answer against the literal, set-theoretic ground truth — about 2.5 million individual checks. When I mutated the code to deliberately reintroduce a bug during testing, that oracle caught it immediately, with a concrete counterexample. That's the property I actually wanted from the 2010 research — not "the tests pass," but "the algorithm is provably doing what it claims."

Try it

pip install omnist gets you the library and the omnist CLI. Head to omnist.dev to get started:

Quickstart — up and running in five minutes
Real-life example — a worked end-to-end walkthrough
API reference — full Python API docs
CLI docs — all commands and flags

The source is at github.com/omnist-dev/omnist.

If you try it, I'd love to hear what you think — what worked, what didn't, or what you wish it could do. Find me on X (@lee_tom) or LinkedIn.

Next up

A Team of One, Running Two Teams — how a sixteen-year-old paper became a working package in two weeks, and the two roles AI played to make that possible: a Research Assistant to argue with, and a DevOps team to run unsupervised.

Omnist: Canonical Schema and Data Model for JSON, YAML, TOML, and XML

Thomas Lee — Sun, 19 Jul 2026 09:55:03 +0000

If you've ever hand-written a converter between JSON and YAML, or guessed whether a schema change would break someone downstream, this might be useful.

I built Omnist — an open-source Python library with one tree model underneath JSON, YAML, TOML, and XML. Read any of them into the same structure, validate it against a schema, write it back out to any of the others. No per-pair converters.

I put together a short deck walking through the model and implementation:

👉 omnist.dev/presentation.html

It also ships a schema algebra, all decidable rather than heuristic:

compatible_with — will every old document still validate under a new schema?
equivalent — do two differently-written schemas accept the same documents?
normalize — collapse a schema to its canonical minimal form
extract — the minimal subschema for a subset of fields

100% line coverage, mypy --strict, property-based fuzzing.

The story behind it

Omnist started as a 16-year-old idea from my PhD research, built out in two weeks of spare time with AI doing most of the heavy lifting on both the thinking and the shipping side. If you're curious how that actually worked — not just "AI writes code fast," but how research decisions and engineering execution stayed separated even with one person running both — I wrote it up as a five-part series:

Try it out

Quick start: omnist.dev/quickstart

Install Omnist:

pip install omnist

Feedback and issues genuinely welcome — github.com/omnist-dev/omnist.