DEV Community: Travis Frisinger

Characterization Tests Are How Agents Enter Legacy Code

Travis Frisinger — Wed, 08 Jul 2026 16:30:08 +0000

Originally published at tddbuddy.com.

Related reading: Your Test Suite Is Your API for Agents names the interface; The Bar for TDD Just Moved names the floor. This post is about codebases that have neither yet, and how an agent gets a foothold anyway.

Legacy code is code without a specification. An agent changing legacy code is changing code with no specification.

Those two sentences are the entire problem. The first is a fact about the codebase. The second is what happens when the first is true and the team hands the code to an agent anyway. The agent reads what is there, infers what the code "probably" does, makes the requested change, runs whatever tests exist, and ships. The change is correct against the agent's inference. The inference was incomplete. The behavior the team did not know was load-bearing has now silently shifted, and the regression will surface in production, three releases later, when a customer notices that an invoice rounds to the wrong cent.

Characterization tests are the move that closes this gap. They are old technique, named decades ago in Working Effectively with Legacy Code, and the agent era did not invent them. The agent era made them mandatory. The discipline that used to be optional, the thing experienced developers did when they had time, has become the precondition for letting an agent operate in any part of a codebase that was never properly tested. Which is most of every codebase.

The Agent's Hardest Problem Is Code With No Spec

Sort the work an agent might be asked to do into three buckets.

Green-field code is easy. The agent and the team start from nothing, write tests first, build the implementation against the tests. The discipline is well-understood. The vocabulary is whatever the team chooses to establish. The agent has rails the entire time.

Well-tested legacy code is also tractable. There is a spec, even if the spec is implicit in the existing tests. The agent reads the suite, learns what the team has decided matters, and produces changes that respect the existing contracts. The tests catch regressions. The vocabulary already exists.

Legacy code with no tests is the hard case. It is also the dominant case. Most teams asking "can an agent help us ship faster" are asking that question about systems written by people who left the company three years ago, in frameworks that are two major versions out of date, with behavior nobody fully understands and a deployment cadence that depends on hoping the smoke tests cover the important paths.

The agent reading this kind of code has no way to know what is contract and what is accident. The function that returns null for the empty case might be returning null because that is the contract callers depend on, or it might be returning null because someone forgot to handle that branch and no caller has hit it yet in production. The agent cannot tell. It will pick whichever interpretation makes the prompted change easier and move on. Half the time it picks correctly. Half the time it ships a regression.

The damage is not loud. It is quiet. A small behavioral shift on an edge case, a rounding direction that flipped, an error message that no longer includes the field name, an off-by-one in pagination at the very last page. Each one survives review because the diff looks reasonable and the tests pass. Each one is found weeks later, in production, by a user, and traced back through three deployments to the change nobody questioned.

The fix is not "be more careful." The fix is to give the agent a specification before letting it touch the code.

Characterization Tests Pin "Is," Not "Ought"

Characterization is the act of capturing what the code currently does, exactly, before deciding what it should do. The captured behavior includes the bugs. That is the point.

A characterization test is not a correctness test. It does not claim that the system behaves the way the team wants. It claims that the system behaves the way it currently behaves, and any change to that behavior will register as a test failure. The failure is the prompt for a human to ask: did we want this to change. If yes, update the test. If no, revert the change. Either way, the change is no longer silent.

The discipline matters because the default for untested legacy code is "any change is silent." A characterized legacy module shifts that default. Every change becomes either an intended change (which the human accepts and the test is updated) or an unintended change (which the test fails and the change gets reverted). The third category, the silent regression, stops being possible.

[Fact]
public void Pricing_engine_currently_rounds_half_to_even_for_wholesale_orders()
{
    var input = new PricingInput(
        unitPrice: 0.125m,
        quantity: 100,
        customerType: CustomerType.Wholesale
    );

    var result = pricing.Calculate(input);

    result.Total.Should().Be(12.50m);
}

That test does not claim wholesale rounding should behave this way. It claims wholesale rounding currently behaves this way. The test name reflects the claim: currently_rounds_half_to_even. The "currently" is load-bearing. A future reader, human or agent, sees that the behavior is pinned but the pin is descriptive, not prescriptive. The next decision (do we want half-to-even or half-up) is a separate decision the team can make explicitly, with a failing test as the conversation prompt, instead of discovering it three quarters later when finance asks why the invoices are short.

The naming convention separates characterization from specification. Tests named currently_does_X are descriptive pins. Tests named should_do_X or named in domain language (Wholesale_orders_round_half_up_to_the_nearest_cent) are specifications. The same suite can hold both, but the distinction has to be explicit, because the two kinds carry different obligations.

Agents Are Good at Capturing, Bad at Judging

Characterization is the rare task where the agent does the bulk of the work and the human does the judgment.

The mechanical part of characterization is exhaustive: enumerate inputs, run the function, record the output, write the assertion. An agent can do this at a scale a human would not finish in a week. Pick a hundred input combinations, exercise the function, capture the outputs, generate a test for each. The agent does not get tired. The agent does not miss the obvious-but-tedious cases (empty string, null collection, max int, negative quantity, leap year date). For mechanical coverage of behavior, the agent is the right tool.

The judgment part is everything else. Reading a hundred captured outputs and asking, for each one: is this the contract, or is this an accident. Is the function returning empty list because that is what callers depend on, or because the implementation happened to take that branch when the input was malformed. Is the rounding behavior intended, or is it a leftover from an old framework's default that nobody ever questioned. Is the timestamp in UTC because that is the team's contract, or because the server's locale was UTC in 2014 and the code never specified a timezone.

The agent cannot answer those questions by reading the code. The answers live outside the code, in the team's history, in conversations with customers, in the original ticket that justified the function in the first place. The agent has no access to those. It has only what is in front of it, which is the captured behavior. The captured behavior cannot distinguish contract from accident on its own.

If the agent is asked to generate the captures and then "approve" them, it will approve all of them. From the agent's perspective they are all equally valid pins on current behavior. The agent's approval ratifies the accidents alongside the contracts, and the resulting test suite is a frozen monument to whatever the code happens to do, including the parts that are wrong.

The human's job is promotion, not approval.

The Human Move Is Promotion, Not Approval

Promotion is the act of reading captured behavior and deciding, per row, what to do with it.

Some captures get promoted into domain vocabulary. The test Pricing_engine_currently_does_X_for_input_Y becomes the test Wholesale_orders_round_half_up_to_the_nearest_cent, with a builder-based setup, a domain-typed assertion, and a name that reflects what the team has decided the system does. The captured row was a description of behavior. The promoted row is a specification of behavior. The vocabulary of the codebase is one entry richer.

Some captures get flagged. The test Pricing_engine_currently_returns_null_for_negative_quantity stays as-is, with a comment: // Captured behavior. Suspected-incidental. Decide before next pricing change. The team has not yet decided what the right answer is, but the wrong answer (silently shifting it) is no longer possible. The capture pins the current behavior until someone decides to move it deliberately.

Some captures get deleted. A row that captures behavior on an input no caller could ever produce is documenting a code path that exists only because the agent's enumeration found it. Pinning it as a test commits the team to maintaining a behavior nobody wants. Better to delete the test and let that branch evolve naturally with the rest of the code.

The three moves (promote, flag, delete) are the actual work of characterization. The agent generated the raw material. The human turned the raw material into a specification. A team that skips this step has a test suite that contains accidents and contracts in equal measure, with no way to tell them apart. A team that does this step has a specification that grows organically out of what the code already does, anchored in the team's vocabulary, defensible under future change.

Rubber-stamping the agent's captures is the legacy-code version of an agent grading its own homework.

Characterization Is Where the Vocabulary Starts in a Legacy Codebase

The other thing that happens during promotion is more important than the tests themselves.

The first promoted test in a legacy codebase is also the first named scenario, the first builder, the first domain type. Before promotion, the codebase has primitives and incidents. After promotion, it has the beginning of a controlled vocabulary. The act of turning Pricing.Calculate(0.125m, 100, "Wholesale") into Wholesale_orders_round_half_up_to_the_nearest_cent() is the act of naming the concepts the system has been operating on all along without naming them.

CustomerType.Wholesale is a new type that did not exist before. aWholesaleOrder() is a new builder that did not exist before. 0.125m * 100 becoming anItemPriced(0.125m).orderedAt(100.units()) is the beginning of a fluent grammar that did not exist before. The legacy codebase had none of this. The characterized codebase has the seeds of it. Each promotion is a seed. Each seed is a vocabulary entry.

This is where the legacy on-ramp connects to the larger arc about test suites as the team's controlled vocabulary. Characterization is the first stretch of road. It is also the road that everything else gets built on. A team that promotes characterizations into named scenarios with builder-based setup is bootstrapping a vocabulary out of an untested system. A team that captures and rubber-stamps is creating a frozen snapshot with no vocabulary at all.

The on-ramp is not just about getting agents safely into legacy code. It is about building the vocabulary the agent will need to do anything useful in that code beyond the first task.

Golden Masters Are Characterization's Blunt Instrument

A golden master is the simplest characterization technique. Capture the entire output of a function or system for a representative set of inputs. Compare against the captured baseline on every test run. Any change to any output fails the test.

Golden masters are cheap to write, exhaustive in coverage, and brutal under refactoring. They catch every change, intended or not. They speak implementation, not domain. The test Module_X_output_matches_baseline tells the next reader nothing about what Module_X does, only that it does the same thing it used to. A failing golden master tells the developer something changed without telling them what was supposed to change.

For the very first stretch of road, that is fine. Golden masters are scaffolding. They keep the agent honest while the team builds toward something better. The trade-off is explicit: the team gets coverage today, without spending the time to write behavioral assertions, and they accept that the tests will be noisy under refactoring and silent on intent.

The discipline is to refine. As the team promotes captures into named scenarios with domain-typed assertions, the golden master shrinks. The behaviors that matter get pulled out into specific tests with specific names. The golden master eventually covers only the long tail of weird inputs nobody has gotten around to naming, and at that point it is fine for the long tail to live in a snapshot file. The named scenarios carry the contract. The snapshot carries the residue.

Treating golden masters as the final answer is the same mistake as treating captures as approvals. Both leave the team with a frozen system that resists change. The point of characterization is not to freeze the system. The point is to get the system to the place where deliberate change is possible. Golden masters are scaffolding. Scaffolding is meant to come down.

No Characterization, No Autonomy on Legacy

Pull the argument back to the original question.

A team that wants agents to do real work on a legacy codebase is implicitly asking: can an agent operate against code with no specification. The answer is no. Not "no with conditions" or "no until the model gets better." Structurally no. An agent without a specification produces changes that are correct against its inference and incorrect against the team's contract, and the gap between those two is invisible until production catches it.

The fix is to produce the specification before the work starts. Characterization is how that specification gets produced for code that has none. The agent generates the captures. The human does the promotion. The test suite that results is not a comprehensive specification of the legacy system, but it is a specification of the parts of the legacy system the agent is about to touch, which is the part that matters for the next task. The next task adds more characterized regions. The road extends.

A year of this discipline, applied to the parts of the codebase that get touched, produces a codebase where every recently-touched region has a specification, every promoted scenario carries domain vocabulary, and every agent contribution lands against a pinned reference that fails loudly on unintended change. The codebase that was legacy at the start of the year is, by the end, a codebase with an interface for agents. The interface was not written all at once. It was promoted, scenario by scenario, out of what the code already did.

The question "can an agent work in our codebase" is mostly "is our codebase characterized." For most legacy systems, the honest answer is no. That is the work. Characterization is the on-ramp, and the on-ramp is the first stretch of every road agents are going to drive on in your code.

Examples Pin Intent. Properties Pin the Invariants.

Travis Frisinger — Wed, 08 Jul 2026 16:28:46 +0000

Originally published at tddbuddy.com.

Examples pin intent. Properties pin the invariants.

That sentence contains two distinct ideas that most teams collapse into one. The collapsing is the mistake. Example-based tests and property-based tests are not two flavors of the same activity. They are two axes of specification, and a codebase with only one axis is incompletely specified regardless of how many examples exist or how well-named they are. Agents do not compensate for incompleteness. They compose against what is there, and they fail quietly where there is nothing.

The Bar for TDD Just Moved established the floor: scenario-rich, builder-driven, domain-typed tests written in the team's vocabulary. That floor is real and it matters. This post argues that the floor is not a ceiling. The floor describes one axis. There is a second axis that the floor does not address and that nearly every test suite, however well-crafted on the first axis, is missing. The second axis is not harder than the first. It is just different. Different enough that most teams have never named it, never budgeted for it, and never noticed it was absent until an agent made the absence visible by failing in a way no example caught.

This post names the two axes, explains why agents need both, and shows what the vocabulary looks like when both are present.

Examples Are for Naming Intent

A scenario test names a concrete case in the team's language. That act of naming is load-bearing. Before the test exists, the behavior lives only in someone's head or in a ticket that nobody will read in six months. After the test exists, the behavior is executable, refactorable, and discoverable by anyone who reads the suite.

[Fact]
public void Loyalty_members_get_a_ten_percent_discount_on_orders_over_fifty_dollars()
{
    var order = anOrder()
        .forCustomer(aLoyaltyMember())
        .containing(anOrderContaining(twoBooks()));

    var receipt = checkout.process(order);

    receipt.discount.Should().Be(order.subtotal * 0.10m);
}

Read that name. An agent browsing the test suite can answer "what does the system do for loyalty members on qualifying orders?" without reading any implementation. A new team member can find the canonical definition of the loyalty discount rule in one search. The builder vocabulary (aLoyaltyMember, anOrderContaining, twoBooks) pulls every piece of setup into the domain's own nouns. The assertion is typed: receipt.discount is a Money, not a decimal, not a double, not a magic number floating in context.

The scenario is a named point in the input space. "This input configuration produces this output." That is the full semantic content of an example-based test, and that semantic content is worth having. Without named scenarios, agents compose against nameless cases and the vocabulary never settles. The first post in the TDD arc made the craft case in detail: builders, factories, and domain types are the grammar of behavioral specification. The claim stands. Scenarios are the vocabulary contract the team uses to speak about behavior across disciplines and across time.

Named scenarios do more than verify outcomes. They document decisions. The test name Loyalty_members_get_a_ten_percent_discount_on_orders_over_fifty_dollars encodes three decisions: who qualifies (loyalty members), what the threshold is (fifty dollars), and what the rate is (ten percent). When any of those decisions changes, the test name changes. When the test name changes, the diff tells the story. The history of the test suite is the history of the system's behavioral decisions, and that history is searchable, compilable, and verifiable.

Examples are not optional. They are the named entries in the specification index.

They are also, by construction, finite. The input space is not.

Examples Cannot Verify the Manifold

A function is not a list of cases. It is a mapping over a space. The space for even a simple discount calculation includes every combination of member tier, order total, promotion state, item count, currency, and rounding mode that could plausibly arrive at the checkout service. The scenario above pins one point in that space. The scenario that covers the edge at exactly fifty dollars pins a second point. A thorough example suite pins twenty, fifty, a hundred points.

The space has dimensions that a hundred named points cannot cover.

This is not a complaint about test quantity. More scenarios help, and every scenario named in domain language is worth writing. The point is structural: examples verify what the author thought to enumerate. They do not verify what the author did not think of. The gap between what was enumerated and what is possible is exactly where silent failures live, and it is the gap that agents most reliably exploit.

Consider the failure mode. An agent asked to add a promotional stacking rule modifies the discount calculation. The modification is plausible: it reads the surrounding tests, infers the shape of the calculation, extends it with the new promotion logic, and verifies that the existing scenarios still pass. The agent has no particular reason to probe the interaction between the new promotion and the edge cases the original author did not name, because those cases do not appear anywhere in the surrounding tests. The build passes. The PR opens. The regression ships to production.

This is not a failure of agent capability. It is a failure of specification coverage. The example suite defined what mattered by what it included, and the agent composed against that definition faithfully. The problem is that the definition was incomplete: it named the cases the author thought to enumerate, and left the rest of the space to chance.

Human reviewers used to partially compensate for this incompleteness. A senior engineer reviewing the PR would notice the stacking logic and ask "what happens if both discounts are at maximum values?" That question came from experience: the reviewer had seen discount stacking bugs before. The question is not in the test suite. It lives in the reviewer's head, gets asked in a comment, gets answered with a manual check, and then disappears from the record when the PR merges. The next engineer to touch the calculation has no evidence that the question was ever asked.

An agent reviewing the same PR does not have the senior engineer's pattern memory. It checks the tests. The tests pass. The calculation looks structurally similar to the calculations around it. The PR is fine.

The unbounded region of the input space is not covered by more examples. It is covered by a different kind of claim.

A Property Is a Constraint on the Search Space

A property-based test does not say "this input produces this output." It says "across all valid inputs, this relationship holds." The quantifier moved from existential to universal. That shift is the entire difference between the two axes, and it changes what the test can catch.

Consider the discount calculation. An example says "a loyalty member with a sixty-dollar order gets a six-dollar discount." A property says something more durable:

[Property]
public void Discount_is_always_non_negative_and_never_exceeds_order_subtotal()
{
    Prop.ForAll(
        anyLoyaltyMember(),
        anyOrderWithSubtotalAbove(50.dollars()),
        (member, order) =>
        {
            var receipt = checkout.process(
                anOrder().forCustomer(member).containing(order.items));

            receipt.discount.Should().BeGreaterOrEqualTo(Money.zero)
                .And.BeAtMost(order.subtotal);
        });
}

The generator anyLoyaltyMember() does not return one member. It returns a source over the space of valid members, varying tier, tenure, promotional state, and any other dimension the type can carry. The generator anyOrderWithSubtotalAbove(50.dollars()) does the same for orders. The property-based runner feeds hundreds or thousands of combinations through the assertion on every run. When any combination violates the assertion, the framework reports the minimal reproducing case: the smallest set of inputs that cause the failure. Shrinking is automatic. The counterexample is concrete.

The assertion does not check a specific value. It checks a relationship: the discount is at least zero and at most the order total. That relationship is an invariant: a fact about the function that must hold regardless of which specific inputs arrive. Encoding the invariant as a test means every generated combination that violates it becomes a failing test with a reproducible counterexample.

Other invariants live in every codebase. Decoding an encoded value should recover the original. Sorting a list should not change its length. Merging two permission sets should produce a result that is a superset of both. The running total across line items should equal the sum of individual totals. Adding a discount and verifying the post-discount total should produce the same result regardless of whether the discount is applied before or after tax, if the domain says so. These are not scenarios. They are structural claims about the function's geometry. Writing them as properties makes them executable specifications that no enumeration could have covered.

The design work in writing a property is naming the invariant. That naming is where most of the value lives. The act of asking "what does this function owe to the domain regardless of input?" is a different kind of question than "what should this function produce for this specific input?" It is a question about the function's contract with the system, not its behavior in a named scenario. Teams that have written properties report that the design question is the hard part and the code is easy. The code is a generator and an assertion. The question is a claim about the domain.

The right vocabulary for properties is: constraints, invariants, relationships, and laws. The right vocabulary for examples is: scenarios, cases, and named intents. The two vocabularies are not interchangeable because the underlying things are not interchangeable. A property is not a generalized example. An example is not a special case of a property. They are answers to different questions.

Agents Need Both Axes

An agent working in a codebase with only example-based tests will generate example-based tests. This is not a criticism of agents. The agent composes against the vocabulary and patterns it finds. If the surrounding tests are all scenarios, the agent writes scenarios. Scenario thinking produces more scenarios. The agent is doing exactly what it should.

Property thinking requires something different: knowing the invariants the domain owes to itself. That knowledge is not present in an example-based test suite. The examples show what happens in specific named cases. They do not say what must always be true regardless of case. An agent reading a suite of well-named scenarios can infer a great deal about behavior. It can learn the vocabulary. It can learn the canonical cases. It cannot infer the universal quantifiers that were never written down, because those quantifiers require a claim about the entire input space, and that claim is not present anywhere in a suite of named points.

The practical consequence is asymmetric. Agents are reliable at extending the named-cases axis and unreliable at covering the invariant axis. They will add a new scenario for the new feature because the pattern for scenario tests is legible in the surrounding code. They will not add a property for the new invariant because there is no property-test pattern in the surrounding code to follow, and there is no expressed invariant to follow even if the pattern were there.

This is where the agent's blind spot lives: in the gap between what the named cases assert and what the input space permits. The agent does not know that discounts must be non-negative unless that invariant is expressed somewhere. The agent does not know that the stacking order of promotions must be commutative unless that invariant is expressed somewhere. The agent does not know that totaling across items must equal the aggregate total unless that invariant is expressed somewhere. Without properties, those invariants are invisible, and invisible invariants get violated by code that passes every named test.

This was always true for human authors too. The senior engineer who knew to ask about stacking interactions was compensating for the missing property with experience and intuition. The problem is that the compensating knowledge never made it into the test suite, so the next author, human or agent, started from scratch. The senior engineer's question lived in a PR comment for thirty days and then became invisible. The property lives in the test suite forever.

A two-axis specification closes the blind spot at the source rather than patching it at review time. The example tests tell any reader what the canonical cases are named and what they produce. The property tests tell any reader what must hold across the full input space. Together they describe the function completely: its named points and its global constraints. A function described that way can be composed against, extended, and refactored with confidence. A function described only by its named points can be extended in ways that satisfy all the names and violate the space, and no reviewer, human or agent, is likely to catch the violation.

Builders Compose With Property Generators

The builder vocabulary that example-based tests already use does not need to be replaced to support properties. The grammar is the same. The quantifier moves.

aLoyaltyMember() returns a single instance configured for the canonical loyalty member scenario. It is used in scenario tests where the specific configuration matters. anyLoyaltyMember() returns a generator over the space of valid loyalty members. It is used in property tests where the relationship must hold across the entire space. Both are defined in the same test-support layer, next to each other, sharing the same understanding of what a valid loyalty member is.

// Scenario builder (example-based axis)
public static LoyaltyMember aLoyaltyMember() =>
    new LoyaltyMember(
        tier: MemberTier.Standard,
        tenureMonths: 18,
        promotionState: PromotionState.None);

// Property generator (invariant axis)
public static Arbitrary<LoyaltyMember> anyLoyaltyMember() =>
    Arb.From(
        Gen.OneOf(
            Gen.Constant(MemberTier.Standard),
            Gen.Constant(MemberTier.Gold),
            Gen.Constant(MemberTier.Platinum))
        .SelectMany(tier =>
            Gen.Choose(1, 120).SelectMany(months =>
                Gen.Elements(
                    PromotionState.None,
                    PromotionState.Active,
                    PromotionState.Suspended)
                .Select(state =>
                    new LoyaltyMember(tier, months, state)))));

The method names follow a consistent a/any prefix convention. a means "give me the canonical one." any means "give me a generator over the space of valid ones." A developer reading the test layer knows immediately which axis each method serves. An agent reading the test layer inherits the same convention: when writing a scenario, reach for a; when writing a property, reach for any.

This is not a new API. It is an extension of the existing API into the second axis. The domain nouns are identical. The generators compose the same way the builders do. anyOrderWithSubtotalAbove(50.dollars()) is built from anyOrder() filtered by subtotal constraint, the same way anOrderContaining(twoBooks()) is built from anOrder() configured with items. The grammar the team learned for scenario composition transfers directly to property composition. The vocabulary investment compounds.

The anyLoyaltyMember() generator is also the most authoritative definition of what a valid loyalty member is in the test context. The scenario builder aLoyaltyMember() returns a canonical example. The generator returns the space: all member tiers, any tenure between one and one hundred and twenty months, any promotion state the type can carry. If the domain adds a new member tier, both the scenario builder and the generator need to be updated. The generator makes the update visible in a different way: if the new tier has a distinct discount behavior that violates the invariant, the property test will find a counterexample. The scenario builder will not, because nobody thought to name the new-tier scenario yet.

The vocabulary investment in the scenario layer is not redundant with the generator layer. The scenario layer names the cases. The generator layer covers the space. Both are built from the same domain nouns, maintained in the same file, and read by the same agents.

The Bug That Examples Miss Every Time

The claim that properties catch bugs examples miss is not hypothetical. Here is the pattern, precise enough to reproduce.

A discount calculation is written to apply the loyalty discount first, then any promotional discount stacked on top. The implementation is a simple sequential reduction: compute loyalty discount, subtract it from the subtotal, compute the promotional discount on the reduced subtotal, subtract that. The example suite covers loyalty-only orders, promotional-only orders, and loyalty-and-promotion orders with specific dollar amounts. All named tests pass. The code ships.

Three months later, a marketing campaign introduces a high-value promotional discount for new members who are also in the loyalty program. The discount can reach sixty percent. The calculation is extended to apply the promotional discount first, then the loyalty discount. The named tests still pass because none of them used a promotional discount large enough, combined with a loyalty discount large enough, to push the combined discount past one hundred percent of the subtotal. The code ships.

Four weeks after that, a subset of orders is producing receipts where the combined discount exceeds the order subtotal, resulting in negative totals. Negative totals are impossible by definition: the checkout flow cannot owe a customer money for buying books. The bug is the stacking logic: two sequential percentage discounts, each computed on a running total, can exceed the original subtotal when the percentages are high enough. Neither discount is wrong in isolation. The combination is wrong.

A property test would have found this the first time the stacking logic ran under test:

[Property]
public void Total_discount_never_exceeds_order_subtotal_regardless_of_promotion_stack()
{
    Prop.ForAll(
        anyLoyaltyMember(),
        anyOrderWithActivePromotion(),
        (m, o) =>
        {
            var receipt = checkout.process(
                anOrder().forCustomer(m).containing(o.items).withPromotion(o.promotion));

            receipt.discount.Should().BeAtMost(o.subtotal);
            receipt.total.Should().BeGreaterOrEqualTo(Money.zero);
        });
}

The generator anyOrderWithActivePromotion() generates orders with active promotions across the full range of configured discount rates, including the high-value rates the marketing campaign introduced. The first time the stacking logic produces a combined discount exceeding the subtotal, the framework surfaces the minimal counterexample: a gold-tier member with eighteen months tenure and an active promotion at sixty percent, on an order with a fifty-five dollar subtotal. The assertion fails. The developer sees the counterexample. The bug is fixed before the code ships.

The example suite did not enumerate this combination because nobody on the team thought to name it when writing the original tests. The property did not need to enumerate it because the property expressed the constraint. The generator found the violation by exploring the space systematically, the way an exhaustive enumeration would if exhaustive enumeration were possible.

The property also survives future promotions. When the next marketing campaign introduces an eighty-percent discount for platinum members, the property runs again against the new logic without any modification to the test. The constraint is expressed once. The verification is automatic across every future input the domain permits.

This is the second axis earning its keep.

Two-Axis Specs Are What Agents Compose Against

A function is fully specified when two conditions hold. Its canonical cases are named in domain language as executable scenarios, so any reader can understand what the function does on the important days and verify that it does that for those inputs. Its invariants are pinned as executable properties, so any reader can understand what the function promises across the full input space and verify that the promise is never violated.

Examples without properties leave the search space unverified. Properties without examples leave the vocabulary nameless. Both conditions must hold. Neither is sufficient alone, and the order does not matter: a codebase with rich properties but no named scenarios is navigable by a verifier and opaque to anyone trying to understand what the system is for. A codebase with rich named scenarios but no properties is intelligible and fragile at the boundaries.

The reason this matters for agents is not that agents are careless. It is that agents compose against explicit specifications. An example-based suite makes the named cases explicit. A property-based suite makes the invariants explicit. An agent working against both has the complete specification. An agent working against only examples has the vocabulary but not the rails. The rails are where agentic boundary mistakes happen, because the boundary is exactly the region the examples did not cover.

The Bar for TDD Just Moved named the floor: scenario-rich, builder-driven, domain-typed tests. That floor is necessary. The argument here is that the floor is not the ceiling. Above the floor is the second axis. Property tests are not an advanced technique for library authors or specialists. They are the invariant layer of every specification that includes a calculation, a state transition, a codec, a constraint, or any function where a relationship matters more than a specific output.

Every calculation has invariants. Totals are non-negative. Rates are bounded between zero and one. Reversible transformations reverse. Every state machine has invariants. Valid transitions produce valid states. Terminal states do not produce transitions. Every encoder has invariants. decode(encode(x)) equals x. Every merge operation has invariants: the result contains everything from both inputs, in the right precedence order. These invariants do not appear in example suites unless someone wrote a property for them. Nobody wrote a property because the property-test pattern was not present in the codebase to follow. The agent did not write a property because agents extend the patterns they find.

Two axes break the cycle. When anyLoyaltyMember() lives beside aLoyaltyMember() in the test-support layer, the pattern is present. When the codebase has properties alongside scenarios, agents encountering new calculations have a model to follow. When the vocabulary includes both the a-prefix canonical-case convention and the any-prefix generator convention, the grammar is teachable in a single prompt and learnable from a single glance at the test-support module.

A team that builds both axes is not doing more work than necessary. It is building the specification the system actually requires. The scenarios carry the vocabulary. The properties carry the constraints. Together they define what a function is: not just what it produces on named days, but what it owes the domain on every day, for every input the domain can send.

A specification with both axes is what an agent can compose against without a human standing guard at the boundary.

Examples pin intent. Properties pin the invariants. The system with only one axis is half-specified, and the half that is missing is the half that fails quietly.

Evals Are Tests Wearing a Lab Coat

Travis Frisinger — Tue, 07 Jul 2026 22:37:57 +0000

Originally published at tddbuddy.com.

Related reading: BDD Was a Coordination Tax. AI Just Repriced It is the prior post that this one extends; TDD Already Does BDD, Without the Gherkin and The Bar for TDD Just Moved name the test discipline this argument assumes.

An eval is a test. The industry just put a lab coat on it.

That sentence is the entire argument. Everything that follows is structural detail. The eval-tooling wave arrived with new vocabulary (golden datasets, judges, scorers, rubrics), new file formats, new dashboards, new job titles, and a confident announcement that "TDD does not work for AI" because there is no single correct output. Both moves are mistakes. The first is a vocabulary reinvention that hides what the discipline already knew how to do. The second is a strawman of TDD that good TDDers abandoned twenty years ago when they started asserting on properties and invariants instead of exact strings.

The cost of the lab coat is not aesthetic. It is structural. Treating evals as a separate discipline produces a separate specification language, owned by a separate role, maintained in a separate tool, drifting from the code it claims to describe. That sentence should sound familiar. It is the exact failure mode of feature files in the BDD era, restated with model-shaped nouns. The industry deleted Gherkin and rebuilt it, line by line, under a new name. The coordination tax that disciplined TDD finally killed is being charged again.

This post is about taking the lab coat off.

The Industry Reinvented the Test and Renamed It

Strip away the surface and look at the structure of an eval.

An eval has a named scenario: "respond to a churn-risk customer." It has an assertion: "the response should include the retention offer." It has a verdict on every run: pass or fail. It is run automatically in a pipeline. It blocks deployment if it fails. It is part of how the team knows whether the system works.

Everything in that paragraph is also true of a test. The vocabulary is different. The structure is identical. The "scenario" is what a test has always been. The "assertion" is what a test has always done. The "verdict" is what a test has always produced. The pipeline integration, the deployment gating, the source-of-truth role: all of it is what tests have done for two decades.

The pieces that look new under closer inspection are not.

"Golden datasets" are example tables. Tests have had example tables in parameterized tests, table-driven tests, and theory tests for as long as testing frameworks have existed. The "judge" or "scorer" is an assertion library. The "rubric" is an assertion list. The "trace viewer" is a test report. The "eval harness" is a test runner. Every piece of the eval-tooling stack maps to a piece of the testing stack that already exists in every mature codebase. The mapping is not approximate. It is exact.

The lab coat is the framing that the work is novel. The work is not novel. The work is testing.

"TDD Does Not Work for AI" Misreads What a Test Is

The standard objection is that TDD assumes one correct output. The model produces variable output. Therefore TDD does not apply. The argument sounds airtight to anyone whose mental model of TDD ends with Assert.Equal(expected, actual).

That mental model is not what disciplined TDD looks like.

A disciplined test asserts on properties of the output, not on exact equality of the output. The test of a discount calculator does not assert that the receipt is one specific dollar value. It asserts that the receipt has a discount line, that the discount is in a valid range, that the total matches the subtotal minus the discount. The test of a search ranker does not assert that the top result is one specific document. It asserts that results are ordered by relevance, that no result has a negative score, that the requested filter is respected.

The shift from exact assertions to property assertions happened years before models entered the suite. It happened because real systems have non-deterministic components everywhere. Floating-point arithmetic. Timestamp generation. Set iteration order. Concurrent execution. Database row ordering. Sort stability. Any of those will produce output that varies run to run while still being correct, and assertion-on-exact-equality breaks under all of them. The test design that survived the encounter is the test design that asserts on what must hold, not on what happened to appear.

A model is one more non-deterministic source. Apply the same discipline. Assert on the shape, length, presence of required content, absence of forbidden content, satisfaction of domain constraints. Skip the exact-string comparison that was never the right tool. The "TDD does not work for AI" claim is a critique of a strawman. The version of TDD it critiques was wrong for everything, not just for model-backed features.

[Fact]
public void Draft_to_a_churn_risk_customer_stays_under_two_hundred_words_and_includes_the_retention_offer()
{
    var draft = drafter.Compose(anEmail().to(aChurnRiskCustomer()));

    draft.WordCount.Should().BeLessThan(200);
    draft.Should().MentionThe(retentionOffer);
    draft.Tone.Should().Be(Tone.Empathetic);
    draft.Should().NotMention(internalCodename);
}

That is a property assertion. The model can produce a thousand different drafts that satisfy it. The test does not care which one comes back today. The test cares that whatever comes back is short enough, mentions the offer, lands in the right tone, and does not leak the codename. Those are the properties the team has decided matter. The test is the encoding of that decision.

No lab coat required.

Non-Determinism Was Always a Test-Design Question

The deeper version of the objection is that model output is not just variable, it is statistical. A test that passes on one run might fail on the next, not because the system changed, but because the model sampled a different token at temperature 0.7. Aggregating across many runs is the standard answer: run the eval N times, measure pass rate, set a threshold.

Aggregate scoring is a real technique. It belongs in the suite. It does not require a parallel specification artifact.

A test that runs N times and asserts on a pass rate is still a test. The framework needs to support running a scenario repeatedly and aggregating the result, which is a feature most test frameworks already have under the parameterized-test banner or which can be added with a few lines of helper code. The assertion changes from "this property holds" to "this property holds in at least 95% of runs." The structure is unchanged.

[Fact]
public void Retention_offer_appears_in_at_least_ninety_five_percent_of_churn_drafts()
{
    var draftsContainingOffer = Enumerable.Range(0, 100)
        .Select(_ => drafter.Compose(anEmail().to(aChurnRiskCustomer())))
        .Count(d => d.Includes(retentionOffer));

    draftsContainingOffer.Should().BeGreaterThanOrEqualTo(95);
}

That test is the eval. It is also the test. The lab coat would have called this a "rubric pass rate" and put it in a separate dashboard. The discipline calls it an assertion on a probability and puts it in the suite. The verdict is the same. The location is not.

The same applies to model-as-judge patterns. If the assertion uses another model to score the output (does this draft sound empathetic?), the judge is just an assertion implementation. Wrapping a model call inside IsEmpathetic(string text) is the same kind of wrapping that wraps a floating-point comparison inside IsApproximately(double expected, double actual, double tolerance). The assertion library grew an entry. The test structure did not change.

The Golden Dataset Is a Feature File

This is where the BDD echo gets loudest, and the warning is the same.

A golden dataset is a separate file, in a separate format, often owned by a separate role, containing input and expected-output pairs that the system is supposed to match. Engineers do not write it directly. They consume it through a harness. The dataset evolves on its own schedule. The code evolves on its. Six months later, the two have drifted: features added to the code that the dataset does not cover, examples in the dataset for features that were removed, expected outputs that reflect an older version of the prompt.

Replace "golden dataset" with "feature file" and the paragraph is verbatim the post that this series wrote two arcs ago about why BDD tooling died.

The pattern is the same. A second specification language, separated from the code, owned by a different role, drifting at a different rate. The justifications are the same: non-engineers need to read it, the format is more accessible, the discipline of writing it is "different" from writing tests. The failure modes are the same: drift, ownership confusion, ceremony, a coordination tax paid every time anyone changes anything.

The industry already learned what to do with this pattern. Delete the separate artifact. Put the specification in the test suite, in domain language, expressed with builders and assertions the engineers actually maintain. Let non-engineers read the tests directly, because the tests are now named in language the business uses and structured around scenarios the business recognizes. The coordination cost collapses because the artifact collapses.

The same move applies to evals. The golden dataset belongs in the suite, expressed as parameterized scenarios with property assertions, named in the team's vocabulary. Anyone who reads the test sees what the team has decided the model-backed feature should do. Anyone who edits the test edits the specification directly. There is no second file to keep in sync. There is no second role to coordinate with. The drift becomes structurally impossible because the artifact does not exist as a separate thing to drift from.

The lab coat sells the parallel artifact. The discipline deletes it.

Evals Belong in the Suite, in the Vocabulary

The right home for model-backed feature behavior is the same home as every other feature's behavior: the test suite, in the team's domain language, using the same builders and domain types the rest of the suite uses.

A model-backed feature is, structurally, a collaborator. It takes input, produces output, has a contract the team cares about. From the suite's perspective it is no different from a third-party API, a database, a calculation engine, or any other collaborator the system depends on. The test pattern for collaborators is well-established. Mock the collaborator at the seam when you want to test how the surrounding code uses it. Hit the real collaborator in contract tests when you want to verify the collaborator itself holds up. Both kinds of test live in the same suite, in the same vocabulary, in the same repo, in the same commit.

The vocabulary is the lever. When the test reads anEmailDraft().forA(churnRiskCustomer()) asserting Should().MentionThe(retentionOffer), the team's domain language is doing the work. The test names a concept (churn-risk customer) that exists in the business. It asserts on a concept (retention offer) that exists in the business. It uses builders that compose with the rest of the suite's builders. A reviewer reading this test does not have to switch contexts to a separate eval framework's vocabulary. The reviewer reads a test in the team's language and understands what the model is being asked to do.

A new team member reading the codebase finds the model-backed feature alongside every other feature, specified the same way, runnable the same way, refactorable the same way. The discoverability problem the eval-tooling stack solves with a dashboard is solved here by the file system: the tests live next to the code they specify, named in language anyone on the team recognizes.

This is what the disciplined-TDD position has been saying for years, applied to a new audience. The eval-tooling wave is the latest group to arrive at the realization that specifying behavior is the work. The discipline that already specifies behavior has nothing new to learn from the wave, but the wave has a great deal to learn from the discipline.

The Split Is a Role Artifact, Not a Technical One

The reason evals live outside the suite in most organizations is not technical. It is organizational. A separate team owns "the AI part" of the product. That team needs a way to specify behavior. The existing test suite belongs, in the team's mind, to a different team (the engineering team that owns the rest of the product). So the AI team builds its own specification artifact, in its own tooling, in its own repo. The split exists because the role split existed first.

That is the same dynamic the BDD post named, with the cast updated.

Feature files lived outside the test suite because a separate role (the analyst, the QA, the BA, depending on the era) needed an artifact to own. The split was a treaty between roles, not a technical decision about where specifications should live. When the role boundary compressed (analyst and engineer became the same person, or worked closely enough that the handoff disappeared), the artifact compressed with it. The feature file evaporated into well-named scenario tests in the same suite as everything else. The treaty was no longer needed, so the protocol stopped being maintained.

The role boundary around AI features is in the same place feature-file ownership was a decade ago. There is a separate team. They need an artifact to own. They build evals. The boundary is real today and produces real artifacts. The boundary is also compressing, fast, as model-backed features stop being "the AI part" and start being "the part that uses a model among five other collaborators." When that compression finishes, the eval suite will collapse into the test suite the same way feature files collapsed into scenario tests. The artifact will follow the role.

Teams that read the trajectory now and merge the suites early get to skip the coordination tax that the rest of the industry is going to pay for the next three years. The merge is not hard. The merge is structurally available the day the team decides to do it. The barriers are organizational, not technical.

One Suite, One Vocabulary, One Source of Truth

Pull the argument back to where it started.

The lab coat on evals hides three things at once. It hides that evals are tests with new vocabulary. It hides that the "TDD does not work for AI" objection critiques a version of TDD nobody disciplined was practicing. And it hides that the separate eval suite is the BDD coordination tax wearing a different costume.

Strip the lab coat off and the work is testing. The same discipline. The same artifacts. The same vocabulary. The same suite. Model-backed features are specified alongside every other feature, in the team's domain language, using the same builders that every other test uses. The agent reads the suite and finds a unified specification. The reviewer reads the suite and finds one place where intent lives. The team maintains one artifact, not two.

The eval suite should be the test suite. The lab coat comes off. The work that was always testing gets called what it was.

There is no eval suite and test suite. There is the suite. Everything the system does, including the parts a model does, is specified there, in the words the team uses for everything else.

The Hidden Output of TDD Was Never Code

Travis Frisinger — Tue, 07 Jul 2026 22:37:57 +0000

Originally published at tddbuddy.com.

This is the first post in the three-part Vocabulary Is the Product arc. The follow-ups are Agents Amplify Whatever Vocabulary They Find and Product Literacy Is the New Core Engineering Skill.

The visible output of TDD was always the wrong thing to measure.

Teams pointed at the green bar, the coverage percentage, the regression safety net. All real. All undersold what was actually accumulating inside the test suite while the team ran the loop. The passing tests were the receipt. The vocabulary was the asset.

Every well-named builder method, every domain type, every scenario name was a small act of product thinking crystallized into a callable identifier. By the time a mature TDD codebase had a few thousand tests, it also had a working dictionary of the system's concepts: what exists, what it's called, and how concepts relate. Nobody called it that. Nobody tracked it. It accreted quietly in the test helpers and the factory methods and the scenario names, and it was the most durable thing the team ever built.

What You Thought You Were Producing

The honest answer is that most TDD practitioners believed the output was the test suite. Specifically: passing tests, regression coverage, and the refactoring safety net that comes when a suite is green and comprehensive.

Those benefits are real. Regression coverage means changes are visible before they ship. A green suite means the refactor step has a net. The safety net matters. It's what lets teams keep moving as the codebase grows. The teams that pitched TDD to skeptical management pitched these benefits because they were measurable. They showed up in bug counts and deployment frequency and time-in-QA. The pitch worked often enough that TDD got adopted, then re-adopted, then adopted again after the first adoption decayed.

But calling those benefits the "output" is like calling the printed page the output of a writing process. The page matters. The ideas the page carries are the point.

The loop produced discipline over naming, shape, and intent on every commit. Red forced a name before implementation existed. Green forced a minimal implementation against that name. Refactor forced the implementation and its name into alignment with the rest of the system. Each step was a constraint on the form of the output, not just on its correctness. Three constraints per commit, thousands of commits over the life of the codebase, and what accumulated was not just a suite of passing tests. What accumulated was a shape. A structure. A set of names that had been pressure-tested against the domain and sharpened by a thousand tiny review loops.

That discipline accumulated. The accumulation had a form. The form was a vocabulary: a set of named concepts, expressed as types, builders, and scenarios, that encoded the team's shared understanding of the system.

The tests were the container. The vocabulary was the contents.

What You Were Actually Producing Was a Vocabulary

Run the loop a thousand times on a single codebase and look at what's left. Not the assertions. Look at the names.

There is a controlled vocabulary: a deliberate, curated set of names for domain concepts. Not every string or variable in the production code, but the terms that appear repeatedly in test code because they represent things the domain cares about. LoyaltyMember, not CustomerType == 1. Money, not decimal. aCartReadyForCheckout(), not a twelve-line setup block.

Every test data builder was a vocabulary entry. Every scenario name was a vocabulary entry. Every domain type was a vocabulary entry. Each one represented a decision: this concept exists in this system, and this is what we call it.

The decision had to be made before the test could be written. Red forces the question. To write a failing test for a behavior, the behavior has to be named. The naming is the first act of design. The implementation that follows is secondary.

Consider the same scenario written three ways:

// WEAK: no vocabulary, only machinery
var c = new Customer();
c.Type = 1;
var o = new Order();
o.Total = 60m;
var d = new DiscountService().Calculate(c, o);
Assert.Equal(6m, d);

// MIDDLING: types exist, but the scenario is still assembly
var customer = new Customer { LoyaltyTier = LoyaltyTier.Gold };
var order = new Order(60.00m, customer);
var discount = new DiscountService().Calculate(customer, order);
discount.Should().Be(6.00m);

// STRONG: the test is a sentence in the system's language
[Fact]
public void Gold_loyalty_members_get_ten_percent_off_orders_over_fifty_dollars()
{
    var order = anOrder()
        .forCustomer(aLoyaltyMember().withGoldStatus())
        .containing(anItemCosting(60.dollars()));

    var receipt = checkout.Process(order);

    receipt.Discount.Should().Be(6.dollars());
}

The third version does not just pass the same test as the first. It carries a different vocabulary. aLoyaltyMember(), withGoldStatus(), anItemCosting(), 60.dollars(), receipt.Discount: these are not just more readable names. They are vocabulary entries. They are decisions about what concepts exist and what the domain calls them. The test body is a sentence in the system's language because the team built a language for the system.

The first version has no vocabulary. It has primitives and magic numbers. The magic number 1 means something, but nothing in the test says what. The decimal 60m means something, but nothing calls it out as a threshold. The implementation might be correct. The vocabulary is absent.

The controlled vocabulary the team built by writing the third kind of test was the actual durable output. The test suite was the place it lived.

Vocabulary Outlives the Code

Implementation has a short life. Frameworks change. Languages get replaced. Runtimes get upgraded. The code that was correct in 2015 has been rewritten two or three times since then, migrated to containers, moved to a new cloud provider, and touched by twenty developers who were not there when the original design was made.

Vocabulary has a long life.

The concepts that mattered to the business in 2015 still matter. "Loyalty member" still means something. "Free shipping threshold" still means something. "First-year member" still means something. What changed was the implementation. What persists is the set of concepts and the names the team chose for them.

The persistence is not accidental. Business domains are stable in ways that technology is not. The rules governing whether a customer qualifies for free shipping change slowly and deliberately, negotiated with stakeholders, reflected in policy documents. The framework the application runs on changes because a new version came out, a security patch required it, the cloud provider deprecated it, or a new hire preferred something else. The domain outlives the stack. The vocabulary that encoded the domain should outlive the stack too. It does, when the test suite was the place it lived.

This is not a theoretical point. Watch what happens when a team rewrites a service in a new language or framework. Two paths diverge.

The first team carries the vocabulary. They extract the domain types, the builder methods, the scenario names, and the factory hierarchies before they start writing the new implementation. The new codebase looks different internally, but the concepts are the same and the names are the same. Developers who know the old system can navigate the new one because the vocabulary is the same. The test suite for the new system reads like a translation of the old test suite, not a reimagining. The system survives the rewrite.

The second team keeps the code and loses the vocabulary. They refactor with autocorrect, copy logic blocks, rename things to match the new framework's conventions, and gradually drift from the domain language the business still uses. Six months later, a product manager says "loyalty member" and the developers say "that's UserRank == 2 in the new system" and that's when everyone discovers how much tacit vocabulary was lost. The domain knowledge that lived in the test factories and the scenario names did not survive the migration. The system got rewritten. The system also got lost.

A team that rewrites a service and keeps the vocabulary keeps the system. A team that keeps the code and loses the vocabulary lost the system.

The code was always replaceable. The vocabulary was the moat.

Builders Are the Lexicon

A test data builder is not a convenience for avoiding boilerplate. It is a grammar.

Every builder method is a verb or modifier in the domain language. forCustomer() is a verb. withGoldStatus() is a modifier. inTheirFirstYear() is a modifier that encodes a temporal concept the business cares about. Every parameter type is a noun: Money is a noun, LoyaltyTier is a noun, Address is a noun. Stack them and you have a grammar: subject, qualifiers, action, assertion. The grammar is not arbitrary. It mirrors the grammar the business uses when it talks about the system.

That mirroring is the point. When the builder API and the business language are the same, new requirements enter the codebase without translation. A product manager says "first-year loyalty member," and a developer types aLoyaltyMember().inTheirFirstYear(), and the phrase compiles. The concept maps directly because someone, during a red step months earlier, was forced to name the concept before implementing it and chose the domain's word over a technical shortcut.

When that mapping breaks, the cost is invisible at first and catastrophic later. The product manager says "first-year loyalty member" and the developer translates it mentally to CustomerTenure.New and two weeks later there's a support ticket about "new customers" getting a benefit they shouldn't because "new" and "first-year" turned out to mean different things and no one noticed the concepts had drifted apart.

To compose a test scenario using a rich builder API is to write a sentence in the system's language. The sentence has structure: subject (who is acting), context (what conditions apply), and event (what action is being taken). Reading a test body out loud is reading a sentence in the domain language. If the sentence doesn't parse, the builder API is wrong, or the domain model is wrong, and the test is telling you so before the implementation gets a chance to hide it.

var order = anOrder()
    .forCustomer(aLoyaltyMember().inTheirFirstYear())
    .containing(anItemCosting(75.dollars()))
    .duringBlackFriday();

var receipt = checkout.Process(order);

receipt.Discount.Should().Be(15.dollars());

Read that out loud: "An order for a first-year loyalty member containing an item costing seventy-five dollars, placed during Black Friday, should produce a fifteen-dollar discount." That is an English sentence. It is also a C# expression. The builder API made those two things the same thing.

Now compare to what happens when the vocabulary is weak and the team needs to rename a core concept. Say the business decides that "loyalty member" should be called "rewards member" everywhere. In the strong version above, the change is one rename: the aLoyaltyMember() factory. Every test that uses it updates in the same refactor sweep. The compiler finds every call site. Nothing is missed.

In the weak version:

// Every scattered usage needs a manual grep
var c = new Customer();
c.Type = 1;  // Is 1 a loyalty member? A rewards member? Both?

The magic number 1 cannot be found by a rename. There is no token to rename. The concept exists only as a convention scattered across dozens of files. The refactor becomes archaeology: find every place where Type == 1 appears and determine whether it means the concept that just got renamed. Some of those usages are loyalty. Some are something else. Nobody is sure. The rename takes a week and still misses edge cases in a reporting module nobody touched.

The third vocabulary state rewrites itself. The first vocabulary state needs grep and hope.

The Vocabulary Was Invisible Because Nothing Tracked It

Here is the design failure at the center of the TDD conversation for twenty years: the vocabulary was the output, and nothing ever made that visible.

There was no file called domain-language.md. No lexicon checked into the repo. No metric that measured vocabulary health. No CI gate that failed when a new primitive type snuck into a test where a domain type should have been. The vocabulary accreted inside the test suite as a tacit norm: the senior developer knew that aLoyaltyMember() was the right factory and new Customer { Type = 1 } was the wrong approach, and they would mention it in code review if they caught it, and they would miss it roughly half the time because the diff was long and the type mismatch was subtle.

The vocabulary was real. It was load-bearing. It was completely invisible to organizational processes.

So nobody talked about it as an asset. When teams discussed the business case for TDD, they talked about bug catch rates and regression coverage and time saved in QA. They did not say "we have been building a domain-specific language for the system and it lives in our test factories and every new developer who reads it learns how the business thinks about the problem." That would have sounded either pretentious or vague. "Our coverage went from 40% to 80%" sounded concrete.

The coverage number was trackable. The vocabulary was not. The coverage number got the business case. The vocabulary got ignored.

And so, over time, vocabulary erosion was also invisible. A factory got duplicated with a slightly different name. A scenario name drifted toward implementation language. A domain type got replaced by a string because a junior developer was in a hurry and the review didn't catch it. Each erosion was small. The cumulative effect was a test suite where the vocabulary was no longer trustworthy, where two factories existed for the same concept with slightly different setups, where scenario names mixed business language and technical language in the same sentence.

Nobody noticed the asset depreciating. They only noticed, eventually, that the codebase felt harder to work with than it used to.

The Test Suite Is the Controlled Vocabulary

Stop thinking of the test suite as a safety net. Start thinking of it as a living lexicon.

The test suite is the controlled vocabulary of the system. Every builder method is an entry. Every domain type is an entry. Every scenario name is an entry. The vocabulary is controlled in the sense that it was decided upon: someone chose aLoyaltyMember() over new Customer { Rank = 2 }, and that choice encoded a judgment about what concept matters to the domain and what the domain calls it.

The vocabulary is the moat.

This is not a metaphor. A codebase with a rich, stable, expressive vocabulary is genuinely harder to replace than one without. The first time a team tries to port it, migrate it, or hand it off to a new team, the vocabulary is what carries the domain knowledge. The new team reads the test suite and learns the system. They learn what a "loyalty member" is, how a "first-year" window is defined, what qualifies as an "order over threshold." They learn this from the code because the code uses the domain's words.

The domain words are doing work documentation cannot do. A wiki page that says "loyalty members are customers who have signed up for the rewards program" is a sentence. aLoyaltyMember().inTheirFirstYear() is a sentence plus a type system plus a suite of scenario tests that describe exactly how "first year" is calculated, what happens at the boundary, and what changes when the policy changes. The vocabulary entry is executable. The wiki page is not. The vocabulary entry is current because it has to be current to compile. The wiki page is current until someone forgets to update it, which is always.

A codebase without vocabulary has to be explained. Every handoff requires tribal knowledge transfer. Every migration requires someone who was there when the original decisions were made. The vocabulary that was never built into the tests has to be reconstructed from memory, which is expensive, slow, and inaccurate.

Every refactor that strengthens names strengthens the asset. Every drift weakens it.

The refactor step in TDD was not just about tidying the implementation. It was about maintaining the vocabulary. When a new concept appeared in an implementation and its name was fuzzy, the refactor step was the moment to sharpen it. When two builders were doing nearly the same thing under slightly different names, the refactor step was the moment to collapse them. When a scenario name described the mechanism instead of the intent, the refactor step was the moment to rename it.

The teams that cheated on the refactor step did not just get tangled implementations. They got vocabulary drift. They got a test suite where the names slowly stopped meaning what the domain meant. They got a lexicon with entries that contradicted each other, that overlapped without explanation, that mixed languages without intention.

That is the real cost of skipping refactor. Not just messier code. A degraded asset.

The Output Was Always a Language

TDD practitioners spent twenty years making the case for the green bar.

The green bar matters. But the green bar was never the point. The green bar was the constraint that forced the naming. The constraint that forced the naming built the vocabulary. The vocabulary was the product.

The test suite is not a safety net draped over the production code. It is the place where the team built a language for the system, commit by commit, factory method by factory method, scenario name by scenario name. The production code is the back-translation: the implementation that had to satisfy the language the tests defined.

Read a well-maintained TDD codebase's test suite and read the system's domain model. They are the same document. The test suite is the specification, the glossary, and the grammar, all compiled, all executable, all current. The production code proves the specification is achievable. The test suite is the specification itself.

The visible output of TDD was a passing test suite. The actual output, the one that compounded value over the life of the codebase, was the vocabulary the team built while writing it. Every test that forced a naming decision, every factory that encoded a domain concept, every scenario that pinned business intent to a callable identifier: these were the asset accumulating. The green bar was just proof the asset was still intact.

That vocabulary is now the interface agents compose against. It is the controlled surface through which the next generation of tools reads the system. Teams that built it will find their agents producing coherent, domain-appropriate output. Teams that skipped it will find their agents producing faster primitives.

The hidden output of TDD was never code. It was always the language.