Determinism Will Not Recover the Spec.

2026-10-23

People say this about rewriting an old system. The old system is deterministic. With the right method, it can be rebuilt completely. The first half holds. The second half has to change.

Determinism means one thing. Given the input and the state, the output is fixed. It does not recover the spec. The hard part of the old code is what was never written down. Implicit behavior left by old patches. Compromises on the boundary. Defects that users already rely on. The reason for doing it that way, lost after the people left.

The hard part of a rewrite is not the coding. Writing code is getting cheaper. What gets expensive is checking that the new system and the old one are the same thing, and finding a person who understands the old behavior and is allowed to say this may change. Before you check, write down three things. What you compare by. How large an input space you compare over. How sure you are right now. This piece is how those three become a process. The example only shows the shape. It is not a manual you can start work from.

1. Inventory the traces. Split them in two.

The first step is not reading the code, and not inventing inputs. Inventory what the system already left behind. Missing logs are normal. The system is not invisible. Database state counts. Requests and responses upstream and downstream. Front-end events and support tickets. Queues and leftover files. Monitoring and packet captures, when they exist.

These traces do not depend on someone who can explain the business. They depend on the system having recorded something. After the inventory, split them.

Traces you can pair with an input and an output can become differential cases. Same input. Compare the two outputs. A trace that has only an outcome, and no input, only says that something happened here. The same row in a table can be left by many inputs. That trace is not a recovered spec. It says there is behavior here, and the input has not been found.

Keep the two kinds apart, or the later cost is wrong. Outcome-only traces need a separate probe to recover the input. If you cannot recover it, register a gap. Do not write it up as a case.

If there are no logs, no upstream or downstream records, no one who knows the business, and the code is too tangled to analyze, do not start by replacing the whole system. Strangle it. Move one function at a time at the edge. This is the same method on a smaller surface. Observe, diff, adjudicate, freeze a baseline. The surface is one that real traffic can hit. A method has to say how to do the work, and when not to replace the whole system.

2. Static analysis proposes. Observation confirms or kills.

Each method has a blind spot. One of them is not enough.

A person clicking only sees paths that person could imagine. A branch that exists in the code and was never triggered will not appear because you spent more time. Rare paths, error paths, and implicit behavior left by old patches sit outside the observer's imagination. Observation gives real samples. The boundary of the sample is the boundary of imagination.

Static analysis can extract branches, conditions, constants, enums, state transitions, call chains, and external dependencies. It cannot recover business meaning. It cannot tell you whether a real input will reach a branch. It often cannot tell dead code from a hidden door. It gives structure. Structure is not behavior.

Static analysis turns paths nobody thought of into hypotheses waiting to be checked. Observation confirms them or kills them. Do not make "at least one input per extracted branch" the task. Old code, dynamic languages, reflection, and implicit entry points fill that list with noise. If the denominator is the number of extracted branches, people will invent inputs for dead branches, and noise will pull the confidence down.

The loop that holds:

Extract the structure of behavior. Treat it as hypotheses. The aim is an objective structure. Understanding the business can wait.

Design inputs for hypotheses that look like real branches. Boundaries and errors first.

Run them. Record input, output, and the judgment. Put that into the differential cases.

Check back. Which hypotheses were confirmed. Which observation overturned. Which are unreachable. Which no input has hit yet. Which are only provisional.

Anything no input has hit is a risk. Do not write it down as absent. Do not count it as covered.

A judgment of unreachable needs a basis. It counts as closed only when the basis says this: no entry on the call chain of this version, and reflection and external triggers have been checked. If the basis stops at "this static call chain shows no entry," mark it provisional. A provisional call is not closed. It is also not "no input designed yet." An input you have not built, and an entry you have not confirmed, are different. List provisional calls on their own, with a condition for review.

Accuracy is not the value of static analysis. The value is that unthought paths become hypotheses you can kill. When observation overturns a hypothesis, the guess in the code and the old behavior disagree. Why they disagree is classified in section 4. Do not switch the new system over to the observation here.

3. Write the equivalence standard before you pick a method.

The new code passing its own tests does not mean the behavior matches. High coverage only means a path was reached. It does not mean the path was right. When generated tests check generated code, the judge and the athlete are separate only if the mechanism, the context, and the standard are different. One generation process used on both sides is self-certification.

Write the standard first. Pick the method after. Do not demand all four kinds on every module. Choose per module, and write the choice down.

Output match. Same input, same returned result. This fits interfaces close to pure functions. This kind defaults to recovering the old behavior.

Side-effect match. Writes to the database, messages sent, calls to external systems: same content, same order. This fits stateful parts. This kind also defaults to recovering the old behavior.

Timing. Response time, concurrency, and retries fall inside a range you wrote down. That range is usually a service level you are willing to sign now. It is not a number read out of the old system. The old system may be slower. It may also be fast where nobody promised speed. Decide the allowed gap first. Then compare.

Error behavior. On bad input, compare the error code, the error message, and whether the fallback matches. An old error may be a compromise left on purpose. It may be an accidental defect. A match is compatibility. A change is a business decision. Classify first. Then decide whether to follow it.

When the first two conflict, the business side ranks them in advance. Do not decide in the middle of testing. For the last two, the acceptable range is itself the result of a decision. Calling all four "equivalence" mixes recovered behavior with a standard you just set.

Methods, cheap to expensive. Contract tests read the interface that was written down. Differential tests feed the same inputs to both sides and compare outputs. Shadow runs copy real traffic to the new system. Users still hit the old one. The new results are recorded and not returned. Formal proof is for a few modules. It proves a containment relation with the old behavior. Most modules will not get there, and do not need to. Proof is not the end of convergence.

Pick by the risk of the module. The whole system does not need one method.

A difference is not a failure. It is an error signal. Measure it. Sort by business impact. Close the high-impact ones first. Register low-impact ones as known deviations. The difference here is the old system and the new one disagreeing on the same input. A code hypothesis disagreeing with the old system is a separate ledger. Section 4 says how to classify after an overturn, how to gather evidence when you cannot tell, and when to stop gathering. Section 5 says how to schedule the adjudicator. "Sort by impact" replaces neither.

4. After an overturn, classify. If you cannot tell, gather evidence, and write down when to stop.

Static analysis can say the path exists. Observation can say what the path actually returns. Together they still cannot say whether this is a business promise. Observed behavior is not a promise by itself.

The person in this role needs both. They understand what the old behavior means in the business. They have the authority to decide whether it may change. Those two often sit in different people. On a legacy system, the expensive part is often finding that person, not writing the new code.

Keep the seats apart. A product owner can change an external promise, and often does not remember why an old defect was left. A domain expert knows why. After they leave, the people who remain often cannot sign. Customer success knows what users rely on, and usually cannot change a contract alone. The engineer writing the new system does not hold either seat. They do not both play and declare the two systems equivalent.

When the two seats are different people, the person who understands explains, and the person with authority signs. The signature belongs to the product owner, or to whoever now owns that line of business. The person who understands may be a domain expert still around, or someone who can match a complaint to the old behavior. A person who cannot sign does not turn "I remember it used to work this way" into a decision.

After observation overturns a code hypothesis, classify before you keep it, change it, or leave it open. The entries are different. Do not collapse them into "look it up and file it."

Intentional compromise. Someone knew it would do this, and left it because of another constraint. Users or downstream systems may already rely on it. The entry is evidence of intent: an old decision, the constraint at the time, a person still here who can say why it was left. A signature to keep it keeps a promise. A signature to change it is a new business decision, and it names who is affected.

De facto commitment. Nobody can show it was intended. Someone now uses this result to finish the next job. The entry is a fact of reliance. Intent evidence is optional. A signature to keep it keeps existing reliance. The record says there is no evidence of intent, the commitment comes from use, and it names the job and the kind of person. Changing it names who is affected. Do not file this class as an intentional compromise.

Accidental defect. Nobody intended it. Nobody uses it to get work done, or using it causes harm. There is one entry. Someone explicitly claims it was a mistake, and can give the grounds for those two sentences. A signature to keep it inherits a defect. The record must say "inherited defect." It must name when the transition stops. Under this class, the default is not to follow the observation. Keeping it needs a separate reason.

Unsettled. The people here cannot say why. The facts of reliance do not point at a specific person and a specific next job. The only evidence is "the code does this," or "this happened and someone reported it." That stays open. Production doing it does not promote a defect into a promise.

The evidence ladder does not produce an accidental defect. When the cheap paths are done and still show no reliance, stop on the lines below, as long-term open or as a known deviation. "We found no reliance" does not fill the accidental-defect class. Without that explicit claim, leave the class empty.

Write the basis on the same record. Who claims it. Whether the basis is an old decision, reliance inside tickets, special handling downstream, or code alone. Code alone, with nobody who can state the intent and no reliance you can point at, is unsettled.

When it is unsettled, gather evidence. Write three lines for stopping.

Do not wait for someone who remembers. They may never appear. Push unsettled toward a class you can sign. You are looking for two things. Was there intent. Does someone use this result to finish the next job.

Cheap to expensive:

Tickets. Read which kind of thing the ticket says. Someone used this result to finish the next job. A manual replacement order, a reconciliation, a way around another process. That is reliance. Someone reported that this should not have happened. That is an objection. An objection says you should not keep the behavior as it stands. It does not support a de facto commitment. Read the support script the same way. It either confirms the behavior is expected, or it apologizes and routes around it. Only the first kind, and only when it keeps showing up, supports reliance.

Upstream and downstream contracts. If a caller or a downstream system has special handling or a special branch for this behavior, the behavior is an interface promise in fact, whether or not it was intended. A downstream system that accepts the result and does not distinguish this path has not blocked it. Failure to block is not a contract that the path must continue.

Data distribution. How many records, how often, whether it clusters on a customer or a period. Volume does not classify. It decides whether more cost is worth spending.

Shadow run. Run both. The new system emits what the code hypothesis says. Users still hit the old system. Watch whether downstream systems and users report a break. A report counts when you can point at whose next job broke. Then the behavior is relied on. No report does not make it a defect. It means current traffic did not hit whoever relies on it.

A controlled experiment. Send a small slice of real traffic through the new behavior and watch the metrics. Highest risk, highest cost. The guardrails are below.

Evidence does not replace the decision. It is material for the person who signs. Write three stop lines, or evidence-gathering becomes a queue that never closes.

Stop line one. Enough to classify. Stop. Reliance has to name who uses this result to finish the next job. One of these is enough: reliance inside tickets, special handling downstream, or a shadow-run report that names the person and the job. With evidence of intent, the class is intentional compromise. With reliance and no evidence of intent, the class is a de facto commitment. "It happened," "someone reported it," and "downstream did not block it" are not enough. A pile of objections, with no fact of reliance, misses this line too. Complaints do not send the new system after the observation.

Stop line two. The cheap paths are exhausted and still will not move. Stop. Tickets, upstream and downstream, and the distribution have all been checked. No reliance you can point at. No evidence of intent. Nobody claims an accidental defect. High impact stays long-term open, and the module is not reported as equivalent. Medium and low impact are registered as known deviations. Do not default to the old behavior because the search stalled. That smuggles unsettled in as a promise. Do not write an accidental defect because you found no reliance.

Stop line three. Cost is over the cap. Stop. Evidence-gathering has a budget: time, people, and traffic affected. Past the cap, stop. Mark it long-term open. Write the cap and the review date. The goal is whether anyone now relies on this result to get work done. The goal is not the historical truth. Do not spend without limit on a rare branch.

Guardrails for a controlled experiment. Who approves: the adjudicator and the technical lead, both. If either is missing, do not run it. The blast radius can be split. It can be rolled back. The metrics and the stop condition are written down. The traffic fraction is set in advance. If a metric crosses the threshold, roll back at once. Do not wait for the experiment window to end. Use this only for high-impact differences the other paths cannot close. It is not a routine tool.

The rest of the duty stays. The difference list is signed. A signed remainder is an explicit risk. An unsigned one comes back, in a later iteration, as "nobody knows if this is right." Each record says who, when, on what basis, which class, and what was decided for that difference. The classes are intentional compromise, de facto commitment, accidental defect, and unsettled. If an accidental defect is kept, the record also says "inherited defect" and when the transition stops.

What cannot be decided is registered as open, with a condition for review. Traffic past a stated volume. One evidence path returning a result. The budget cap reaching its review date. Do not leave it open with no date. A high-impact difference that is still unsettled means the module is not equivalent. If the open items are the bulk of the work, you do not have equivalence. You have a queue. If this role is empty from the start, the rewrite stops at observation. Do not finish the new code and then go looking for the person.

5. Confidence is three counts. The adjudicator's time goes in a queue.

Confidence is computed from the hypothesis list in section 2. Report all three counts. Split the middle one. Provisional calls do not enter the three.

First count. How many hypotheses observation confirmed. The hypothesis and the old behavior agree.

Second count. How many are closed, and the result is not the original hypothesis. Split this. How many observation overturned: the path is live, and the real output disagrees with the guess in the structure. That behavior is closed. It still goes to section 4. It does not automatically follow the observation. How many are judged unreachable: dead branches, no entry, the basis written down, the noise removed, no business consequence. A basis that stops at "this call chain shows no entry" does not enter this half. Mark it provisional, on its own. Both halves are results. They are not still unknown.

Third count. How many still have no input. Only inputs you have not built go here.

Reporting only the first count is a coverage illusion. Adding the two halves of the second count into one number makes closed behavior look like residue. Folding provisional calls into unreachable reports an unconfirmed entry as closed. What is still open is the third count, plus the provisional list. The standard says what you compare by. The input space says where these hypotheses were hit. Confidence is these three counts, with the second count split, and the provisional list visible on its own.

On a legacy system, human observation is often the only path that works. "Give it more time" must not become waiting. Four further constraints:

Directed construction. Rare paths and error paths get inputs designed from the hypothesis list and from the data. More time spent rewatching the hot path does not raise coverage by itself.

A frozen baseline. The longer you watch, the more the old system changes. The business changes. Dependencies get upgraded. Patches still land. Write down the time window, the version, and the batch of data. Later changes are registered as differences. The target does not move.

Deliver in stages. Do not wait until observation is finished before you rewrite. For each module, hand over the three counts, the provisional list, and the classification of high-impact differences. Without a convergence people can see, the method holds on paper and runs out of patience inside the organization.

Freeze the cases. Observation is not a spec. The record of input, output, and judgment is. Every observation enters the differential set. Without that, a longer watch only scatters.

Cost sits on a separate sheet. Hours of human observation. Tools and people for static analysis. Upkeep of the differential environment. The adjudicator's time. The budget for evidence. The risk you will pay later for differences that remain. Without that sheet, you cannot say whether the process can continue. The adjudicator's time is usually the hardest line to schedule. When there is not enough of it, the queue below is the schedule. Another sentence of "look at the high-impact ones first" is not a schedule.

Grade first. Then decide whether this hour includes a person.

High. Money, an irreversible side effect, a promise the user can see, or many callers. The module is not equivalent until these differences are classified. The class may be intentional compromise, de facto commitment, accidental defect, or unsettled. Unsettled stays open, and section 4 starts, with the stop lines written down. An accidental defect needs an explicit claim. "We found no reliance" does not fill it in.

Medium. The behavior differs, the blast radius is one internal caller, or a mistake can be taken back. No separate meeting. The adjudicator reads one short list. Each line already has four things: the code hypothesis, the observed output, the proposed class, and who claims it, on what basis. One sitting, one batch. Incomplete claims stay off the list. A proposed de facto commitment names who uses the result to finish the next job. A ticket number alone is not complete.

Low. Unreachable, or no output difference a caller can see. Register it as a known deviation. It does not take adjudicator time this round. If traffic hits it later, raise it to medium or high. Provisional calls do not enter this grade. An unconfirmed entry is not "no consequence."

Incomplete evidence does not take the adjudicator's time. It goes back to observation or to evidence-gathering. What is still missing is the actual output, and who claims a compromise, a de facto commitment, or a defect. "The code looks like it" is not complete. "Someone reported it" is not complete.

Fix a block of time each week. Cap the list. What does not fit stays in the queue. Staying in the queue does not become "follow the old behavior for now." That would collect a defect into a promise with no decision. If a high-impact item is still unclassified past this stage, this stage does not report the module as equivalent.

6. One small module. The path only.

This section is a sketch. It is not a manual. The module is fictional. One interface, four hypotheses, one signature. The traces that pair an input with an output are drawn as a few neat lines. The sketch makes the shape visible. On a real system, traces with no input are more common, and the evidence for a class is rarely as complete as "someone still remembers." Do not start work from this section. The classification rules, the stop lines, and the adjudicator queue are in sections 4 and 5. Those sections are the rules. They are still not a full operating manual.

Sort the traces first. The gateway log has requests and responses: order id, cancel reason, status code. Those pair an input with an output. They can enter the differential cases. The order table has many rows marked cancelled, and no request from that time. Outcome only. Register a gap. Do not write a spec. A support ticket says cancel still succeeds after shipment. That is a hint, not a case. It tells observation to hit the input "already shipped." The ticket does not yet say whether this is reliance or an objection.

Static analysis proposes four hypotheses, before any business explanation.

1. Paid, and not shipped. Cancel returns success and writes the status to cancelled.
2. Already shipped. Cancel returns a conflict. The status stays.
3. An empty cancel reason still cancels successfully. The branch is in the code. The comment looks added later.
4. An internal flag skips every check. No entry is visible on the call chain.

Observation hits them. Hypothesis 1: the input matches, the output matches, confirmed. Hypothesis 2 is overturned. A shipped order still returns success, and the status becomes cancelled. The path is live. The real output disagrees with the hypothesis in the code. Stop here. Do not announce that the new system follows the observation. Hypothesis 3: the logs contain no request with an empty reason, and this round built no input for it. Not hit yet. Hypothesis 4: the static call chain of this version shows no entry. Reflection and external triggers are unchecked. Provisional, listed on its own, pending review. Not written down as confirmed absent. Not placed among the closed.

The three counts. Confirmed: 1, cancel succeeds before shipment. Closed, and not the original hypothesis: 1, the overturn. Cancel still succeeds after shipment. Unreachable this round: 0. Not hit yet: 1, the empty cancel reason. Hypothesis 4 is provisional, on its own, outside the three counts. Still open: the one not hit, plus this one provisional call. Only the overturned hypothesis enters classification.

The equivalence standard was written in advance. This interface needs matching output and matching side effects: the order status, and the cancel message that goes out. Timing is out of scope this round. The overturned line is high. A user can see it, and it changes an order.

Take the branch where someone remembers. An old operator remembers. Years ago the warehouse interface was unreliable. They left "cancel still allowed after shipment" on purpose, so a person could file a manual replacement. Support can show that people still use it, and that the use is the replacement order. That is an intentional compromise. The basis is the old constraint and the current use. The basis is not "the code does this," and not "a ticket mentioned it." The product owner signs. The class is intentional compromise. What is kept is a promise. The new system follows the observed output. It does not follow the hypothesis in the code that this should return a conflict.

Then take the branch where nobody can say why. Gather evidence as in section 4.

Tickets. Of 14 tickets in the past year, 11 are support using "cancel still works after shipment" to file a manual replacement. The user is waiting on a refund or a reroute. Someone uses this result to finish the next job. The other 3 are users objecting: the goods already shipped, so how can cancel still succeed. Those are objections. They do not support a de facto commitment.

Upstream and downstream. Callers do no special handling of this status. The warehouse only recognizes cancelled. It does not distinguish before and after shipment. That only means downstream did not block the path. It is not a contract. The reliance does not come from here.

Distribution. This path is about 3% of cancels, clustered in promotion season. The volume says the cheap paths are worth finishing. It does not classify by itself.

The three cheap paths are done. There is reliance, from those 11 replacement orders. There is no evidence of intent. Stop on line one. The class is a de facto commitment. The signature keeps existing reliance. The record says: no evidence of intent. The reliance is support using cancel-after-shipment to file a replacement, not the 3 objections. A change has to notify support, and the users who finish a refund or a reroute through that replacement. Do not write intentional compromise. Do not write accidental defect. The 3 objections only say that some people do not want this result. Nobody has claimed that this was a mistake and that nobody uses it to get work done.

If all 14 tickets had been objections, stop line one would fail. A report alone is not a de facto commitment, and it does not send the new system after the observation.

If the three paths end with no reliance you can point at, and nobody claims an accidental defect, stop on line two. High impact stays long-term open. The module is not reported as equivalent. Medium and low impact become known deviations. Do not default to the old behavior.

Hypothesis 4 stays provisional. It does not enter this adjudication, and it does not enter the low grade of known deviations. The note says: no entry on the current call chain; reflection and external triggers not checked; confirm at runtime. Hypothesis 3 has not been hit, so it is not classified. The high-impact line is already signed as a de facto commitment, so this module can enter differential testing. Hypothesis 3 is not absent. Hypothesis 4 is not unreachable. This sketch is not a procedure other modules can copy.

What this round hands over is not new code. It is a frozen baseline, cases that have inputs, three counts split open, one provisional call on its own, and one signature that names the class.

7. Quality control on the new code is not equivalence with the old system.

Some teams string red lines, mutation tests, human review, and a final acceptance check into one quality gate. The problem is real. Generated code cheats. Coverage gets inflated. A local edit does not converge. A team writing code with a model can use those constraints directly.

What that gate checks is whether the new code stands up, with a person at the end. The spec is mostly someone reading the old code and generating a document. That is a sample. The gate checks the new code's own tests. It does not check the old system and the new one on the same input. There is no paired run, no three counts, no classification after an overturn, and no stop lines for when you cannot tell.

A small team with a strong reason, treating the old project as the requirements document, can run that gate. It cannot answer why the rebuild should be called complete. As a quality constraint inside one effort, the description is fair. As the whole method of a rewrite, it sits in the wrong place.

Close

Seven things turn a rewrite into a process.

Inventory the traces. Only an input paired with an output becomes a case. An outcome alone is a gap. If the traces as a whole are too thin, do not replace the whole system. Run the same loop on a surface that real traffic can hit.

Static analysis produces hypotheses. Observation confirms or kills them. What you cannot hit is a risk. The number of extracted branches is not the coverage denominator. An overturned hypothesis is a closed disagreement. It is not yet a decision about which side the new system follows. Unreachable needs a written basis. A thin basis is provisional. List it on its own. Keep it out of the closed count. Do not write it down as absent.

Write the equivalence standard per module. Output and side effects default to recovering the old behavior. For timing and error behavior, decide the allowed gap first, then compare. Rank conflicts in advance. The comparison is between the old system and the new one.

The business decision classifies first. Intentional compromise enters through evidence of intent. A signature to keep it keeps a promise. A de facto commitment enters through reliance you can point at. Evidence of intent may be missing. The record names the job. An accidental defect enters through an explicit claim that it was a mistake, and that nobody uses it to get work done, or that using it causes harm. To keep one, write "inherited defect" and the date the transition stops. Finding no reliance does not fill this class. Unsettled stays open. Live behavior, or a report that it happened, does not promote it into a promise. When the person who understands and the person who can sign are different, the first explains and the second signs. The person writing the code holds neither seat. A high-impact item that is still unsettled means the module is not equivalent. An empty role means you stop at observation.

When it is unsettled, gather evidence, and write three stop lines. Enough to classify: stop. Enough means you can name who uses this result to finish the next job, or you have evidence of intent. Cheap paths exhausted: stop. Over the cost cap: stop. Split tickets into reliance and objection. A downstream system with no special handling is not a contract. The question is whether anyone now relies on this result to get work done. The question is not what the original author meant. A controlled experiment needs both signatures, a way back, and a stop condition. Use it only when the impact is high and the other paths are stuck. If you cannot close it and the impact is high, leave it open for the long term. Do not report the module as equivalent.

The adjudicator's time enters a queue by grade. High and still unclassified: this stage does not report the module as equivalent. Medium: one short list, read in one sitting. A proposed de facto commitment names who uses the result to finish the next job. Incomplete evidence does not consume this time. Low: register only. Provisional calls stay out of the low grade. What does not fit the cap stays in the queue. It does not default to the old behavior.

Report confidence as three counts. How many confirmed. Inside what is already closed, how many were overturned by observation, and how many are unreachable on a written basis. Both of those are results. How many no input has hit. Provisional calls stay on their own list. They are not folded into unreachable. What is still open is the not-yet-hit count, plus the provisional list. Freeze the baseline. Deliver in stages. Freeze the cases.

With all seven, "rebuilt completely" stops being a sentence nobody can judge. It becomes a process you can measure, decide, and stop. Drop one piece, and the work falls back to this: the new code passed its own tests, so it is done. The example only shows the shape. It does not stand in for the classification, the stop lines, or the adjudicator's queue.