DEV Community: Yura Solovey

Can We Debug AI Models?

Yura Solovey — Mon, 27 Jul 2026 12:58:59 +0000

We are not really working with AI models yet — we are playing with them like children who have been allowed to press buttons on a supercomputer. We already know a few commands. We can produce an impressive result and repeat a sequence that sometimes works. But we do not understand what happens inside, why the model chose this particular answer, or what will break after the next edit.

Until we learn, at least well enough, how a model moves through an instruction and reaches a decision, we will use only the small fraction of its capabilities that we happened to notice from the outside. We will keep pressing buttons instead of controlling the process. We will obtain isolated results instead of using the system to its full potential.

In the first three parts of this series, a long instruction became a hidden program, a correct output stopped being proof of a correct process, and local validators proved unable to guarantee global control. One final question remains: can we step through a model’s execution like a program in a debugger and see exactly where it took the wrong turn?

We Are Still Changing AI Instructions Like a Black Box

In a conventional program, a bug may not be fixed on the first attempt, but a debugger radically narrows the search space. We set a breakpoint, run the program, inspect variable values, step through transitions, and see which condition fired. With an AI process, we often work differently: we add a rule about approval, run the process, see the same failure, repeat the rule in the correction section, get a pass — and then discover that another branch has broken. From the outside, this looks like debugging. In practice, it is black-box testing by trial and error.

The worst part is not that one change can cause a regression. The real problem is that we often do not know why the second run passed. Did the model actually start applying the new rule? Did it happen to choose another branch? Did the new wording outweigh another part of the context? Or did the old failure simply not appear in this run? When all we can see is input → output, a positive result explains the mechanism of improvement no better than a negative result localises the mechanism of failure.

One Wrong Result Can Have Several Different Causes

Let us return to the running example with the PREFERENCE_CHANNEL_CHANGED event. After a correction, the technical specification moved to version 4, but Jira, QA, and approval remained on version 3. In the final package, the result looked the same regardless of where the failure actually occurred:

expected:
  technical_spec_version: 4
  jira_version: 4
  qa_version: 4
  approval_version: 4

actual:
  technical_spec_version: 4
  jira_version: 3
  qa_version: 3
  approval_version: 3

From that output alone, we can form at least five different hypotheses:

the model did not notice the correction;
the correction was noticed but classified as non-material;
the invalidation rule was not selected;
dependent steps were not rerun;
the completion check ignored the stale status.

All of them can produce almost the same final package, but they require different fixes. If the model failed to notice the correction, we need to change the change-detection mechanism. If it saw the change but classified it as insignificant, we need to refine the materiality rules. If the classification was correct but the invalidation rule was not applied, the problem lies between decision and action. If invalidation happened but the workflow reused cached artifacts, rewriting the prompt may be the wrong layer of intervention altogether. And if everything was updated correctly but the completion check ignored stale, then the final invariant is wrong.

Without a debugger, we see one failure and five possible places to patch. So we add a rule where it seems most logical and hope we guessed correctly.

What Should a Debugger for an AI Process Show?

In a conventional debugger, we care about more than the final exception. We want to know which data entered a function, what values the variables held, which condition returned true, which branch was selected, and what state existed after the step. For an AI process, the questions are remarkably similar:

which node was executing;
what data the model received;
what answer the human provided;
what state the model believed was current;
what change it detected;
how it classified that change;
which rules it considered;
which rule it applied;
which path it selected;
which alternatives it rejected;
what state existed after the decision.

That record already looks much closer to step-by-step debugging:

debug_step:
  node: classify_correction

  input:
    old_trigger: "old_value == email and new_value == sms"
    new_trigger: "new_value == sms"

  state_before:
    technical_spec_version: 3
    jira_version: 3
    qa_version: 3
    approval_version: 3

  detected_change:
    field: trigger_condition
    detected: true

  decision:
    classification: non_material
    selected_rule: non_material_change_keeps_approval

  state_after:
    technical_spec_version: 4
    jira_status: current
    qa_status: current
    approval_status: current

Now we can see a specific breakpoint. The model did not miss the correction. It noticed the change in the correct field, classified it as non_material, and then applied a rule that kept dependent artifacts current. That is a completely different quality of diagnosis. Instead of the vague “the model updated the approval incorrectly,” we get a specific place to change: explicitly define every change to trigger_condition as a material correction.

But State and Transition Still Do Not Explain Why the Model Chose That Path

Recording that the model selected non_material is useful, but it is not always enough to make a precise change to the instruction. We also need to see which alternatives it considered, which rule it selected, which facts it relied on, and what it ignored. That is why the debug_record below contains considered_rules, selected_rule, short_reason, and evidence_refs: they show not only the external transition, but also the working representation of the decision at that node.

In our example, the model may have given too much weight to the fact that the event name and schema did not change, while underweighting the change to trigger_condition, which expanded the population of affected events. Without this layer, we fix the symptom and risk adding an overly broad rule such as “every correction invalidates approval.” That rule would close this failure, but it would also force Jira, QA, and approval to be regenerated after a cosmetic typo correction. Debugging is valuable precisely because it lets us change the classification boundary instead of merely strengthening the prohibition.

A Debug Log Should Not Be One Large Explanation at the End

The easiest way to obtain an explanation is to ask the model after the failure: “Why did you reuse the old approval?” It will almost certainly produce a coherent answer. But that answer may have been created after the decision. The model sees the output, sees our question, and generates a plausible story that fits the result. That is useful as a hypothesis, but weak as a debugger.

A stronger approach is to record the decision while the model is passing through the critical node and to separate execution facts from the model’s own explanation immediately:

debug_record:
  sequence: 17
  node: classify_correction
  recorded_before_transition: true

  execution:
    state_before_ref: snapshot_16
    input_refs:
      - correction_message_12
      - technical_spec_v3
    selected_transition: update_technical_spec_only
    state_after_ref: snapshot_17

  model_report:
    classification: non_material
    considered_rules:
      - material_change_invalidates_dependents
      - non_material_change_preserves_approval
    selected_rule: non_material_change_preserves_approval
    short_reason: "Event name and output schema did not change"
    evidence_refs:
      - field:event_name
      - field:event_schema
      - field:trigger_condition

In this record, execution says what the system actually did, while model_report says how the model described its own decision. The first layer proves which transition was selected and what state resulted. The second helps us localise the rule or criterion that may need to change. They must not be merged, because otherwise a polished explanation can silently replace the factual execution record.

State Diff Works Like an Inspection Window

During a long process run, a full snapshot is still necessary as a recovery point, but for a human reader the diff is usually more useful. It immediately shows what changed after the node and what should have changed but did not.

state_diff:
  from: snapshot_16
  to: snapshot_17

  changed:
    canonical.trigger_condition:
      from: "old_value == email and new_value == sms"
      to: "new_value == sms"
    canonical.version:
      from: 3
      to: 4

  unchanged:
    - artifacts.jira.status
    - artifacts.qa.status
    - approval.status

  expected_but_missing:
    - artifacts.jira.status: stale
    - artifacts.qa.status: stale
    - approval.status: stale

The failure is visible here almost without explanation. The canonical value changed, but the dependent statuses did not. The debug_record then helps explain why invalidation did not happen.

Knowledge Trace Shows What the Model Actually Used

In a large instruction, one rule may be described in several places. In addition, the model sees chat history, user responses, examples, templates, and previous artifacts. When it makes a decision, we need to know not only the rationale it gives, but also which sources it treated as relevant.

knowledge_trace:
  node: classify_correction

  rules_used:
    - rule: approval_required_before_completion
      source: section.approval_rules
    - rule: non_material_changes_preserve_approval
      source: section.correction_handling

  user_evidence_used:
    - correction_message_12

  artifacts_consulted:
    - technical_spec_v3

  available_but_not_used:
    - dependency_map
    - material_change_examples

Now the improvement can be even more precise. Perhaps the necessary rule already existed in the instruction, but the model did not use it because the dependency map was too far from the correction section or was not explicitly bound to this node. In that case, copying the rule again may be worse than restructuring the context or binding the rule explicitly to the step.

Accumulating Knowledge by Node

One debug trace can help fix one run. The real value appears when failures accumulate not as a random bug list, but as knowledge about specific nodes, transitions, and rules.

node_improvement_record:
  node: classify_correction

  observed_failures:
    - run_017:
        failure: trigger_change_classified_as_non_material
    - run_024:
        failure: dependency_impact_underweighted
    - run_031:
        failure: cosmetic_example_overgeneralized

  suspected_pattern:
    "Materiality is inferred mainly from schema compatibility"

  proposed_change:
    "Add explicit semantic-impact criteria before structural compatibility"

  regression_risk:
    "May over-classify wording-only edits as material"

  required_tests:
    - trigger_condition_change
    - event_name_typo
    - description_only_change
    - schema_extension

This is no longer just a log. It is improvement memory for one node. When we change the instruction, we can see not only one failure, but the history of how this node failed, which fixes were already tried, which regressions they caused, and which scenarios must be rerun.

Replay Turns Debugging into a Controlled Experiment

In a conventional debugger, after a patch we repeat the same execution path. With an AI process, that is harder because a new run may receive different wording, a different context, or reach a different decision before the problematic node. If we simply restart the entire playbook, we are not always testing the change we intended to test.

We need replay with fixed inputs and the state captured immediately before the failure:

replay:
  source_run: run_017
  start_from:
    node: classify_correction
    state_ref: snapshot_16

  fixed_inputs:
    correction_message: correction_message_12
    technical_spec: technical_spec_v3
    dependency_map: dependency_map_v2

  changed_instruction:
    material_change_rules_version:
      from: 5
      to: 6

  expected:
    classification: material
    transition: invalidate_dependents

The improvement loop now begins to resemble real debugging: locate the node, inspect state and decision, change one rule, replay from the same snapshot, and only then run the regression suite.

Without replay, we change a rule and restart the entire process, but we do not know whether the new result was caused by that change.

Does Such a Trace Show the Model’s Real Thinking?

This is the hardest question in the article — and it is where the analogy with a conventional debugger begins to break down.

In a traditional program, a variable really does hold a concrete value at a concrete moment. A condition either returned true or it did not. A function received certain arguments and selected a particular branch. A debugger does not ask the program to explain itself; it reads the actual state of the mechanism.

With a model, the situation is not so simple. When it writes short_reason, considered_rules, or factors_used, we are not necessarily revealing a hidden variable that existed inside all along. We are obtaining a structured reconstruction of how its behaviour can be described in the language of our playbook. That reconstruction may be useful, accurate, and sufficient to localise a problem. But it does not become a fact merely because the model wrote it into a debug log.

That is why debug_record must separate two layers rigidly:

execution — what the system actually received, did, and changed;
model_report — how the model described the cause of its decision.

The first layer can serve as the foundation of control. The second must be treated as a hypothesis.

Suppose the model claims that its decision depended on correction_material: false. We change only that factor and rerun the node:

causal_probe:
  baseline:
    correction_material: false
    selected_action: reuse_approval

  intervention:
    correction_material: true

  expected:
    selected_action: invalidate_approval

  observed:
    selected_action: reuse_approval

If the action does not change, the claimed factor was not a sufficient cause. The behaviour may have been determined by another part of the context, cached state, or a combination of signals that does not fit a discrete scheme such as “the model chose rule A instead of rule B.”

That distinction matters. Sometimes the problem is not that the real decision is hidden from us. The problem may be that no clear decision existed in the form assumed by a traditional debugger. In that case, model_report is not an exposed internal mechanism. It is our best behavioural model.

A good AI debugger should therefore not promise full internal transparency. Its job is more modest and more practical: turn the model’s explanation into a testable hypothesis, and prevent that hypothesis from replacing the facts of execution.

What Do We Actually Get?

Such a debugger does not turn us into “the adult who finally understands the supercomputer.” It does not guarantee full access to the model’s internal mechanism, and it does not prove that each decision existed as a clean sequence of rules, alternatives, and causes.

It gives us something else — and for engineering, that is more important: an instrument panel around a system that may remain partly opaque.

We can reliably record:

which inputs arrived;
which node executed;
what state existed before and after the step;
which transition was selected;
which artifacts changed;
which gates passed;
which path can be reproduced through replay.

Separately, we can ask the model to describe the rules, factors, and alternatives it supposedly relied on. But that description is not the foundation of control. It is a diagnostic hypothesis that must be compared with the execution record, state diff, and causal probes.

The real transition from play to engineering does not happen when the black box finally explains itself completely. It happens when the process remains controllable even when the model’s explanation is incomplete, simplified, or wrong.

At that point, improving a long instruction stops being an endless cycle of “rewrite the prompt and see what happens.” It becomes a process of accumulating evidence: this node fails under this combination of inputs; here is the rule it selected; here is the factor that, according to its own report, influenced the decision; here is the causal probe; here is the patch; here is the replay; here is the regression suite; here is the new improvement record.

The Boundary of the First Series

At the beginning of this series, we had a large prompt that behaved like a hidden program. Then we learned to distinguish a polished output from a correct process. Next, we made global state explicit and saw that local pass results do not guarantee correct completion. The final step is to make the execution path visible enough that the process can be debugged and deliberately improved, not merely tested.

If, after every failure, all we can see is the prompt at the beginning and the document at the end, are we really debugging an AI model — or merely asking the black box for another attempt?

Every Step Passed. Who Was Responsible for the Process?

Yura Solovey — Thu, 23 Jul 2026 13:38:18 +0000

In the second part, I wrote about a package that passed formal validation, looked complete, and still turned out to be a weak handoff. That problem was relatively easy to explain: our validators checked whether sections existed, but not whether the next person could actually do the work. The natural response was to add more checks, tighten the schema, strengthen the checklist, and assume that control had been restored. We did exactly that — and almost immediately ran into a worse case, where the problem was no longer the quality of the documents.

Every Step Was Completed

Let’s use the same adapted domain with the PREFERENCE_CHANNEL_CHANGED event. The workflow completed every step in the correct order. Each local validator approved its document. A person confirmed the key decision, and the final package contained every required artifact. The original requirement was simple: when profile.notificationChannel changes from email to sms, the system should emit one event and record the old and new values in the audit log. The process had several stages: clarify the requirement, create the technical spec, prepare Jira, generate QA scenarios, obtain approval, and assemble the final handoff. At the orchestration level, everything looked completely correct:

workflow:
  - define_requirement
  - generate_technical_spec
  - generate_jira
  - generate_qa
  - request_approval
  - assemble_handoff

The workflow skipped nothing and preserved the intended order. The technical spec passed its validator, Jira passed its own, and QA passed another. The approval was recorded as well. Then the user made a small correction: the event should no longer be emitted only for the transition email -> sms, but for any change where the new value equals sms. The technical spec was updated, passed validation again, and the process continued from where it had stopped. At the level of individual operations, everything still looked logical: the correction had been applied, the updated document was valid, an approval already existed, and every step had completed. We didn’t discover the mistake during validation. We found it only when the implementer started working from Jira and asked why the trigger in the ticket didn’t match the technical spec. At first, we looked for a defect in Jira, then in the validator, and only after several repeated checks did it become clear that the documents were individually valid but belonged to different versions of the decision. Untangling that mismatch took longer than making the correction itself. Jira and QA were still based on the old rule, while the approval referred to the previous version of the technical spec.

The local technical-spec validator saw only the current document:

technical_spec_validator:
  required:
    - event_name
    - trigger_condition
    - audit_fields
  result: pass

The Jira validator saw that the ticket contained acceptance criteria, implementation notes, and test expectations:

jira_validator:
  required_sections:
    - acceptance_criteria
    - implementation_notes
    - test_expectations
  result: pass

The QA validator had no obvious reason to reject its artifact either:

qa_validator:
  every_case_has:
    - setup
    - action
    - verification
    - cleanup
  result: pass

Each of these results was locally correct. The technical spec really was valid. Jira really did contain the required sections. The QA scenarios really were self-contained. The problem wasn’t that one of the validators had “done its job badly.” The problem was that none of them answered a different question: did all of these artifacts belong to the same version of the decision?

Where the State Disappeared

After the correction, the global picture should have looked roughly like this:

process_state:
  canonical_requirement:
    version: 4
    trigger_condition: "new_value == sms"

  approvals:
    technical_spec:
      approved_version: 3
      status: stale

  artifacts:
    technical_spec:
      based_on_version: 4
      status: current
    jira:
      based_on_version: 3
      status: stale
    qa:
      based_on_version: 3
      status: stale

  completion_allowed: false

But in practice, the process stored only separate successful outcomes: technical spec valid, Jira valid, QA valid, approval present. It had no single place that recorded which version of the canonical requirement had produced each artifact, which version the approval applied to, and which dependencies had become invalid after the correction. As a result, the system combined correct local facts into an incorrect global conclusion:

completion_check:
  technical_spec_valid: true
  jira_valid: true
  qa_valid: true
  approval_present: true
  result: complete

There’s no obvious false statement in this check. All four values can be true at the same time. They’re simply insufficient to justify complete because they ignore versions, causal dependencies, and invalidation. That distinction matters. Sometimes an AI process ends incorrectly not because of a hallucination, not because a step was skipped, and not even because the output was poor. It fails because the system can’t combine locally correct results into a correct process state.

A Workflow Is Not the Same as Process Control

A workflow is good at answering, “What should run next?” It can enforce order, retries, timeouts, branches, and even returns to an earlier step. But a sequence of steps does not, by itself, prove that every dependent object remains current after a change. In our example, the workflow correctly ran generate_jira, then generate_qa, then request_approval. After the correction, it just as correctly ran generate_technical_spec again. If its model contains no rule saying, “A change to the canonical requirement invalidates Jira, QA, and the approval,” then it has no reason to rerun them. From the workflow’s perspective, that isn’t a failure. It’s a missing rule.

A minimal orchestration fragment might look like this:

on_correction:
  rerun:
    - generate_technical_spec

But the real process needs something stronger:

on_correction:
  invalidate:
    - technical_spec_approval
    - jira
    - qa
  rerun:
    - generate_technical_spec
    - generate_jira
    - generate_qa
    - request_approval

The difference between these two blocks is not the quality of the wording, and it is not the “intelligence” of the model. In the second case, the process dependencies are explicit. In the first, they’re left as assumptions. The model may infer them, or it may not. Even if it infers them correctly nine times in a row, the tenth run still does not become deterministic.

A Validator Is Not the Same as Process Control

A validator answers, “Does this object satisfy a specific rule?” That is a strong and necessary mechanism, but its scope is usually local. It can check a schema, business constraints, completeness, consistency within a document, and even compare two artifacts. But if it is not given the current canonical version, previous approvals, and the dependency graph, it cannot determine whether a document is current in the context of the entire process.

For example, this validator checks Jira against the technical spec:

jira_consistency:
  compare:
    - jira.acceptance_criteria
    - technical_spec.trigger_condition
  result: pass

It works only if it receives the current technical spec. If the orchestration layer passes cached version 3, Jira version 3 will successfully validate against technical-spec version 3 even though the canonical requirement is already at version 4. Local consistency is preserved; global correctness is not. That is why “let’s add another validator” does not always solve the problem. First, you have to define which state the validator is supposed to see and why that state is considered current.

An Evaluator Does Not Own the Process Either

An evaluator usually scores the quality of the result: completeness, clarity, alignment with criteria, risks, and sometimes an overall score. That is useful, especially when the rules cannot be expressed fully through a strict schema. But an evaluator works with the package it is given. If the package contains mutually consistent but stale Jira and QA artifacts, the evaluator may score them very highly. It is not required to know that a correction existed and should have invalidated them.

evaluation:
  clarity: 0.92
  completeness: 0.88
  internal_consistency: 0.95
  handoff_quality: 0.90

That score can be honest and still fail to answer the main question: is this the final package for the current decision? Quality evaluation does not replace provenance, version tracking, or process invariants.

What Does “Predictable Behavior” Mean Here?

When people talk about making an AI model deterministic, they sometimes mean getting literally the same response to the same prompt. That is not the strict meaning I’m using here. For generative models, full textual determinism is often both unrealistic and not especially useful: wording may change, arguments may appear in a different order, and the same idea may be expressed in different ways. For a long-running work process, another form of predictability matters more. We do not need identical text, but we do need consistent enforcement of process invariants: after a correction, old approvals become invalid; stale artifacts never enter the final package; missing mandatory facts stop the process; every final artifact has known provenance; and completion is allowed only when all dependencies are current.

process_invariants:
  - correction_invalidates_dependent_artifacts
  - approval_applies_to_exact_version
  - stale_artifacts_block_completion
  - mandatory_missing_data_causes_hard_stop
  - every_final_artifact_has_provenance

A system like this does not make the model deterministic in the mathematical sense. It makes the boundaries of acceptable behavior predictable. The model can still propose different wording, choose another document structure, or ask an additional question. But it should not be able to complete the process with a stale approval or an unnoticed artifact version.

Who Needs to See the Whole Picture?

In practice, you need a layer that owns not one document and not one step, but the state of the process as a whole. That layer may be called a process engine, state machine, execution controller, protocol runtime, or simply an explicit control layer. The name matters less than the responsibility. It needs to see:

canonical values and their versions;
the provenance of every artifact;
the dependency graph;
the validity of approvals;
allowed transitions;
invalidation rules;
hard stops;
completion conditions.

The workflow can use this state, validators can read it, the evaluator can score the package in its context, and the model can handle individual judgment-heavy tasks. But the conclusion “the process is truly complete” should not appear as a side effect of every local component independently returning pass.

This does not mean every AI process needs to become a complex engine. For a short creative task, that level of control would be excessive. But when a process is long, contains branches, approvals, corrections, several dependent documents, and a requirement for repeatability, global state already exists whether or not we describe it. If we fail to make it explicit, it remains scattered across the prompt, chat history, user memory, and the model’s assumptions.

A Practical Check

Before calling an AI process controlled, I would check a few things:

Does every artifact record the version of the source it was created from?
Does a correction invalidate all dependent documents and approvals?
Can a validator determine that it is checking the current version?
Is there an explicit distinction between valid, current, and approved?
Does a stale artifact block final completion?
Is provenance preserved after retries and backtracking?
Is completion a separate rule rather than merely the fact that the last step finished?
Can one layer explain why the process is in its current state?

If those questions do not have clear answers, the process may be well orchestrated, well validated, and even produce high-quality output, but that still does not mean it is controlled.

If no layer can see the full state of the process, are we controlling the behavior of the AI model — or merely hoping that its locally correct decisions happen to add up to the right result?

You Validated the Output. Who Validated the Process?

Yura Solovey — Wed, 22 Jul 2026 16:13:34 +0000

In the first part, I wrote about the point where a long instruction stops being just a prompt and starts behaving like an undocumented program. It develops state, transitions, recovery paths, dependencies between documents, and rules that are already difficult to change locally. But even if you accept that, an uncomfortable question remains: how do you know the model actually followed the process you needed? The easiest thing is to look at the result. There’s a Jira ticket, a technical description, acceptance criteria, and several test scenarios. The format is correct, the required sections are present, and the names match. During a quick review, the package looks complete — and that’s exactly where we went wrong more than once. A result that looks correct doesn’t prove that the model used the right sources, performed every required check, and avoided skipping a mandatory step. It also doesn’t prove that the next person won’t have to reconstruct part of the work on their own. The clearest way to see this is through a concrete handoff.

The Package Looked Ready

The example below is adapted from a controlled run. I changed the domain, names, fields, and values completely, but kept the failure mechanism intact. Imagine a team adding an internal event called PREFERENCE_CHANNEL_CHANGED, which should be created when profile.notificationChannel changes from email to sms. At the data level, the requirement might look roughly like this:

event:
  name: PREFERENCE_CHANNEL_CHANGED
  trigger:
    field: profile.notificationChannel
    from: email
    to: sms
  expected:
    emit_once: true
    persist_audit_record: true

The model follows a long process and produces a Jira ticket, a technical description, acceptance criteria, manual QA scenarios, a short rollback, and a requirement to update tests and documentation. At first glance, everything looks fine: there’s a positive scenario, a negative case, a duplicate check, and a description of the expected event. The names are consistent, the document structure is correct, and the process ends successfully. We’d accepted the package with almost no comments at first. Only when the tester started reading it as a working instruction rather than a neatly assembled document did they stop at the second scenario and ask, “What data am I supposed to prepare here, and what do I clean up afterwards?” The package didn’t answer that, and figuring out what the model had meant took longer than reading the document itself.

The first gap appeared in Jira. The requirement looked perfectly normal at first, but it was already missing the level of detail needed for implementation. It said something like this:

technical_requirements:
  - update unit tests
  - update documentation

Formally, the requirement exists, but in practice it doesn’t answer several basic questions: which component needs coverage, which tests are mandatory, which fixtures should be used, where the audit record is stored, and which documentation page should be updated. QA had the same problem. The first scenario was detailed:

{
  "case": "channel changes from email to sms",
  "setup": [
    "create user with notificationChannel=email",
    "clear previous audit events"
  ],
  "action": "update notificationChannel to sms",
  "verify": [
    "event PREFERENCE_CHANNEL_CHANGED exists",
    "event was emitted once",
    "audit record contains old and new values"
  ],
  "cleanup": [
    "restore notificationChannel=email",
    "remove generated audit event"
  ]
}

The second scenario already started with “repeat the previous steps, but change the value,” and the third referred back to the first two. The tester had to decide which data was inherited, what needed to be cleaned up between runs, and whether the duplicate check should use the same profile or a new one. There was a rollback too, but only one generic instruction: “return the notification channel to its previous value.” That wasn’t enough for real testing. If a scenario creates an event, changes a profile, and leaves an audit record behind, the rollback needs to explain what to do with each of those effects. Otherwise, the next run already starts from a different state than the one described in setup. The package didn’t look obviously wrong: it covered the right topics, contained every expected section, and even read convincingly. The problem only became visible when someone tried to use it as a handoff. That was when it became clear that the document was simply pushing part of the work onto the next person.

A Valid Format Isn’t the Same as a Ready Handoff

In many AI processes, result checking starts with schema validation. That’s useful: a schema can confirm that required fields exist, value types are correct, the document has the expected structure, and the JSON or YAML can be parsed programmatically. For example, this check really will catch an empty rollback and prevent a required field from being omitted:

{
  "required": ["title", "test_cases", "rollback"],
  "properties": {
    "rollback": {
      "type": "string",
      "minLength": 1
    }
  }
}

But it doesn’t know whether that rollback is sufficient for an independent tester. The string "restore previous value" will pass the check just as easily as a complete instruction that cleans up the profile, events, and audit records. The same is true for test_cases: an array with five elements is formally better than an empty array, but the number of elements says nothing about whether those tests are self-contained. A Jira ticket can also contain acceptance criteria while still failing to define all the technical deliverables. We wrote our first validators at exactly this superficial level: does the section exist, is the field filled in, is there a list of tests? The package passed them honestly because all the required elements were formally present. The problem was in our criteria, not just the model’s behavior. After that, we had to split what we had previously treated as one question into three separate ones:

Does the result have the correct form?
Does it contain enough information for the next person?
Was it produced through the right process?

The answers don’t always match. A document can be formally correct and still be a weak handoff. A handoff can be sufficient even if the model skipped some of its internal checks. The model can follow every step and still produce a badly written document. When all of that is collapsed into one score or one final gate, some classes of failure simply disappear from view.

What Does “the Right Process” Actually Mean?

It’s easy to go too far in the other direction and demand that the model repeat every internal step word for word, but for most tasks that would be unnecessary. I’m not interested in exposing the model’s hidden chain of thought or controlling every internal operation. A practical process is what can be recorded externally: which sources were used, which mandatory checks were completed, which decisions a person approved, what happened after a correction, and why the process ended in that particular final state. If a document changes after approval, the old approval should no longer count as valid. If the model goes back to an earlier step and changes a canonical value, dependent documents should be regenerated or explicitly rechecked. If mandatory data is missing, the process should stop rather than fill the gap with a plausible assumption. You can think of this as a very simple process state. There’s nothing complicated here: a few versions, statuses for dependent artifacts, and an explicit answer to whether completion is allowed:

state:
  canonical_value: sms
  approved_version: 3
  current_version: 4
  dependent_artifacts:
    jira: stale
    qa: stale
    technical_spec: current
  completion_allowed: false

This fragment doesn’t explain how the model “thought.” It only records what matters for execution: the current version, the status of dependent documents, and whether the process may finish. An ordinary prompt often doesn’t make those things explicit. They exist somewhere in the text, the chat history, and previous responses, but not necessarily as controlled state.

One Successful Run Proves Almost Nothing

Another trap is testing a long process only on the happy path. The model receives complete input, the user answers every question, the documents are approved on the first attempt, and the package is created successfully. After a run like that, it’s easy to conclude that the instruction works. In the first part, I already named a minimum set of five scenarios, and each one catches a different class of problem. A normal successful run checks the base route. A branch-heavy scenario shows whether the model gets lost when alternatives and returns appear. An invalid-or-irrelevant-input scenario tests whether it can refuse to build a plausible but incorrect package. A correction scenario shows whether earlier documents and approvals are invalidated. A hard stop checks the most uncomfortable case: whether the model invents a mandatory fact simply because the process “has to” finish. In our case, the happy path produced a polished package, while the scenario in which an independent person had to use the document showed something else: part of the process still had to be reconstructed manually. That’s the difference between “generation completed” and “the process is usable.”

Why the Model Could Finish the Package Successfully

In the example above, the model didn’t ignore the task. It produced every expected artifact and touched on the main topics. If the check had asked only whether there was a Jira ticket, QA, rollback, a mention of unit tests, and a full set of documents, the package would have passed without trouble. In fact, it did pass our first validators. But the criterion rollback exists is much weaker than a real check of rollback quality. For this case, a stronger criterion might look roughly like this:

rollback_quality:
  applies_to_each_mutating_case: true
  restores_initial_state: true
  removes_generated_events: true
  clears_audit_records: true
  executable_by_independent_tester: true

The same applies to QA: having scenarios doesn’t guarantee that each one contains setup, action, verification, and cleanup. The model optimizes for what you describe. If the criterion is defined at the level of section presence, the model can satisfy it at exactly that level. This isn’t unique to AI: a person can also write a formally complete but practically weak document. The difference is scale. A model can generate a large package very quickly, and it can scale incompleteness just as quickly.

What You Can Check Right Now

Before accepting an AI-generated package, I’d suggest asking at least a few questions. They don’t replace a full review, but they quickly reveal whether the same gaps we saw in the handoff are hiding behind a correct structure:

Can the next person do the work without an extra explanation from the author?
Is every test case self-contained?
Are specific technical deliverables defined, rather than just general wishes?
Does every state-changing action have its own rollback or cleanup?
Is it clear which sources support the key values?
Do earlier approvals become invalid after the document changes?
Are dependent artifacts updated after a correction?
Can the process still end successfully when a mandatory fact is missing?
Are you checking not only the format, but whether the handoff is actually usable?

This checklist doesn’t require a new platform or language. It can be added to code review, QA review, or a normal document review. Sometimes that’s already enough to reveal the difference between a document that looks good and one people can actually work from.

Where Should the Control Live?

After cases like this, the natural response is to add more validators. One checks the Jira structure, another checks QA completeness, and a third checks consistency across documents. The workflow controls the order of steps, while a person approves critical decisions. That works to a point, and for a while it can feel as though the problem is solved. Then another question appears: who is responsible for the process as a whole? A workflow knows which step comes next, a validator checks a specific document, an evaluator scores the result, and a person can stop the process. But none of those layers necessarily sees the whole picture on its own: state, sources, approvals, correction paths, final process states, and dependencies between artifacts.

What if the workflow completed every step, the validators approved every document, and the result was still wrong — who was actually in control of the process?

When Does a Prompt Become an Undocumented Program?

Yura Solovey — Mon, 20 Jul 2026 09:47:13 +0000

When we first decided to bring AI into analysts’ work, the task seemed fairly down-to-earth. The company has several development teams and roughly fifteen analysts. They collect requirements, prepare tasks, describe changes in Confluence, think through testing, and help align future implementation with both business and development. A lot of this work is repetitive, so giving analysts a tool that could prepare first drafts felt like an obvious idea.

The problem is that a document in this kind of process almost never stands on its own. The same change exists in Jira, in Confluence, in mockups, in test scenarios, and in technical notes. Sometimes there are also separate instructions for making changes in the codebase. Each artifact has its own purpose, but all of them describe the same future system behavior. If the wording drifts apart, that can go unnoticed for several days, until different people begin working from different versions.

A typical case looked roughly like this (the details and names are changed, but the mechanism is real). Jira said that a status field could have three values: draft, active, and archived. Confluence still contained an older table with only two values, while the test scenario also checked for disabled, which had been discussed early on and later rejected. The developer implemented what was written in Jira. The tester opened the scenario and filed a defect because disabled was missing. Only after that did the analyst compare the documents and realize that each of them preserved a different version of the decision.

Nothing catastrophic happened. We simply had to go through the documentation again, update the tests, clarify the task, explain to the developer that the code did not need to change, and send the package through review one more time. It’s exactly these “nothing serious” moments that add up to delays, when the same task travels two or three times between an analyst, a developer, and a tester. At some point, it becomes difficult even to say where the original mistake was, because everyone is already working on the consequences.

So our original goal was practical: help analysts assemble a consistent package of materials more quickly and reduce the number of such returns. We didn’t begin with a theory about formal languages or AI process architecture. We simply wrote prompts and tried to make them good enough for real work.

This is the first article in a four-part series. The second will look at situations where the final result appears correct even though the process was carried out incorrectly. In the third, I want to examine what workflows, validators, and evaluators actually control, and where a gap remains between them. The last part will review existing approaches. Here, I will focus on how a simple instruction gradually turns into something much more complicated.

At First, the Model Only Helped with Drafts

The first scenarios were small. The model asked clarifying questions, helped structure the information, prepared a Jira draft, suggested text for Confluence, and produced several test scenarios. The analyst reviewed the result, added what was missing, and asked the model to correct inaccuracies.

AI still works well in that format. A short conversation, one or two documents, and a quick human review are enough for many tasks. Building a separate mechanism around every such interaction would simply create unnecessary work.

Technically, these were ordinary chats with sets of files that we passed to the model together with the instructions. Later, we moved such processes into a controlled environment without direct access to production Jira or Confluence.

The complexity grew gradually. Even a relatively small change could involve forty or fifty analytical steps. First we collected the input information, then clarified data structures and parameters, prepared several related drafts, sent them for review, collected comments, made corrections, and checked the documents against one another again. At the end, we had to add real links to the pages and tasks that had been created. Technical operations were happening in parallel: values were copied between files, required fields were checked, dependent artifacts were updated, and validation was run again.

In the end, the model had to follow a long route and prepare around seven artifacts. Each individual result could look acceptable, but that was not enough. We also had to make sure that names, parameters, rules, and constraints did not drift apart across documents after several rounds of edits.

We still called it a prompt because, technically, we were running one large textual instruction. At the same time, it already contained a sequence of actions, dependencies between steps, transition conditions, returns to previous stages, repeated checks, and rules for handling errors. I didn’t see the significance right away. For quite a while, I thought it was simply a very large prompt that needed better organization.

Maintenance Turned Out to Be Harder Than the First Version

It’s almost impossible to write a process like this correctly on the first attempt. Even when the business logic is well understood, there is still the question of how the model will interpret each phrase in a particular context.

We ran the instruction and looked for where it failed. If the model skipped a check, we added a clarification. If it filled a field incorrectly, we added an example. When two documents started contradicting one another, another validation step appeared in the process. The next run exposed a new problem, and the instruction grew by several more paragraphs.

Every change had a specific reason, so for a long time it didn’t feel as though we were doing anything dangerous. The text was simply becoming more detailed. After dozens of iterations, however, it contained both the description of the work and traces of all the failures we had encountered before. In one of the larger experiments, the total instruction set grew to almost one megabyte. It was spread across several files and included many sections, examples, and checks that had accumulated over a long period.

That was when local changes stopped staying local. A rule added for one document could alter behavior in another. A new check could affect how the model interpreted a step that appeared much earlier. After fixing one problem, a new one sometimes appeared in a place that had previously been stable.

Splitting the instructions into files helped people: the material became easier to read and edit. The model, however, did not treat those files as real boundaries. It saw one large shared context, found connections between fragments, and drew conclusions that we hadn’t intended.

What made this especially unpleasant was that the model’s explanations often sounded convincing. It could find another rule, connect it to the current step, and explain why it had chosen a particular behavior. The reasoning was not random. It really did follow from the way the model had read the full text. But for our process, that connection was wrong, and it became harder and harder to notice such interactions in advance.

At some point, we couldn’t confidently predict the consequences of every new clarification. The instruction still worked, but maintaining it began to feel like editing a large system without a dependency map.

And, to be honest, I kept trying for quite a while to solve the problem with the same method that had created it: adding a few more carefully written paragraphs.

We Tried to Explain Everything More Clearly

The most obvious response to incorrect model behavior is to describe the requirement in more detail. Early on, that works: an unclear phrase can be rewritten, a weak example can be replaced, and a missing exception can be added to the rules.

The difficulty begins when a new paragraph has to coexist with dozens of older ones. The author sees it as a local correction. The model receives one more piece of context that can be connected to every other part. This increases both the size of the text and the number of possible ways its parts can be interpreted together.

We regrouped rules, moved examples closer to the steps they belonged to, separated checks, and removed repetition. Some of those changes had a noticeable effect. The main difficulty remained: editing one fragment of a large instruction didn’t guarantee a change in only one part of the model’s behavior.

For a long time, I thought we simply needed a better way to write prompts. Eventually, I saw a different picture. The task already had state, transitions, conditions, dependencies between artifacts, and recovery after errors. We were trying to maintain all of that as ordinary prose, even though in terms of complexity it already resembled a small program.

The Way the Model Worked with Code Changed How I Saw the Problem

By that point, I had been using language models in software development for more than two and a half years. They made plenty of mistakes there too, but the environment was very different.

When a model changes a function in a large project, it sees the signature, types, call sites, surrounding code, tests, and the rules of the language itself. Changing an interface affects concrete implementations and usage points. Type violations can be caught during compilation or static analysis. Even a poor change is attached to a specific structure, so its consequences are easier to find and inspect.

A large textual instruction offers far fewer such anchors. The same requirement can be expressed in dozens of ways, and each phrasing leaves room for interpretation. In a short conversation, that’s usually fine: a person quickly clarifies the intent. In a long process, the decision made at one step becomes context for the next, so a small ambiguity can pass through several stages and surface in a different document.

There is a temptation to draw too broad a conclusion from this, and I’d rather avoid that. Code is not a universally better way to describe every kind of work. Natural language lets us discuss unclear tasks, change direction, explore options, and avoid defining the whole structure in advance. For analytical work, that is one of its main advantages. In our case, the problem appeared where the model was expected to follow a repeatable procedure with a large number of mandatory conditions.

What Happens as Models Become More Capable?

Modern models are increasingly good at solving underspecified tasks, and for analysis or exploration that is a major advantage. A repeatable process is different: the model may see a “better” route where the team needs the agreed one. I’ve already caught myself describing two separate modes of work — “think for yourself” and “follow this literally.” I don’t know yet whether new control mechanisms will reduce that tension.

Where I Draw the Boundary Now

I don’t tie the boundary to a specific number of steps. Ten steps can be harder than one hundred if they contain many branches, returns, and dependencies. A long linear process can be easier to control than a short one in which every decision changes several later actions.

I now test such instructions with a few simple questions:

Does the process need to preserve state between stages?
Do earlier decisions affect later ones?
Are there branches, error recovery paths, or several possible final outcomes?
Do several documents need to change together?
Does a correction make earlier approvals or artifacts obsolete?
Will someone later need to retrace how the result was produced?
Would one skipped step force another person to reconstruct a missing part of the process?

If most answers are yes, I no longer treat the text as an ordinary prompt. It is a process specification, even when it lives in a Markdown file. That doesn’t mean it must be rewritten in a formal language: sometimes it is enough to move part of the control into code, a workflow, validators, or human review. But testing it with only one successful run is certainly not enough. The minimum we now check is a normal scenario, a branch-heavy scenario, invalid input, recovery after correction, and a hard stop when mandatory facts are missing.

For now, the practical observation I want to preserve is simple: a large text containing transitions, state, exceptions, and dependencies between artifacts becomes difficult to maintain in the same way as an ordinary instruction.

Have you had a moment when a prompt stopped feeling like plain text? What caused it: the number of steps, branching, the number of documents, the cost of an error, or something else? And did you manage to keep such a process under control without additional formalization?

In the next part, I will move to another aspect of the same problem: a final document can look correct and pass formal validation even though the model didn’t follow the process that was expected of it. Later in the series, I will show which control mechanisms we tried, where they helped, and where they merely moved the problem to another layer.