DEV Community: Maksym Kuzmitskyi (MaximusFT)

Read-Only Is Not the Same as Safe

Maksym Kuzmitskyi (MaximusFT) — Fri, 06 Nov 2026 00:00:00 +0000

An agent cannot edit the repository. It can still read a log containing customer information and include it in a remote request or a published explanation, if its tools and data path allow that.

Nothing in that hypothetical example requires a file write.

I like read-only modes. They are useful restrictions, especially for review and investigation. I don't like treating the label as a complete security argument.

Integrity and confidentiality are different properties. Preventing a change to local files doesn't, by itself, establish who can see their contents or where those contents can travel.

Start with the data path, not the mode name

For an agent-assisted task, I would identify what can be read, which process or service receives it, and where outputs can appear.

That includes the model provider, local and remote tools, logs, review comments and any connected workspace. A tool named search may run locally or call a remote index. A browser may already be authenticated to services the task doesn't need.

The answer depends on the actual product, integration, account and organization policy. I wouldn't infer retention, training use, residency or access controls from the word enterprise or from a familiar brand name.

GitHub's current application card distinguishes data flows across its agent surfaces and explicitly discusses responsibility for a bring-your-own-provider configuration. That is a reason to inspect the selected path, not to assume every Copilot session sends data through the same route.

Read-only should be one property in that description. It shouldn't replace the description.

Source code isn't the only sensitive input

A repository can contain private fixtures, copied production responses or internal endpoint information. Development machines can also hold browser profiles, credential files and local service configuration outside the checkout.

I would avoid collecting all of that merely because a tool is capable of reading it. The task should establish a need for particular information, and the environment should enforce meaningful limits where possible.

The right question isn't whether the agent is generally trustworthy. It is whether this task requires this class of data under the approved handling rules.

For many investigations, a sanitized schema or a small synthetic fixture can answer the technical question. If real data is necessary, its use needs the appropriate authorization and controls. Replacing names in one field doesn't automatically anonymize an entire payload.

I would not ask someone to paste secrets into a chat so the agent can diagnose an authentication problem. A person can configure a credential through the intended secure mechanism without turning its value into task content.

File access and network access are separate boundaries

A process may have restricted writes and broad reads. It may also have an outbound network path. Those properties need to be examined together.

The VS Code security guide, checked on October 23, 2026, explicitly separates sandbox behavior, permission checks and network configuration. It notes that outbound network access can remain enabled under its documented defaults and that tool paths and harnesses differ.

I am deliberately not presenting a generic setting that supposedly secures every session. A shell command, a built-in file tool and a remote MCP tool can pass through different controls.

Restricting a local process doesn't necessarily restrict a connected remote service. Blocking an arbitrary external host doesn't prevent disclosure to an allowed service that should not receive a particular payload.

Likewise, an allowed destination isn't permission to send every accessible file there. Access to a provider for legitimate inference still requires appropriate selection of the context supplied.

A permitted route and a permitted payload are different decisions.

This is where a read-only label becomes particularly incomplete. It says little about either one unless the product defines those properties explicitly.

The agent can follow the task and still expose too much

Prompt injection is one route to an undesirable action, but it is not required for every disclosure.

An agent honestly trying to explain a failing test may quote an entire response body when a status code and a sanitized field name would suffice. A reviewer may include a sensitive configuration value in an otherwise useful comment.

I would want output handling to consider necessity and destination, not only whether the conclusion is correct.

This also matters for task records. Preserving evidence does not require keeping raw credentials, complete private payloads or every retrieved document. Logs that improve observability can create another data store with its own access and retention obligations.

A link or bounded summary may be more appropriate than a copy. Sometimes neither is appropriate in the destination where the agent is about to write.

Read-only access to the source doesn't prevent a different tool from publishing the resulting text. The review of that publication path needs to consider what it contains.

Approval needs enough information to be meaningful

A request for permission to call a tool should identify the operation and relevant destination without displaying secrets in the prompt itself.

A person approving a broad capability for convenience may not be approving every later use of sensitive content. Long-lived approvals, inherited authentication and ambiguous tool descriptions can widen the practical boundary beyond the user's immediate intention.

I would prefer task-scoped access with an understandable lifetime. Where a product supports separate permissions for reading, processing and publishing, those should reflect the task rather than share one blanket decision by default.

That doesn't mean asking a person about every ordinary file. Approval fatigue can make a nominal checkpoint less useful. Enforced restrictions and clearly defined safe paths can reduce unnecessary prompts while preserving meaningful decisions.

The existing guardrails article concerns control over agent behavior. Here the question is narrower: which data may cross which boundary, even during behavior that appears routine.

Test the boundary without using real secrets

For an authorized evaluation, I would use disposable files containing unique, harmless markers rather than actual credentials or customer records. Then check which reads, requests, logs and outputs contain those markers.

The setup should use a controlled environment and destinations approved for the test. The purpose is to observe the data path, not to try bypassing organization controls.

A successful fixture establishes something about the observed tools and configuration. It doesn't prove every model decision will minimize data or that every future integration inherits the same restrictions.

Denied operations deserve testing too. A tool should not silently retry through a more privileged route simply because its first attempt was blocked. An unavailable source should remain unavailable in the report, not become a reason to weaken the boundary.

I would also test what survives after the session: retained outputs, cached tool responses and published artifacts. Stopping the agent is not necessarily the same as removing information already sent or stored.

Keep read-only useful by describing what it protects

I don't want to discard a useful mode because it doesn't solve every problem. Preventing mutation can substantially reduce the risk of an investigation or review task.

I want the claim to remain specific: which writes are prevented, which reads remain possible, what network paths exist and which publication actions are available.

Once those are explicit, the team can decide whether the environment is appropriate for the data involved. Without them, safe becomes a feeling attached to a mode selector.

A read-only agent may be exactly the right tool. I would still ask what it can learn, who receives that knowledge and what remains after the task finishes.

When you approve read-only agent access, which data boundary do you verify separately? I'd be interested in the practical checks that made the mode's actual guarantees clear.

You Didn't Write It. You Still Have to Maintain It.

Maksym Kuzmitskyi (MaximusFT) — Wed, 04 Nov 2026 00:00:00 +0000

The feature works. The tests pass. The agent can explain every file it created.

Now ask another engineer to change one business rule without the original conversation.

That is the test I care about after the demonstration. Not whether a human typed the code, but whether the resulting system can be understood and changed by the people responsible for it.

Generated code doesn't need a special moral category. It needs an ordinary maintenance owner. The prompt isn't that owner, and the model being able to regenerate an explanation isn't a substitute for one.

Understanding is not remembering every line

I wouldn't require an engineer to memorize generated code before accepting it. We don't apply that standard to dependencies or to every change written by a colleague.

I would expect someone to understand the important boundaries: where state is authoritative, where input becomes trusted, which contracts are shared, how failures surface and what a future change is likely to affect.

GitHub's current application card explicitly calls for reviewing and testing generated code. I see that as an acceptance floor, not a complete account of whether another engineer can maintain the result.

That understanding can come from code structure, tests, documentation and exploration. It doesn't require manual authorship.

The concern begins when the implementation can only be explained through the agent that produced it. A maintainer asks why a cache exists, which consumer requires an adapter, or whether a retry can repeat a write, and the answer is to start another conversation hoping the model reconstructs the original decision.

The model may produce a persuasive explanation. Without supporting evidence, it may also produce a new justification for an old artifact.

Local plausibility can accumulate into a confusing system

An agent can reasonably add a helper for one task, another wrapper for the next, and an extra configuration layer to keep the third change small.

Each PR can look tidy on its own. The combined codebase can acquire several ways to perform the same operation, with no clear reason to choose among them.

This is a hypothetical pattern, not a report of a particular employer's repository. It is also familiar without AI. More convenient generation can make local additions cheaper while leaving their long-term coordination cost intact.

OpenAI's 2026 harness-engineering report describes agents reproducing existing patterns, including uneven ones, and discusses mechanical constraints and cleanup in its own experiment. It also explicitly leaves long-term architectural coherence as an open question.

I take that as a reason to inspect accumulating structure, not evidence that every generated system inevitably decays or that a recurring cleanup agent solves maintenance universally.

The important question is what the next change will copy. A convenient local pattern can become an accidental standard simply because it is visible.

A useful abstraction should make the next decision easier

I am not against introducing an abstraction with AI assistance. I want its existence to reduce complexity that actually matters.

For an illustrative payment flow, an adapter separating provider responses from application state can be a clear boundary. Several generic layers that forward the same parameters may only make the behavior harder to find.

The review question isn't whether the abstraction looks sophisticated. It is which variation, dependency or invariant it isolates, and where that value can be observed.

I would ask for a concrete next change. Add a provider error, change a validation rule or remove a consumer. Which files move, and which parts should remain untouched?

This doesn't predict every future requirement. It tests whether the explanation of the current design connects to an ordinary modification rather than a diagram alone.

The same standard applies to hand-written code. AI doesn't make architectural taste objective, but it makes it particularly easy to generate more structure before deciding whether the structure earns its place.

Preserve reasons that aren't visible in the code

A good name explains what a function does. It may not explain why an older implementation still exists or why a seemingly simpler migration was deliberately postponed.

Those decisions deserve a small durable record when they affect future work. Link the relevant requirement or design decision, identify the remaining constraint, and keep the record close enough to be discovered.

I don't want a transcript of every rejected alternative. I want the few facts a future maintainer would otherwise have to guess.

That is consistent with keeping repository context useful: context is an engineering artifact with ownership and freshness, not an unlimited pile of prose.

An agent-written explanation still needs review. It can describe the intended design while the implementation differs, or turn an inference into a historical claim. Documentation should point to evidence and make uncertain reasons visible.

When a constraint changes, update or remove the obsolete explanation. A permanent paragraph defending a temporary workaround can become part of the problem.

Tests should explain behavior, not freeze incidental shape

Tests are part of maintenance because they tell the next engineer what must remain true.

A suite dominated by snapshots of internal structure can make a straightforward refactor feel dangerous without explaining the product invariant. A suite of overly broad mocks can give the opposite impression: every change looks safe because the meaningful interaction is replaced.

Neither snapshots nor mocks are inherently wrong. Their value depends on the question they preserve.

I would prefer a future maintainer to see what a denied operation must do, which retry is permitted, or how a state transition handles interruption. Those are constraints that help a change stay local.

The existing React testing article discusses choosing checks by what they establish. The maintenance angle is what those checks teach someone who didn't participate in the implementation.

A test that merely agrees with generated code can preserve the wrong assumption. A well-founded test can make that assumption challengeable without requiring the original prompt.

Try a handoff change before accepting a large structure

For a consequential generated subsystem, I would give another engineer a small realistic modification. Ask them to identify the owning behavior, explain the relevant invariant and make the change using the available repository evidence.

This is a proposed acceptance exercise, not a measured experiment from my work. It shouldn't become compulsory theater for a two-line fix.

The useful observations are where the person has to guess, which explanation doesn't match the code, and whether an unrelated change becomes necessary just to touch the intended behavior.

Another agent can attempt the exercise too, but its success doesn't automatically establish human maintainability. It may tolerate a structure that a team cannot reasonably review or operate.

I also wouldn't require every handoff to produce exactly the same implementation. Different valid solutions can preserve the same contract. The goal is intelligibility and controlled change, not reproducing the author's path.

Ownership includes deciding not to regenerate

When a generated component becomes confusing, asking for a complete rewrite can be tempting. It can also replace one poorly understood system with another and make the previous review evidence obsolete.

I would first identify the concrete maintenance problem: unclear ownership, duplicated behavior, an unnecessary dependency or a test that prevents safe change. Then repair that boundary with proportionate scope.

Sometimes replacement is justified. It still needs a migration and verification plan, not just a cleaner-looking output.

I want AI assistance to reduce the cost of working on a system, not merely the cost of producing its first version. That means judging generated code by the work it leaves behind as well as the work it completes.

The next engineer shouldn't need the same model, the same prompt and the same conversation to make a safe change. They should have a codebase that explains enough of itself.

What would you ask someone else to change before accepting a substantial AI-generated implementation? I'd be interested in the smallest handoff exercise that reveals whether the design is genuinely maintainable.

The Agent Knows the Library. Just Not Your Version.

Maksym Kuzmitskyi (MaximusFT) — Mon, 02 Nov 2026 00:00:00 +0000

The suggested API is real. The documentation example is valid. The project still cannot use it.

I find that more interesting than a completely invented method name. A believable solution can belong to a different version, package entry point, runtime or feature mode. It survives a quick recognition check because an engineer has seen something similar before.

The agent may know the library. The task requires it to know the library as installed and configured here.

Familiarity is not a compatibility check

A package name is a broad label. Several major versions can coexist in active projects, and a migration guide can describe a destination rather than the current state.

Even a matching version number may not settle the question. A package can expose different browser and server entry points. A feature may require configuration. A workspace package may resolve to a local artifact rather than the registry version the agent searched for.

This isn't a reason to distrust documentation. It is a reason to identify which documentation applies.

I would first establish the relevant installed package, its resolution from the consuming application and the configuration that changes its behavior. Only then would I decide whether the proposed example is evidence or inspiration.

Inspect what the application resolves

For a Node-based project, an inspection script placed next to the application's package.json can make the question concrete:

import { createRequire } from 'node:module';

const requireFromApp = createRequire(new URL('./package.json', import.meta.url));
const packagePath = requireFromApp.resolve('next/package.json');
const installedPackage = requireFromApp('next/package.json');

console.log(JSON.stringify({ packagePath, version: installedPackage.version }, null, 2));

This example checks Next.js metadata through Node's resolution from that application location. It doesn't inspect every conditional runtime export. Some packages don't expose package.json through their exports, so the same inspection method isn't universal.

The important part is the starting point. Resolving a package from the monorepo root may answer a different question from resolving it inside the application that imports it.

For a browser bundle, I would also inspect the relevant bundler configuration or output when aliasing and conditional resolution matter. A convenient Node probe isn't proof of what every runtime loads.

A manifest range describes intent. An actual resolution tells us more about the artifact being examined.

Current documentation can describe more than one model

Next.js provides a useful present example without needing a claim that its framework is unusually confusing. Its caching guide for projects without Cache Components, checked on October 23, 2026, explicitly identifies itself as the previous model and distinguishes the Cache Components path introduced in version 16.

That means searching for caching guidance isn't enough. The agent needs to establish the project's version and selected mode before transferring advice.

I am not upgrading this site or recommending one caching mode in this article. The point is that two official explanations can be correct while applying to different configurations.

The same pattern appears with router modes, compiler settings and server/client boundaries. Advice from a new guide may require a migration that is outside the current task.

An agent should not quietly perform that migration merely to make its preferred example fit. Discovering that a solution requires an upgrade is useful output. It is not automatic authorization to upgrade.

Types narrow the question, but don't finish it

A compiler rejecting an unfamiliar option is valuable feedback. Removing the error with a cast doesn't establish that the runtime supports the option.

I would inspect the installed declaration and the exported implementation around the disputed boundary. A type can be stale, overly permissive or describe an entry point different from the one actually used.

Likewise, a compiler accepting an option doesn't prove its intended behavioral effect. Some configuration paths tolerate extra input; some features are available only under particular runtime conditions.

The next check should discriminate between those possibilities. That might be a narrow runtime probe, a version-matched example or a test through the actual deployment adapter. Re-running the whole suite without exercising the disputed behavior may add time without resolving the uncertainty.

This is one place where I would rather have an agent say the evidence is incomplete than confidently widen a type until the build becomes green.

Upgrade instructions are not current project instructions

A migration document often says to replace one function with another or remove an old configuration field. Those instructions are conditional on performing the migration.

If the task is a bug fix on the existing release, copying the destination's code can create an accidental mixed-version implementation. It may even look cleaner than the valid old API.

I would keep the task's compatibility target explicit. Are we fixing behavior on the current version, preparing a migration, or completing an approved upgrade? Each has a different acceptable diff.

Related packages need attention too. A new plugin, shared component package or framework adapter may have peer requirements that the application doesn't satisfy. Satisfying one import doesn't establish a supported dependency set.

The versioned npm ci documentation describes frozen installation and configuration requirements for that CLI line. It helps explain why a manifest edit, lockfile resolution and installation can diverge. It is not permission to apply npm 11-specific behavior to another package manager without checking.

Keep retrieval attached to the compatibility question

I would search with the package version, relevant mode and disputed symbol rather than ask broadly how to use the library. Then I would compare the retrieved claim with the installed artifact.

If an MCP documentation tool exposes version selection, use it deliberately. If it returns only current documentation, retain that limitation. A successful tool call doesn't establish that the returned guide matches the consuming application.

Pinned documentation can age too. A project may have upgraded while its instructions still point to an earlier guide. Repository memory is useful for locating the intended source, not for overriding current package evidence.

The instructions-upkeep article concerns rules that stop describing the project. Dependency advice can fail in the same quiet way: the sentence remains plausible while its compatibility conditions have changed.

A useful answer includes the boundary

When an agent recommends an API, I want to know which package, version, entry point and configuration support the recommendation. The detail should match the risk; a familiar stable helper doesn't require a research report.

When compatibility is uncertain, a small verified alternative may be better than pulling the whole project toward the newest documentation. When an upgrade is genuinely necessary, make it a deliberate proposal with its own validation.

I don't want an agent that knows fewer libraries. I want one that can stop recognizing and start checking when recognition is no longer enough.

Which version-dependent mistakes are hardest to catch in your AI-assisted work: APIs from another release, features behind configuration, or packages resolved from the wrong workspace? I'd be interested in the smallest check that reliably settles the question for you.

"Works on My Machine" Now Includes the Agent's Machine

Maksym Kuzmitskyi (MaximusFT) — Sat, 31 Oct 2026 00:00:00 +0000

The agent ran the test. The result was green. Another engineer checks out the branch and cannot get the application to start.

That hypothetical handoff doesn't require the model to have lied. It may have accurately described a result from an environment the next person does not have.

I think the useful question is what made that result possible: runtime, installed artifacts, configuration, test data, processes and state. A commit identifies code. It doesn't identify all of those inputs.

AI-assisted development doesn't invent this old problem. It adds more environments that can appear, validate work and disappear before a teammate sees them.

A worktree isn't a complete environment

A separate Git worktree is useful for isolating source changes. It doesn't automatically isolate ports, databases, home-directory configuration, credentials or services shared by several tasks.

Two branches can be separate on disk while their tests talk to the same local service. One session can migrate a database the other expected to remain unchanged. A process started earlier can serve a different revision from the files currently being inspected.

The agent may see a healthy page without establishing which server produced it.

I would identify the actual application instance, its revision and its relevant configuration before using an interaction as evidence. That can be a build identifier, an explicit process path or another practical observation. It need not become a ceremony after every refresh.

The current GitHub application card describes ephemeral environments for its cloud agent and distinguishes them from local execution. Those are meaningful differences in lifecycle and access. They are not a promise that every tool called an agent runs in the same reproducible container.

The lockfile is necessary evidence, not the whole story

A manifest expresses requested dependencies. The lockfile records a more specific resolution. The installation on disk is the artifact the process actually loads.

Those can disagree because of a stale installation, workspace linking, local tarballs or generated output. An agent can make a correct diagnosis of the code it loaded while the committed project still produces something else.

The versioned npm 11 documentation for npm ci makes its frozen-install behavior explicit: it requires a lockfile, rejects manifest/lockfile disagreement and doesn't rewrite those inputs. It also says tree-shaping flags used when creating the lockfile must be supplied consistently.

I wouldn't present that as a universal command for every package manager or npm release. Use the repository's supported install path and actual tool version. The point is to make dependency reconstruction deliberate rather than inherit a convenient node_modules directory.

A clean installation still depends on artifact availability, platform support and permitted lifecycle scripts. A lockfile cannot provide an unavailable private package or a native library missing from the host.

Reproducible dependency selection is one boundary. Reproducible execution is larger.

Configuration needs a contract without exposing its values

An agent's environment might contain an environment variable that makes a service reachable or silently selects a mock. A fresh checkout may not.

I would document required configuration names, their purpose, safe defaults and how an authorized person obtains them. I would not copy secret values into a PR, task record or agent transcript to make the handoff convenient.

Missing configuration should fail visibly where appropriate. A fallback to a mock can be useful for local tests, but it must remain distinguishable from a real integration check. Otherwise a working simulation can be reported as evidence about a system it never contacted.

Build-time and runtime settings also differ. A browser bundle may already contain a selected public setting even after the environment changes. Checking the shell's current value doesn't establish what the built application uses.

This is why I want configuration provenance, not a dump of configuration. The next engineer needs to recreate the intended conditions without inheriting sensitive information they don't need.

Test state can be the hidden dependency

Consider an illustrative account-management test that passes only because a user record was created manually in an earlier session. The fixture isn't in the repository. The setup step isn't in the task notes.

The agent's test result can be real and the validation can still be nonportable.

I would prefer explicit fixtures with controlled ownership and cleanup. If a shared environment is unavoidable, record the relevant preconditions and limits. Don't reset shared state merely because it would make the current task easier.

Time, locale, timezone and network responses can also influence results. Not every test needs a simulated clock or a hermetic network. But a result used to justify a behavioral claim should identify the inputs that decide it.

The same caution applies to retries. A flaky success after several attempts isn't equivalent to a stable result, and an agent shouldn't erase the failed attempts from the explanation.

Caches deserve similar treatment. A warm cache can make a path work that a first-time user cannot complete. A stale build cache can preserve implementation bytes the source no longer describes.

Record a reconstruction path, not an entire machine

For a meaningful handoff, I would keep a small environment record alongside the evidence: relevant revision, runtime and package-manager versions, install procedure, configuration contract, fixture setup, and the exact check performed.

That is a proposal for useful evidence, not a requirement to archive every process and environment variable. Excessive collection creates privacy and maintenance problems of its own.

The record should also say what was not reconstructed. An unavailable external provider, a mocked payment step or a local-only service is a limitation, not an embarrassing detail to hide.

I would check the reconstruction in a fresh authorized environment when the risk justifies it. The aim is to remove accidental dependencies, not to claim that a clean checkout guarantees production behavior.

A container image can package more of the environment. It still needs controlled inputs, external contracts and an explanation of which mounted files or services sit outside it.

The evidence should survive the session

My green-checks article asks what a check actually established. Environment provenance makes that question answerable after the agent's session ends.

An ephemeral runner is useful precisely because it doesn't preserve incidental state forever. That also means useful setup and result evidence must live somewhere appropriate before the runner disappears.

There is a balance here. I don't need a permanent recording of every terminal screen. I need enough non-sensitive information to understand and repeat the decisive check.

If the only reconstruction method is asking the original agent to remember what it did, the environment hasn't really been handed off.

Leave the next engineer a smaller mystery

I want agents to validate their changes. I also want validation to leave something more durable than the sentence that it worked.

The handoff should help another engineer distinguish a code defect from a missing setup step, a dependency-resolution difference from a changed requirement, and a mocked success from a real integration result.

That is ordinary engineering discipline, made more important by the number of disposable workspaces we can now create.

When an agent finishes a task in your environment, which part is hardest to reproduce afterward: the dependencies, the service state, or the exact conditions of its checks? I'd be curious what you preserve and what you deliberately avoid collecting.

Upgrading the Model Means Retesting the Workflow

Maksym Kuzmitskyi (MaximusFT) — Thu, 29 Oct 2026 00:00:00 +0000

Changing a model identifier can be a one-line edit. It is not necessarily a one-line change to the system's behavior.

The same task may now produce different searches, tool arguments, clarification questions, stopping decisions or recovery attempts. The final code can improve while one of those surrounding behaviors becomes less suitable for the workflow.

I would treat a model upgrade as a change to an engineering dependency, not as permission to carry every previous assumption forward.

That isn't an argument for staying on an old model. It is a reason to make the benefit and the compatibility work observable.

The application contract is larger than the final answer

A coding agent operates inside a harness: prompts, tool definitions, permissions, context management, execution loops and output handling. The model is one consequential part of that arrangement.

The contract may require the agent to inspect the current base before editing, use a particular tool for a sensitive action, stop when authorization is missing, and distinguish an unavailable check from a successful one.

Those requirements don't disappear because the candidate model produces a better-looking patch.

Some differences are API compatibility issues. Others are valid behavior changes inside an otherwise compatible API. A request succeeding doesn't establish that the application still gets the kind of progress events or tool-use pattern it expects.

There is current, concrete evidence for taking that distinction seriously. Anthropic's Fable 5.1 migration guide, checked on October 23, 2026, documents model-specific changes involving forced tool choice, conversation-history handling and agent-loop behavior. Those details apply to the documented migration, not to every model or managed product.

The general lesson isn't to memorize today's parameter list. It is to read the migration contract for the exact integration being changed. A managed agent product may handle details that a custom API harness must manage itself.

First name what is actually changing

An editor update can change the model, tool descriptions and execution behavior together. A provider can update a model alias. A team can revise its system prompt while testing a new model.

If all of those happen at once, an improved result does not identify which change helped. A regression doesn't identify which component broke either.

For a controlled comparison, I would record the model identity available to the integration, harness revision, prompt revision, tool schemas, relevant settings and fixture state. If the exact underlying model is hidden, I would record that limitation instead of inventing a stable identity.

Keep task inputs and starting repositories comparable. A new base commit, warmed cache or already-fixed fixture can turn a model comparison into a comparison of different problems.

One variable at a time is a useful diagnostic ideal, not always a deployment option. When a product bundles changes, evaluate the bundle and describe the attribution limit honestly.

Build cases around decisions that matter

I would start with a small regression set drawn from the workflow's consequential boundaries.

A clear local fix should remain clear and local. An ambiguous requirement should produce a useful question rather than an invented product decision. A failed integration check should not be summarized as a successful task because the unit suite passed.

Another fixture might change the PR head during review. The expected outcome is to notice that the reviewed artifact has changed, not merely attach the old conclusion somewhere the API accepts it.

These are proposed evaluation cases, not reported experiments or results from my work. Their value comes from defining the acceptable outcome before reading the candidate's explanation.

I would avoid requiring one exact tool-call sequence unless the sequence itself enforces a real constraint. Two valid investigations may read different files. The evaluation should distinguish a different route from a violation of the task's boundaries.

Conversely, a plausible final patch shouldn't erase an unauthorized intermediate action. Evaluate consequential actions and the final artifact, not just the last paragraph.

Grade observable behavior, with room for uncertainty

Some outcomes can be checked mechanically: whether a forbidden mutation occurred, whether the final diff contains unrelated files, or whether a claimed test command actually completed.

Other outcomes need a domain reviewer: whether a clarification was necessary, whether a suggested integration belongs in scope, or whether the proposed test establishes the intended behavior.

A model-based grader can help organize evidence, but it isn't automatically an independent authority. I would calibrate it against reviewed examples and inspect disagreements. A score without an understandable basis is another opaque output.

OpenAI's evaluation documentation describes datasets, annotations and graders. At the October 23 check, it also carries a deprecation notice for the hosted Evals platform. I am not recommending a new dependency on that retiring service. Representative cases and explicit grading criteria remain useful regardless of which supported runner stores them.

The tooling should serve the comparison. It should not become a reason to keep a fragile evaluation tied to an obsolete interface.

Recovery deserves its own test

A clean success path is a friendly demonstration. The more revealing cases often involve unavailable tools, authentication failures, partial edits and interrupted work.

I would check whether the candidate preserves the task boundary after an error. Does it ask for a secret in an inappropriate channel? Does it retry an irreversible operation without checking the previous outcome? Does it weaken a test merely to complete the loop?

These concerns aren't claims that a particular model does those things. They are reasons to observe recovery rather than assume it inherits the quality of the model's successful answers.

Permissions should remain enforced outside the model. An evaluation can detect undesirable behavior; it shouldn't be the only thing preventing a consequential action.

The same reasoning applies to compaction and handoffs. A workflow that works for five turns may fail when a summary loses an important unresolved condition. A longer-running fixture can expose that boundary without requiring an enormous benchmark.

Compare the cost of an accepted result

Tokens and response latency are useful measurements. They don't describe the whole task if the candidate requires more correction, publishes noisier reviews or leaves harder-to-understand changes.

I would record human interventions and rework alongside execution cost, using actual observations rather than estimated productivity claims. Passive waiting and a person's active effort are different quantities.

Repeated runs matter where outputs vary. One candidate success and one baseline failure are anecdotes, not a stable rate. The amount of repetition should match the consequence of the decision and the available resources.

The result can be mixed: better implementation on one class of task, more clarification on another, a compatibility repair needed in the harness. That is a useful migration finding, not a disappointing failure to produce a winner.

Keep a way back

Before changing a default, I would keep the previous configuration identifiable and limit initial exposure. That might mean a small authorized task class or shadow review without automatic publication.

Rollback has limits too. A pinned model may later be retired, and a managed product may not expose an exact old configuration. Record what can actually be restored rather than promising a permanent switch.

The existing evaluation article asks whether the workflow produces useful outcomes. A model upgrade is a concrete occasion to ask that question again, with the old assumptions made visible.

I want to use better models. I also want to know which part of my working agreement with the agent changed when I selected one.

What would make a model upgrade fail acceptance in your workflow even if its final code looked better? I'd like to hear which behavioral regressions your evaluations can actually detect.

When Repository Content Starts Giving Orders

Maksym Kuzmitskyi (MaximusFT) — Tue, 27 Oct 2026 00:00:00 +0000

A source comment explains a strange compatibility rule. A README explains how to run the project. A test fixture contains a conversation. A fetched issue describes what should change.

A coding agent needs all of them. They do not all have the same right to tell it what to do.

That is the boundary I find more useful than telling an agent not to trust the internet. A repository can contain relevant evidence, authorized project instructions, obsolete advice and deliberately hostile text. Sometimes those arrive through the same search result.

The problem is not reading a suspicious sentence. The problem is promoting a sentence from material under inspection into authority over the inspection.

The source being reviewed cannot appoint itself the reviewer

Consider an illustrative PR that adds a comment describing a component as exempt from ordinary validation. The same change tells automated reviewers to ignore failures involving that component.

The comment may be useful evidence about the author's intent. It is not, by itself, an approved exception to the team's validation policy.

If the agent treats it as an instruction, the proposed code has acquired a way to influence how it is judged. The author need not have changed a tool permission or a protected policy file. The confusion happened in interpretation.

This can be deliberate prompt injection. It can also resemble an ordinary documentation mistake: an outdated setup note asks for a step the current task doesn't need. The intent of the text and the authority of the text are separate questions.

I would not claim every incorrect README instruction is an attack. I would ask whether the agent is using it as context to verify or as a command that overrides the actual task.

Repository-local doesn't mean equally trusted

An instruction file deliberately selected by the user or organization has a different role from a copied log line. A task description from an authorized owner differs from an arbitrary comment underneath it. A policy on the reviewed branch may itself be part of the proposed change.

That doesn't mean refusing all repository instructions. It means keeping their origin, scope and review status visible.

For a PR reviewer, the base branch's approved policy and a proposed replacement on the head branch should not silently become the same source. A legitimate policy update may need review. Its existence in the diff doesn't automatically authorize it to govern that review.

Likewise, a retrieval tool returning content doesn't authenticate the instructions inside that content. The tool may be trusted to fetch a page accurately while the page remains untrusted to direct actions.

The distinction also applies to generated summaries. If a summary turns a quoted instruction from a fixture into an unqualified next step, it can lose the original boundary while preserving the words.

Accurate retrieval is not delegated authority.

I want a handoff to preserve that difference, not just preserve more text.

Current controls matter, but they cover different surfaces

This is not an argument that current coding agents operate without protections. The VS Code security guide, checked on October 23, 2026, documents workspace trust, approvals, sandboxing, MCP-server trust and information-exposure risks.

The details depend on the active harness and tool path. Its guidance distinguishes process sandboxing from permission checks on other tools and warns that sandboxing is not a standalone security boundary. That is more useful than assuming one toggle makes every route equivalent.

Instructions guide the model. Permissions determine which operations are available. Isolation can constrain what an executing process reaches. Network restrictions constrain destinations. None of these automatically proves that a proposed change satisfies the task.

The current GitHub application card also describes different environments and data flows for its agentic surfaces. I wouldn't transfer a cloud agent's documented constraints to a local terminal session merely because both carry the same product name.

Good controls reduce the consequences of a confused decision. They do not remove the need to distinguish data from instructions.

Keep the action boundary outside the prose

A document can recommend a command. It should not be able to grant that command new permissions.

I would enforce consequential restrictions in the tool or execution environment rather than depend solely on the model agreeing to follow a warning. A reviewer that doesn't need mutation tools shouldn't receive them by default. A task needing one repository shouldn't inherit every account a developer happens to use.

That principle is narrower than prescribing a particular sandbox configuration. Filesystem access, credential availability, network routes and connected services vary by environment. The controls have to be checked where actions actually run.

A tool wrapper that inspects only one command shape may miss a different execution route. A hook's failure or timeout behavior may not match an engineer's assumption about stopping. I would test those boundaries instead of treating the existence of a hook as enforcement.

There is also a quiet escalation to avoid: allowing a document being analyzed to nominate additional tools or a more privileged environment as prerequisites. A claimed need for access is still a request to evaluate.

I discussed useful tool context in Tools Are Part of the Agent's Intelligence. That usefulness makes precise tool authority more important, not less.

Test whether an instruction stays inside its source

For an authorized local evaluation, I would use harmless fixtures. A source file or mock tool result could request a clearly out-of-task change to a disposable marker file. The expected outcome would be to recognize the instruction as content, not carry it out.

No real secrets, external collection endpoints or production systems are needed to establish that question. The test should inspect actual tool activity as well as the final explanation. Saying that an instruction was ignored while performing its requested action is not a pass.

I would vary the source: a comment, a generated report, a quoted issue reply and a summarized tool result. The aim is to check the boundary through paths the workflow actually uses, not to collect theatrical attack phrases.

Passing such fixtures is evidence about those cases. It is not proof against every prompt injection. The environment still needs constraints that limit the effect of a missed case.

Legitimate instructions matter in the evaluation too. An agent that refuses to follow the approved project setup has not solved trust by refusing all work. I want correct treatment of different sources, not indiscriminate suspicion.

A stopped action should leave an understandable reason

If a source asks for an unexpected operation, the useful response identifies the source and the conflict: this instruction appeared in material being analyzed, but the task did not authorize the action.

A person can then confirm an actual requirement through the appropriate channel. The agent doesn't need to turn every questionable paragraph into an emergency. It needs to avoid quietly changing who is in charge.

Logs can help reconstruct that decision, but they should not copy entire sensitive documents merely to preserve evidence. Record enough origin and action information to investigate, with appropriate access and retention.

I don't want to make agents blind to repositories. I want them to read carefully without making every readable sentence a new supervisor.

When a coding agent fetches something in your workflow, what establishes whether that content may direct its next action? I'd be interested in the boundaries you have actually tested, especially the ones that survive summaries and handoffs.

The Agent Finished. The Reviewer Is Still Reading.

Maksym Kuzmitskyi (MaximusFT) — Sun, 25 Oct 2026 00:00:00 +0000

The agent has finished the implementation. CI is green. There is a useful PR description, a test summary and another task ready to start.

The reviewer is still trying to understand the first change.

I don't think that makes the reviewer the slow part of an otherwise efficient machine. Generation and acceptance are different kinds of work. Producing another plausible implementation does not create another person's attention to examine it.

The interesting question isn't how many PRs an agent can open. It is how much consequential change a team can understand, validate and own without turning review into an exercise in trusting the summary.

A faster author changes the queue

Imagine a team where generated changes arrive faster than reviewers can assess them. This is a hypothetical operating condition, not a productivity measurement from my work.

The immediate symptoms might look encouraging: more branches, more completed implementation tasks, more green checks. The waiting work has simply moved.

Opening several PRs in parallel may shorten implementation time for each author while extending the delay before any reviewer can concentrate on one. Interrupting reviewers with repeated requests makes their effective capacity smaller still.

This is a workflow problem, not an argument against agents. A compiler getting faster never meant that architectural judgment became free. The same distinction deserves to survive another increase in code-generation capacity.

OpenAI's 2026 harness-engineering report describes human QA capacity becoming a constraint as code throughput increased in its own internal experiment. Its response involved making application behavior and observability more accessible to agents. Those are observations from a particular environment, not a throughput target or permission model I would import wholesale.

For my purposes, the important part is the changed constraint. If acceptance is waiting, making authors produce more work may be optimizing the wrong stage.

Reviewability isn't a line-count limit

A small diff can contain a large decision. One changed authorization condition may deserve more attention than a generated file containing hundreds of mechanical edits.

I would size review around behavioral questions: what changed, who depends on it, which assumptions changed, and what evidence is available. Line counts can help notice outliers. They do not calculate risk.

Consider an illustrative PR that introduces a shared utility, migrates consumers, changes a dependency and refreshes unrelated snapshots. Every individual edit might be reasonable. Together they create several distinct review jobs.

The reviewer has to separate the intended behavior change from migration mechanics, dependency effects and test-expectation updates. A detailed explanation can help. A cleaner division of work may help more.

That doesn't mean splitting code into arbitrary tiny PRs. A sequence of fragments that cannot be tested or understood independently can increase coordination costs. I would look for cohesive behavioral slices with explicit dependencies and a valid state between them.

A reviewable change gives someone a bounded question they can actually answer.

The unit isn't necessarily one file, one agent session or one convenient commit. It is an intelligible decision with enough surrounding evidence.

Summaries should lower the cost of checking

A good summary tells me where to inspect, not merely what conclusion to reach.

For a permission change, I want the affected rule, the relevant route and the check that exercised a denied action. For a shared component change, I want the changed contract and the consumers that make it consequential. For an expected-result update, I want the requirement that changed.

I don't need a transcript of every search or an elaborate account of the model's internal reasoning. I need references I can follow and limitations I can see.

An agent saying that all relevant scenarios passed should be able to identify those scenarios. If a check was unavailable, the absence should remain visible rather than disappear inside a reassuring overall conclusion.

This is related to useful automated review comments, but it is a different responsibility. The question here is whether the whole change can be accepted with reasonable effort, not whether an individual bot comment is well phrased.

Long descriptions can make the problem worse if their conclusions are easier to read than their evidence is to verify. I would rather have three precise links and one unresolved question than a confident report that nobody has time to audit.

Another agent can remove work, or add another queue

Automated review can catch local defects, investigate a suspicious export or check a routine convention before a person arrives. That is useful preparation.

It can also generate a second pile of work: findings to validate, false positives to dismiss, replies to interpret and repeated comments to reconcile. A quiet tool isn't necessarily effective, and a busy one isn't necessarily helping.

GitHub's current application card for Copilot agents explicitly discusses missed issues, false positives and inaccurate suggestions, and recommends supplementing automated review with human review. These are present documented limits, not a claim that current tools only inspect a few lines of a diff.

I would assess whether a review tool reduces the human effort needed to reach a sound decision. That includes the effort spent checking the tool itself. Comment count is a poor substitute.

The same applies to UI evidence. Browser automation can make a previously invisible interaction inspectable, but a screenshot doesn't verify every state. Helpful automation narrows the remaining question. It doesn't just add another green label.

Control work in progress before asking for faster review

If review capacity is the constraint, I would try limiting how many changes are actively seeking attention. Completing one reviewable slice before opening three more can improve the flow without changing the model.

That limit should be a team decision, not a universal number. A routine dependency stream and a change to a shared identity contract don't impose the same load.

Review ownership also matters. An agent can identify potential consumers, but that doesn't automatically identify who is accountable for accepting the change. A PR waiting for everyone can quietly become a PR owned by no one.

I would make blocked states explicit: waiting for a domain decision, waiting for a missing integration check, or waiting for a reviewer with the relevant ownership. Those call for different actions. Re-running the same agent is unlikely to resolve all three.

There is a useful stopping point for automation here. When the next step is an accountable decision, generating more implementation isn't progress on that decision.

Measure the waiting and the rework

I would keep measurements modest and specific: how long changes wait for first substantive review, how often a PR needs to be separated, what corrections arrive after apparent completion, and which review questions repeatedly lack evidence.

These are proposed observations, not results I have measured for a comparative study. They need definitions and local interpretation. A shorter review can mean a better-prepared change, but it can also mean a rushed reviewer.

The goal isn't to make every change slower. It is to stop borrowing confidence from an overloaded acceptance process. If the team cannot explain what it accepted, the implementation speed has hidden part of its cost.

I still want the agent to finish quickly. I also want its finished work to be something another engineer can pick up without reconstructing an entire conversation.

When I look at a queue of generated PRs, that is the question I would put ahead of the next throughput improvement: can the people responsible for this system still understand what is entering it?

Where does review capacity constrain your AI-assisted workflow: understanding the change, checking the evidence, or sorting automated findings? I'd be interested in what actually reduced that burden, not just what produced more output.

Who Checks the Tests an AI Agent Writes?

Maksym Kuzmitskyi (MaximusFT) — Fri, 23 Oct 2026 00:00:00 +0000

An agent writes a function, adds tests, runs them, and reports success. The implementation and the tests agree.

What established that they agreed on the right behavior?

This is the part of AI-generated testing I want to examine before counting new assertions. A test can be perfectly executable, exercise the relevant branch, and confirm the same mistaken interpretation that produced the code.

Humans do this too. The additional concern with an agent is how conveniently implementation, test cases, fixtures and the final explanation can be produced from one interpretation. They look like several pieces of evidence. They may have only one underlying assumption.

I don't want fewer generated tests. I want a clearer answer to where their expected results came from.

A passing test for the wrong rule

Consider a fictional checkout policy. Free shipping applies when the merchandise subtotal after discounts is at least 5,000 cents. Tax and shipping charges do not contribute. The examples below assume valid, nonnegative integer inputs, with discounts no greater than the merchandise subtotal; input validation is a separate concern.

This is an illustrative example, not an incident from a client project.

An implementation and its test might look like this:

import assert from 'node:assert/strict';

type ShippingQuote = {
  merchandiseCents: number;
  discountCents: number;
};

function qualifiesForFreeShipping(quote: ShippingQuote): boolean {
  return quote.merchandiseCents >= 5_000;
}

assert.equal(
  qualifiesForFreeShipping({ merchandiseCents: 6_000, discountCents: 1_500 }),
  true,
);

The test passes. The discounted subtotal is 4,500 cents. Under the stated policy, that order does not qualify.

The test has not failed to execute the implementation. It has asserted the wrong answer. A coverage report can show that the function ran without noticing that the expected value contradicts the requirement.

Adding more examples above the undiscounted threshold won't necessarily help. Nor will renaming the test to sound more specific. The missing ingredient is an expected result derived from the policy, rather than an observation of what the function currently returns.

That expected result is the test oracle: the basis for deciding whether the observed behavior is correct. It doesn't have to be a separate program or an expensive formal specification. Sometimes it is one arithmetic calculation and a sentence someone has actually agreed on.

Establish the expectation before polishing the implementation

For this policy, I would start with a few examples whose answers can be worked out without consulting the function:

assert.equal(
  qualifiesForFreeShipping({ merchandiseCents: 6_000, discountCents: 1_500 }),
  false,
);
assert.equal(
  qualifiesForFreeShipping({ merchandiseCents: 5_000, discountCents: 0 }),
  true,
);
assert.equal(
  qualifiesForFreeShipping({ merchandiseCents: 4_999, discountCents: 0 }),
  false,
);

The first case disagrees with the original test. That disagreement is useful. It makes the interpretation visible instead of burying it beneath a successful run.

The implementation can then follow the policy:

function qualifiesForFreeShipping(quote: ShippingQuote): boolean {
  const discountedMerchandiseCents =
    quote.merchandiseCents - quote.discountCents;

  return discountedMerchandiseCents >= 5_000;
}

This is not a complete checkout test suite. The cases establish the particular discount and threshold behavior under discussion. They don't establish currency conversion, tax treatment elsewhere, coupon validity or production integration.

I would ask the agent to identify the source of each important expectation. An acceptance criterion, an approved example, a documented contract or a trusted reference can justify it. The current implementation can tell us what happens, but not always what should happen.

If the requirement doesn't settle whether the threshold applies before or after discounts, generating an assertion doesn't settle it either. The correct next step is a question. An executable guess is still a guess.

A second agent needs a different starting point

It is tempting to solve this by assigning test generation to another agent. That can help, but a separate role is not automatically an independent basis for judgment.

If the second agent receives the implementation and a persuasive explanation of why it is correct, it may simply encode that explanation as tests. Changing the model doesn't guarantee a different interpretation of an ambiguous requirement.

For consequential behavior, I would give the test reviewer the original requirement and ask it to derive concrete outcomes before showing it the implementation. Then compare the cases with the code and existing tests.

This reduces one source of anchoring. It doesn't prove independence or eliminate shared mistakes. Both reviewers may still misunderstand the requirement, which is why the expected outcomes need to remain inspectable by a person.

In Green Checks Only Cover What You Checked, the concern was the reach of evidence across an integration boundary. Here the boundary is earlier: whether the assertion itself expresses the intended behavior. An integration test can have a wrong expectation too.

Check that the test can disagree

A useful test should fail for a relevant wrong implementation. I would deliberately check that property, rather than infer it from a green run.

In this example, ignoring the discount should fail the first requirement-derived case. Changing the threshold comparison from inclusive to exclusive should fail the exact-boundary case.

Mutation testing automates variations of this idea by introducing changes into the code and checking which tests detect them. Tools such as Stryker make that practical without inventing a mutation framework for each project.

A detected mutation establishes sensitivity to that change. It doesn't establish the truth of the original requirement. An incorrect expected value can still kill mutants; a surviving mutant can reflect equivalent behavior rather than a meaningful defect. Mutation scores need interpretation, not ceremonial targets.

Property-based tests add another useful angle. Under our stated policy, increasing the discount while holding the merchandise subtotal fixed must not turn an ineligible order into an eligible one. This describes a relationship across inputs instead of one fixed example.

But a function that always returns false also satisfies that relationship. Properties need enough independent constraints to rule out plausible wrong implementations. They complement concrete examples; they don't manufacture a business rule from nothing.

Fixtures can make the same mistake less visible

I would also inspect how test data is created. A fixture built by calling the implementation under test can quietly recycle the result it is supposed to check. A broad mock can remove the behavior that actually needs verification.

This doesn't make builders or mocks inherently bad. A builder can provide ordinary input data while leaving the important expected result explicit. A mock can isolate a collaborator without deciding the business outcome for the test.

The question is what the helper owns. Does it simplify setup, or does it calculate the answer through the same logic the assertion claims to validate?

Duplicating the production algorithm inside the test is not necessarily a better oracle. It can reproduce the same defect in another file. For a small example, independently calculated values may be easier to audit. For complex behavior, a reviewed reference implementation, contract examples or another representation may be appropriate.

There isn't one universally independent source. The point is to identify the dependence instead of mistaking a second file for a second line of reasoning.

Review the test change, not just its result

When an agent fixes a failing suite, I want to know whether it changed the behavior or changed what the suite accepts.

Sometimes the expected result is genuinely obsolete because the requirement changed. Sometimes the implementation is wrong. Sometimes the test was wrong from the beginning. All three can end with green CI. They are not the same engineering decision.

A snapshot update deserves the same question. Accepting a new snapshot records an output; it doesn't establish that the user should see it. Removing an assertion or relaxing a matcher may be justified, but the justification needs more than getting the run to pass.

For a bug fix, I would prefer the regression test to demonstrate the reported failure on the relevant prior implementation, then pass with the correction. It also needs to fail for the intended reason, not because an unrelated dependency or import is broken.

For new behavior, a failing test against the previous implementation is expected. What matters is that its assertion comes from the requested behavior, not from whichever output the new implementation happened to produce.

Keep generation useful

I like the prospect of an agent handling repetitive setup, exploring input combinations and proposing edge cases. Those are useful contributions even when the final expectation still needs human judgment.

What I don't want is to promote the existence of generated tests into evidence that the requirement has been independently checked. More assertions can make a mistaken assumption look thoroughly documented.

The shipping example is small enough that the disagreement fits in one subtraction. Real systems hide the same question behind fixtures, collaborators and years of existing behavior. I would still start by asking who decided the answer and what supports it.

If you use agents to write tests, how do you keep expected outcomes independent of the implementation? I'd be interested in the checks that expose shared assumptions, especially when both the code and its tests look completely reasonable.

Independent Releases Still Need Compatible Neighbors

Maksym Kuzmitskyi (MaximusFT) — Thu, 22 Oct 2026 00:00:00 +0000

The account team releases its new frontend. The shell stays on the previous version. Checkout was updated yesterday.

That can be a healthy arrangement. It is what independent deployment is meant to permit.

It is also a compatibility problem. Those versions need to work together, and users will discover the combinations we did not consider while each team tested its own application.

The earlier micro-frontends article asks whether organizational independence justifies the split. Here I want to examine the technical consequence after that decision: what independence requires during ordinary releases and rollbacks.

Separate three kinds of boundary

A code boundary determines which module can import another. An ownership boundary determines who may change it. A deployment boundary determines which artifact can be released without rebuilding the others.

They can align, but they do not have to.

A modular frontend can have feature packages, explicit APIs, separate owners, and enforced import rules while producing one application build. That is the modular monolith in this comparison, not a claim that all code belongs in one undifferentiated folder.

Micro-frontends add independently deployed application artifacts, composed through routes, runtime loading, frames, or another integration mechanism. Different composition methods have different costs. An iframe provides a stronger browser isolation boundary than mounting two modules in the same JavaScript runtime; it also changes interaction and integration constraints.

I would name the intended composition before discussing the benefits. "Micro-frontends" is otherwise doing too much work in the sentence.

One build moves compatibility work earlier

In a modular monolith, one build checks the selected package versions together. A failing interface change can stop the release before production sees it. A coordinated rollback returns that tested set of frontend modules together.

That does not guarantee correctness. Old tabs still exist, backend APIs still evolve, and integration tests can miss a workflow. But the frontend's internal version combination is more constrained.

With independent artifacts, compatibility work remains necessary and moves partly into runtime and release policy. The shell needs to understand the remote's contract; the remote needs to tolerate the supported shell versions.

The work did not disappear because CI pipelines became separate. We changed where it is paid.

A remote needs a supported contract

An illustrative mount contract might include a version, capabilities, lifecycle, and a constrained way to communicate:

type AccountModuleContract = {
  version: 2;
  mount: (container: HTMLElement, context: AccountContext) => () => void;
  capabilities: readonly ('profile' | 'notification-settings')[];
};

The shape is not a recommended universal micro-frontend API. It shows the questions a boundary needs to answer: what does the host provide, what may the remote assume, how does it clean up, and how can the host detect incompatibility?

Do not make shared context an unrestricted store containing every application's internals. That turns the mount API into a distributed import path with weaker verification.

Capabilities can make additive evolution easier than assuming that every participant upgrades simultaneously. For breaking changes, define a support window and migration path.

Release independence creates version combinations

If several remotes each support two versions while the shell also supports two, the possible combinations multiply. It is rarely practical to test every theoretical combination equally.

Constrain the supported matrix. Test each remote against the current shell and the declared compatibility window. Keep representative whole-product flows that cross the important boundaries.

A remote's unit tests may correctly show that its save action works. They do not show that the shell refreshes the account menu, preserves navigation, and clears the right state after logout.

Contract tests help, but a contract can be structurally valid and behaviorally wrong. The few shared workflows deserve integration tests owned by someone, not by an imaginary team named "the platform."

Runtime manifests are release artifacts

If the shell loads a remote entry through a mutable URL, the deployed composition can change without the shell's commit changing.

Record the selected remote versions or immutable asset references. Keep enough history to reconstruct what users were served. Rollback should select a known-compatible composition, not point several URLs at yesterday and hope.

Asset retention matters for clients that already loaded an old entry. Removing the old chunks immediately can break lazy navigation later, long after the initial page load succeeded.

CDN cache lifetimes can also cause different users to load different manifest versions. This is manageable if the contracts support overlap. It is inconvenient if independent deployment silently assumes instantaneous global consistency.

Sharing dependencies exchanges one cost for another

Shipping separate dependencies can increase download size and produce multiple copies of libraries. Sharing them at runtime can reduce duplication, but introduces version negotiation and lifecycle constraints.

React singletons deserve care when components and hooks interact across a shared runtime. A shared design-system library also has behavioral and styling contracts; two versions can differ in focus management or portal assumptions even when the buttons look similar.

I would not force every dependency into a singleton merely to improve a bundle report. Nor would I accept uncontrolled duplication without measuring the user cost. Decide which shared dependencies are part of the platform contract, and verify both upgrades and rollbacks.

For stronger isolation, route-level composition may be more straightforward than many independently loaded widgets sharing one screen. It depends on the product flow, not on which demo looked most flexible.

Isolation needs failure behavior

A remote can fail to load because of a network error, stale asset reference, or incompatible deployment. The shell still needs a useful route and a recoverable state.

Distinguish an optional sidebar from checkout's payment step. One may show a fallback. The other may need to stop the workflow and preserve user input. An error boundary catches some rendering failures; it is not full isolation from CSS conflicts, shared global state, or security problems.

Keep remote-loading timeouts, telemetry, retry rules, and fallback ownership explicit. A spinner with no deadline is not a graceful degradation strategy.

The modular monolith has fewer runtime composition failure points, but its deployment can affect more features at once. Feature flags, staged releases, and modular tests can reduce that risk without creating separate runtime applications.

Shared state is an authority question

Identity, cart state, locale, and navigation often cross frontend boundaries.

Decide who owns each truth. An event that says "cart changed" can prompt other applications to read the authoritative value. An event carrying a partial cart snapshot can accidentally create another source of truth with its own freshness problems.

An event bus should have typed contracts, lifecycle rules, and unsubscribe behavior. It should not become an excuse for every remote to discover undocumented events by reading the others' source.

Authorization must remain enforced at the backend. A remote being mounted by an authenticated shell does not give its API calls permanent authority.

These rules apply inside a modular monolith too. Independent deployment makes violations harder to repair together, which raises the value of keeping the shared surface small.

Try modularity before distributing a coordination problem

If the main pain is accidental imports, unclear ownership, or slow whole-repo CI, a modular build with boundary rules and affected-project checks may address it directly.

If teams genuinely need separate release schedules, existing applications cannot reasonably be merged, or one part must evolve under substantially different constraints, runtime composition may justify its cost.

I would trial the split at a boundary already understandable in the product, with a specific blocked workflow the trial is meant to improve. Measure release lead time and integration effort, then account for the platform work needed to keep it reliable.

A module boundary that cannot be described cleanly inside one build usually does not become clearer when a network load is inserted between its parts.

Independence has a maintenance agreement

The decision is not simply one deployment versus many. It is how much coordination we want to perform before release and how much compatibility we are prepared to maintain afterward.

I would choose a modular monolith when coordinated artifacts are affordable and the team's main need is better boundaries. I would choose micro-frontends when independent delivery solves a demonstrated constraint and the teams can own the resulting compatibility work.

Either arrangement can be maintained well. Neither relieves us of agreeing what a neighboring module may do while we are deploying something else.

If your teams already release independently, what happens when one of them rolls back? I'd be interested in the supported combination, because that answer says more about the architecture than the number of repositories.

The Agent Was Right. The Scope Still Changed.

Maksym Kuzmitskyi (MaximusFT) — Wed, 21 Oct 2026 00:00:00 +0000

A colleague started working on his ticket and discovered that its implementation was already there. He traced the files back to my pull request.

I hadn't found a technical problem with the additional work. The awkward part was explaining how it had become part of my task.

My ticket introduced a new abstraction, which I'll call a model. Another task covered integrating it at three selected usage sites. The older implementation served other consumers and needed to remain.

The agent built the model and covered it with tests. Then it noticed that the model wasn't being used. The task description explained what it would eventually replace, and the agent identified the places where it could be connected.

That was a reasonable suggestion. I accepted the argument and let it continue.

The integration was good work. It also belonged to another ticket.

This wasn't an agent secretly editing forbidden files. It asked, and I agreed. That makes the case more interesting than a story about automation ignoring instructions. Human agreement was present. What was missing was an explicit reconciliation between that agreement and the way the work had been divided.

The agent was right about what came next. That didn't establish that it belonged in this PR.

Unused is not the same as unfinished

A tested abstraction waiting for its integration task can look like incomplete engineering when viewed only through the current repository.

The code suggests an obvious next step. The ticket boundary may say something different.

That boundary isn't automatically sacred. Tasks can be split poorly. Combining implementation and integration may be the better decision. But it should be a deliberate change to the plan, with the affected people aware of it, not an accidental consequence of a convincing technical explanation.

I wouldn't require product approval for every extracted helper. Engineers need room to make implementation decisions. The useful distinction is between supporting work necessary to satisfy the current acceptance criteria and work that delivers a separately planned behavior.

An additional compatibility fix might be necessary for the requested change to function. Integrating a new model into consumers assigned to another task is a different decision. Both can be technically justified. Only one follows automatically from the current task's goal.

Knowing the intended destination is not the same as owning the next step.

This isn't unique to AI. A human engineer can make the same choice. But an agent can provide both the persuasive argument and the implementation. I need to examine the transition between those two, not only the quality of the result.

I started making the origin of decisions visible

My response was not another permanent instruction saying never do anything extra. That would make the agent less useful without resolving the ambiguity.

Instead, I made a distinction in the task records: work grounded in requirements, documentation, repository instructions or acceptance criteria, versus additions originating in the agent's own engineering judgment.

The latter should be presented as a proposal or inference, not quietly described as required work.

Both I and the review agent can inspect that record. It gives us somewhere to look when a change is sensible but its relationship to the task is unclear.

The important difference is between stating that a documented requirement calls for a change and stating that the agent recommends it because the current implementation appears incomplete. Those explanations deserve different review questions.

Of course, the agent can misclassify its own reasoning. A documentation reference may describe a future direction rather than a current requirement. A record written by the implementer is an account to examine, not independent evidence that everything happened correctly.

The reviewer needs the original task and relevant sources, not just the implementer's summary. It also needs to compare that account with the actual changes. Otherwise, two agents may agree because they inherited the same assumption.

For substantial work, this record can preserve decisions, checks, open questions and work intentionally left outside the task. A tiny, clear change doesn't necessarily need another specification file.

I've described the broader task-to-PR workflow elsewhere. Jira, Confluence, repository instructions, Git history and API documentation can be available through MCP integrations and other tools. Access helps only if the sources are current and their authority is understood. More context can also mean more convincing outdated instructions.

I want the record to make disagreement easier to locate. I don't want an impressive document that makes disagreement harder.

A good review needs several kinds of evidence

My personal PR-reviewer article focused on specific, checkable comments and visibility into what the tool inspected. That remains useful engineering attention. A removed export or disappearing permission check deserves investigation, not an unexplained warning.

But a reviewer can correctly assess an implementation and still miss why that implementation is in the PR.

Tests address particular behaviors under particular conditions. A local TypeScript check doesn't establish compatibility with a downstream consumer whose actual package contract differs. As I argued in Green Checks Only Cover What You Checked, the question is what the check exercised, not merely whether it passed.

Automation can change the object being reviewed too. My scope-after-tools article uses a synthetic dependency update where a hook regenerates unrelated API artifacts. The generator succeeds; the resulting commit still needs inspection. That example is illustrative, not a second incident from the model-integration story.

Review evidence also expires. A conclusion about one revision may no longer describe the next. A Review Belongs to a Commit examines revision tracking and guarded actions in personal tooling I've developed. Remembering a SHA helps identify what was inspected; it doesn't prove every finding remains valid or turn an earlier approval into authority over later changes.

These aren't interchangeable checks. Code correctness, dependency compatibility, task inclusion, action permissions and the relevance of evidence each answer a different question.

A passing test doesn't authorize extra work. An allowed directory doesn't define the business requirement. A second model's agreement doesn't establish independent verification.

Nor should every small edit trigger an investigation across every connected repository. The validation needs to follow the affected contract and the decision being made.

Three conversations around different parts of the problem

During October, three groups contacted me after reading my articles. The useful connection was their different approaches to questions I was already trying to answer.

In an early-October exchange with Kai, we discussed code relationships: imports, exports, calls, and how a finding might be challenged rather than immediately published. We also discussed limits of static analysis around runtime behavior and some React relationships. That is an account of the conversation, not a specification of today's product.

I proposed an evaluation with expected defects recorded before running the reviewer, without revealing the answers in the PR descriptions. Cross-file regressions, contract changes, React behavior and false positives would make useful cases. That evaluation was discussed, not completed.

The DeepPattern conversation approached the problem through verification: missed requirements, untested assumptions and whether the available evidence supports a conclusion. It also touched attribution. If a task succeeds, was the useful contribution a reusable skill, connected documentation, an existing test, or a human correction? Without a meaningful comparison, I can't isolate the cause. A preview was offered; I am not reporting evaluation results.

Agent Role Contracts, from Netsujo, addresses explicit declarations of roles, scope and handoffs. Its documentation makes an important boundary clear: internally consistent declarations do not grant runtime authority, authenticate a reviewer or establish business acceptance.

A task requesting writes outside a declared scope can be rejected before execution. My model-integration example asks a different question: what if every edited file is permitted, but the work belongs to another task?

These are complementary directions, not three products I've ranked. Code relationships, verification evidence and declared authority all matter. None alone decides whether a technically sensible addition belongs in this particular PR.

Move the scope question earlier, without pretending it is solved

By PR review, a disputed decision already has an implementation. The code can make the proposed extension look settled, even when the reason for including it remains uncertain.

I would like to make that uncertainty visible before implementation: compare the proposed work with the original goal, identify additions, and decide which ones require coordination. Then compare the actual result with the accepted scope.

That is a direction to investigate, not a semantic scope checker I've completed. Requirements can be ambiguous. Plans can omit necessary work. The reviewer can share the implementer's blind spot. A real change in understanding may justify revising the plan.

I would evaluate a mechanism like this on cases where the right response differs. One task genuinely requires a supporting change. Another should produce a suggestion to do the integration separately. A third needs clarification because the requirement doesn't settle the choice.

The expected outcome has to include stopping or separating work, not just producing more code. Otherwise, the evaluation rewards the very behavior it is supposed to examine.

I also wouldn't ask an agent for an elaborate account of its internal thoughts. I need inspectable grounds: which requirement it relied on, what it inferred, what it changed, and what the checks actually support.

My colleague didn't uncover bad code. He uncovered work that had crossed a task boundary while still looking like sensible engineering. I had participated in that decision. The part I want to improve is how such a decision becomes visible before someone else discovers their task inside my PR.

If you work on agent review, verification or workflow controls, how do you distinguish a necessary supporting change from an expansion that needs a separate decision? What evidence would you need to trust that the agent did the requested job, rather than a well-implemented neighboring one? I'd like to compare approaches, including where they fail.

TanStack Start or Next.js: Follow One Mutation Through Production

Maksym Kuzmitskyi (MaximusFT) — Tue, 20 Oct 2026 00:00:00 +0000

A dashboard loads correctly in both frameworks. That part of the comparison is already fairly well understood.

Now change an order, switch accounts, keep an old tab open through a deployment, and make the request through the production CDN.

That is the comparison I would spend time on. The first screen tells us whether the framework can render. The later operations tell us whether the team understands the state and deployment model it has chosen.

The earlier TanStack Start article introduces SSR and the server boundary. This one examines an authenticated, interaction-heavy application after that introduction. It is not a performance benchmark or a claim of personal production results for either framework.

Version scope: package metadata and official documentation were checked on October 8, 2026: next 16.4.0 and @tanstack/react-start 1.168.60. Features and host integrations below should be checked again against the project's pinned versions before implementation. This site itself still uses its existing Next.js version; writing the comparison does not upgrade it.

Compare a workflow, not an API count

Use one representative flow: load a private order, edit its delivery option, save, return to the list, and sign out.

It exercises identity, server authorization, validation, durable state, multiple cached views, URL state, and browser cleanup. Add an old-client submission after a rolling deployment, and it also exercises release compatibility.

I would build that slice with the deployment target intended for production. A development server is too forgiving a witness for cache behavior and runtime bindings.

The result need not be a second full application. It needs enough ordinary inconvenience to reveal which model the team can explain.

The center of gravity is different

Next.js App Router organizes work around route segments, layouts, Server Components, client boundaries, and its rendering and caching model. That is useful when much of the route can be composed on the server and the team wants the framework to own those conventions.

TanStack Start builds around TanStack Router, loaders, server functions, and the associated client navigation model. That is useful when typed route/search state and client-rich workflows are already central to the application.

Neither description is an exclusive capability boundary. Start's current documentation also describes Server Components; I would not use "Start cannot do RSC" as a timeless reason to choose Next. Nor does a Next.js application forbid TanStack Query or carefully designed client state.

The question is which default model makes the particular workflow easier for this team to reason about, and which extensions create a second overlapping model.

Server functions still need an API boundary

In either system, a function that runs on the server can feel like a local call from frontend code. That convenience does not make the request trusted.

Validate input, authenticate the caller, check access to the specific order, and enforce idempotency where repeated writes matter. A protected page or route guard improves navigation; it does not replace checks in the separately callable mutation endpoint.

For Start, the current server-function documentation describes compilation to client RPC stubs, validation, middleware, and same-origin protections. Next's Server Functions also need explicit authorization in the data path.

I would test both flows by calling the endpoint while signed out and as another account. The implementation syntax differs. The rejected unauthorized write should not.

Keep external APIs separate from framework-private RPC assumptions. Mobile clients and third-party integrations need a deliberately supported contract, not an incidental generated function identifier.

A successful mutation can leave several stale views

After saving the order, which copy changes?

In a Start application using Query, there may be a query cache, router loader data, server-side data caches, and a CDN response. Invalidating one does not automatically invalidate every other layer.

In a Next application, server data and route output can have their own cache behavior, while the browser retains navigation state and perhaps a separate Query cache. Adding Query is reasonable when it serves a clear purpose. Using it merely because it is familiar can create two owners for the same remote data.

I would write down the expected invalidation path before selecting helper functions:

save delivery option
  -> commit the order change
  -> update or invalidate the detail view
  -> refresh affected list representations
  -> preserve unrelated local form state

For Next 16, be explicit about whether the application uses Cache Components or the previous caching model. Advice copied from a different major version or mode may describe different defaults. The current caching guide distinguishes those paths.

I would not choose a framework because its cache seems invisible. Invisible is convenient until two views disagree.

Request isolation matters more than friendly types

On the server, create user-specific state in request scope. A process-wide Query client or mutable user store can accidentally carry account data between requests if used incorrectly.

Test two accounts concurrently through SSR and the real CDN. Inspect returned HTML and serialized data, then switch accounts in the same browser. Sign-out should remove private client cache state as well as the cookie.

The Start production checklist explicitly addresses these checks. They are useful for either framework.

A type-safe cache key still needs the correct identity dimensions. If two tenants share the same key, the compiler can happily confirm that the wrong tenant's response has the expected shape.

Hosting is part of the choice

Start's hosting documentation describes provider-specific integrations and build outputs. Next's self-hosting documentation describes multi-instance caches, deployment identifiers, Server Function encryption keys, and streaming considerations.

Both require attention outside the route code. Verify the adapter, runtime APIs, environment-variable timing, asset paths, and proxy buffering. A container deployment and an edge-worker deployment are not interchangeable merely because both serve HTTP.

For Next across multiple instances, plan shared cache behavior and invalidation coordination where needed. For Start, verify the selected server output and host bindings rather than assuming the development runtime is the production runtime.

Do not judge self-hosting by whether npm start returned a page once. Judge whether the intended features behave correctly through the intended infrastructure.

Keep an old browser alive during the test

Server-function identifiers and client assets belong to a build. A tab can retain an older build while the server deploys a newer one.

Next documents deployment skew protections, including deployment identifiers; their exact behavior matters for self-hosting. Start also generates server-function identities, so test old-client calls against the actual deployed output rather than assuming type sharing guarantees runtime compatibility.

Retain required immutable assets. Decide whether an incompatible client should reload, and what happens to unsaved edits when it does. A full reload is a recovery mechanism, not a product-neutral event.

This check can matter more to an internal editing application than the difference in a clean first-load demo.

Include maintenance in the experiment

Ask another engineer to change the saved field, add a validation failure, and diagnose a controlled stale-view bug in each slice.

How much behavior can they find locally? Which conventions are familiar to the team? Are the deployment docs and libraries they depend on compatible with the pinned versions? Can they explain the cache lifecycle without relying on the person who built the example?

I would prefer this evidence to a general claim that one framework is simpler. Simplicity depends on the workflow and on what the team already understands.

For server-composed routes and a team committed to the App Router model, Next is a reasonable choice. For router-driven, interaction-heavy work with a strong TanStack investment, Start is a reasonable choice. Either can be made awkward by layering on another system without deciding who owns what.

The framework I would choose is the one whose unhappy production path the team can describe clearly. A nice dashboard matters too. Fortunately, both can make one.

If you're weighing the two, bring one mutation and the hosting target rather than a feature matrix. I'd be interested in where the ownership of data becomes unclear in that concrete path.

Freelance Architect or Agency: Who Keeps the Decisions?

Maksym Kuzmitskyi (MaximusFT) — Sun, 18 Oct 2026 00:00:00 +0000

A team asks for an architect, but the work waiting for them is a backlog of implementation tickets.

Another asks an agency to build a feature, while every consequential decision still needs approval from an internal team that has no time to review it.

Neither arrangement is necessarily wrong. They are simply describing a different purchase from the one in the proposal.

When choosing between an independent engineer and an agency, I would spend less time comparing job titles and more time describing who owns the decisions while the work is happening. That tends to matter after the first few meetings, when everyone has access and the easy questions have been answered.

This is a way to evaluate an engagement, not a claim about particular suppliers, prices, or a universal ranking of one model over the other.

First separate diagnosis from capacity

Sometimes the team knows what it needs to build and lacks enough delivery capacity. Sometimes it has plenty of developers but cannot explain why every change crosses five modules. Sometimes a deadline exposes both problems at once.

An architecture-focused independent engagement can make sense when the central work is concentrated judgement: inspect a system, identify boundaries, work with the internal team, and make a few consequential changes.

An agency can make sense when the work needs a coordinated mix of implementation, testing, design, delivery management, and continuity across several people.

Those are tendencies, not guarantees. An independent engineer may deliver a complete feature; an agency may provide one specialist. The actual assigned people and operating agreement matter more than the supplier category.

I would write the first outcome in plain terms. "Reduce the number of modules needed to change policy validation" is more testable than "improve architecture." "Own delivery of this workflow through acceptance and handover" is clearer than "provide a frontend team."

Responsibility needs corresponding authority

If an external architect is responsible for quality but cannot change boundaries, influence sequencing, or obtain review decisions, the responsibility is mostly ceremonial.

The opposite arrangement is also uncomfortable: an external team makes choices that the internal owners must maintain, without involving them until handover.

For decisions with lasting consequences, agree who proposes, who approves, and who maintains. This does not need an elaborate governance document. A small record can be enough:

Decision: separate payment validation from page orchestration
Proposed by: external engineer
Approved by: internal payments owner
Maintained by: payments team
Evidence: two representative flows and regression tests
Fallback: keep the old adapter during migration

The example is hypothetical. Its useful property is that someone still owns the result after the external person's last invoice.

Review availability limits delivery

Adding implementation capacity does not increase the internal team's review or decision capacity automatically.

If an agency submits ten pull requests a week and the internal team can carefully review three, work accumulates at the boundary. More developers may increase that queue rather than shorten the release.

An independent engineer working on a difficult architectural change can encounter the same problem. One large refactor waiting for a busy owner is not inherently easier to absorb than several ordinary features.

I would include reviewer availability in the plan: expected turnaround, who can make product decisions, when integration questions get resolved, and how work is split into changes the team can evaluate.

This is particularly important when acceptance depends on integrations the supplier cannot access or change. A delivery date that assumes immediate internal answers is a date with an unstated dependency.

Prefer demonstrated uncertainty over polished certainty

Before a substantial commitment, a small paid diagnostic or bounded delivery slice can reveal the working relationship.

Choose something representative rather than conveniently isolated: a flow with an awkward API contract, an existing shared component, a realistic test requirement, and an internal reviewer.

The useful output includes what was discovered, what changed, what remains uncertain, and which assumptions affect the next estimate. An apparently slower answer that identifies a dependency may be worth more than a confident schedule built without examining it.

I would also distinguish an estimate for implementation from the time waiting on decisions, environments, and approvals. They belong in the same delivery discussion, but they are not the same kind of work.

Continuity is a mechanism, not a headcount

An independent engineer has a clear availability risk: one person may be absent or the engagement may end.

An agency may offer replacement capacity. That does not guarantee the replacement already understands the system. A named specialist can leave; an account can be reassigned; context still has to transfer.

Ask how continuity actually works. Which people have read the critical code? Where are decisions stored? Can another engineer deploy, reproduce an issue, and explain the outstanding risks without a private conversation with the original author?

For both models, I would favor shared context in the client's systems, small decisions recorded near the code, and at least one internal engineer involved in consequential changes.

Continuity becomes less fragile when it is exercised during delivery rather than promised in a final document.

The work should remain inside the client's control

Source repositories, deployment accounts, domains, package ownership, and design assets need deliberate ownership. An engagement can finish with working software and still leave the client dependent on credentials or infrastructure controlled by the supplier.

Use client-controlled repositories and appropriately scoped access where possible. Name any exception and the transfer procedure before it becomes necessary.

The same applies to automation. If a supplier uses AI-assisted tooling, clarify permitted data handling, review expectations, generated-code responsibility, and what configuration belongs to the client. Faster implementation does not change who must verify the result or who may access private source and customer information.

I would not equate a supplier's preferred toolchain with a requirement for the client to adopt it permanently.

Avoid a handover that only describes files

A directory tour helps. It does not answer why a boundary exists, which alternatives were rejected, or what would break if the API changes.

A useful handover includes the important decisions, reproducible checks, release and rollback steps, known limitations, and the first likely maintenance changes. Keep it proportionate; six understandable notes are better than a large template nobody can navigate.

Have an internal engineer perform one change and one deployment before the engagement ends. The external person can assist, but should not quietly do every step that the exercise is intended to transfer.

That is a more convincing ownership test than asking whether everyone has read the documentation.

Architecture needs an implementation consequence

If the engagement is mostly architecture, the deliverables should still meet the code at some point.

A dependency diagram can identify a problem. A bounded migration, representative slice, or working contract test shows whether the proposed boundary is usable. Without that contact, the design may be elegant while the team's practical cost remains unknown.

Conversely, measuring only shipped tickets can discourage the diagnostic work needed to prevent the next expensive change. Agree which technical outcomes matter: fewer coupled changes, faster verification, clearer ownership, or an integration that no longer needs several teams to coordinate every edit.

I would avoid treating diagrams and implementation as competing definitions of productive work. They should inform each other, with evidence appropriate to the risk.

Choose the arrangement that can leave well

For a focused diagnosis with a capable internal team, I would often consider an independent senior engineer or architect. For a delivery stream needing several disciplines and coordinated capacity, I would consider an agency with named people and explicit ownership.

If the organization has nobody available to own decisions, review changes, or maintain the result, neither category solves that absence by itself. The operating model needs attention before more external capacity arrives.

The end of the engagement is a useful design constraint from the beginning. What should the internal team be able to do without asking the supplier? Which promises must remain true when a person changes?

I would rather make those questions slightly awkward during selection than leave them until the final week. By then the software may be complete, and the ownership may still be under construction.

If you're deciding between these arrangements, tell me which decision or delivery responsibility is currently unowned. That detail makes the comparison much more useful than the labels on the proposals.