DEV Community: Scott Mallinson

The guardrail is the feature

Scott Mallinson — Mon, 31 Aug 2026 16:25:18 +0000

I've argued before that retiring feature flags is worth doing deliberately. The argument holds. The problem is that agreeing with it changes nothing, because the work stays a chore: find every reference, work out what the flag controls, figure out which paths go dead, clean them up without breaking anything. None of it is hard. It's fiddly enough to keep losing to whatever else is on that week.

So I've been building an AI-assisted flow to take the fiddly part out. Most of what's interesting about it isn't the automation. It's the two constraints around the automation: the rule that decides which flags it's allowed to propose removing, and the rule that decides who ends up responsible for the change.

What it actually does

The agent works through a small set of standardised skills rather than being handed a codebase and trusted to be sensible.

The first skill goes to the feature-flag service (LaunchDarkly, in our case) and establishes the flag's real state rather than its assumed one: whether it's enabled in a given environment, whether it has variations, and whether it's a candidate for removal at all. That last judgement is the one everything else depends on, and it's deliberately conservative.

If the flag clears that bar, the next step is a plan: what removing it would mean across every codebase that references it, not just the one you happen to have open. That includes the effect on tests: which cases exist only to exercise a branch that's about to stop existing, and what that does to coverage once they go. The output is a plan a developer can act on, not a set of changes already made.

The rule that makes it safe

The guardrail that does the most work is narrow and boring: a flag with variations is flagged as not safe for removal.

Variations mean the flag is still doing something. It might be mid-rollout, it might be splitting traffic for an experiment, it might be serving different values to different segments for a reason nobody wrote down. In every one of those cases the flag has a live job, and the honest answer to "can this go?" is no, regardless of how dead the code around it looks, and regardless of how confidently a model could argue otherwise from reading the source.

That rule is worth more than any amount of cleverness in the analysis, because it fails in the safe direction. The cost of wrongly excluding a removable flag is that somebody removes it by hand later, which is exactly the situation we were already in. The cost of wrongly removing a live one is a production behaviour change nobody asked for, arriving inside a cleanup pull request where no reviewer is looking for it. Those two errors are not remotely symmetric, and the tool should be built as though they aren't.

This is the part I'd tell anyone building something similar to spend their time on. Getting a model to trace references and spot dead branches is the easy half now. Deciding what it is not allowed to conclude is the half that determines whether you can leave it running.

Who owns the change

The second constraint is about people rather than code, and it turned out to matter more than I expected.

The developer is shown the implications of removing the flag and remains responsible for orchestrating the removal. Every pull request that comes out of the process is authored by the developer, not by the agent. That isn't ceremony. A pull request carries an implicit claim: I have read this and I believe it's correct. That claim has to belong to somebody who can actually make it. An agent-authored PR quietly moves that claim nowhere, and review gets thinner precisely when the change is mechanical enough to skim.

So the agent produces understanding and a plan. The person produces the change.

What actually changed

Here's the part I didn't anticipate when I started.

The bottleneck was never really the mechanics. It was that flag removal was distributed across code owners, and it only happened when each of them independently got round to it. A flag might be dead in three repos and live in a fourth, and the person who most wanted it gone (whoever introduced it, shipped the feature, and would happily never think about it again) had no way to finish the job themselves. They could ask. Then they could ask again.

With the analysis and the plan handled, that changes. Removal can be orchestrated end to end by the flag's owner, across all the codebases at once, which is the only way you get the guarantee that actually matters: that the flag is gone everywhere rather than mostly gone. What used to take weeks of waiting on other people's priorities now takes a day or two.

The speed is nice. The completeness is the real win. A flag removed from three of four codebases isn't 75% removed, it's still there, and the conditional you left behind is now the confusing kind: the one that disagrees with its own name in the other three repos.

What I can't tell you yet

Two honest gaps.

The first is that I can't give you a failure rate. It gets things wrong rarely, which is the kind of claim I'd be sceptical of if someone else made it, and the sceptical reading is fair: "rarely" is what you say before you've counted. The conservative guardrail means most of its errors should be refusals rather than bad removals, and refusals are cheap and invisible, which is comfortable and also exactly the condition under which you'd fail to notice a pattern. Counting properly is the obvious next thing.

The second is that everything above describes a tool being used by the person who built it, which is the easiest possible case. The packaging work (standardising the skills, tightening the prompts so results are consistent, making it runnable without knowing how the thing is wired underneath) was aimed at someone else picking it up. Whether that survives contact with a developer who has no investment in it, and who meets its refusals as obstacles rather than as design, is not a question I can answer from here.

What I'd defend is the shape. A tool that reads a system, forms a view, and hands a person a plan they have to sign their name to is a more comfortable thing to leave running than one that reads a system and starts making changes. Not because the model can't be trusted with the mechanics (it mostly can) but because the mechanics were never the part that was going wrong.

Bad acceptance criteria ship as bugs

Scott Mallinson — Mon, 24 Aug 2026 16:57:00 +0000

Doing requirements review on a complex booking flow, I hit an edge case the acceptance criteria hadn't addressed: what happens when a user retrieves a saved trip quote and amends it?

The naive answer and why it's wrong

The obvious implementation is to replace the original quote with the updated one. It's the simplest path: save creates a record, amend updates it. One record, current state.

But if the original quote was already shared with a traveller (sent as a link, or handed off to a booking agent), updating it in place changes what they see when they open the link. The flight options, the price, possibly the itinerary: all different from what was quoted. That's not an amendment, it's a mistake that looks like an amendment.

The right structure is parent/child lineage: the original persists unchanged, the amended quote references it. The sharing link always resolves to the specific version it was created for. Lineage lets you answer "what was this quote when it was shared?" without archaeology.

This is a data model decision. A small one, in the grand scheme, but once the schema is built around simple replacement, retrofitting lineage means a migration. Caught in requirements review before implementation starts, it's a question: "should amended quotes keep the original?" Caught during a sprint, it's a refactor.

The other places ambiguity lives

The same set of acceptance criteria had a few other unresolved questions.

What does "stale" mean for a trip quote? Time-based (saved more than N days ago), availability-based (one of the flights is no longer bookable), or price-based (the fare has changed)? Each definition implies different logic and a different user experience. The acceptance criteria used "stale" without specifying. A developer implementing staleness detection would have to pick one, and they'd probably pick the easiest one, which may not be the intended one.

What does "traveller-ready" mean? This turned out to be a status concept: a quote that's been validated for handoff to the traveller, with complete information and confirmed options. But the criteria used it more loosely, sometimes to mean "ready to book" and sometimes to mean "ready to share." These are different states with different requirements.

"Refresh" was similarly underspecified, used in some places to mean re-fetch live pricing and in others to mean rebuild the search from current parameters. The distinction matters when a traveller returns to a quote that was priced under a sale fare that's since expired.

None of these are traps. They're honest gaps in a set of requirements being written for a complex system. The job of requirements review is to find them.

As I write, both the lineage question and the definition of "stale" are still with Product, undecided. That's not a failure of the process: it's the process working. A question sitting with the people entitled to answer it is in a considerably better place than an answer a developer improvised at implementation time and nobody else ever saw.

The one that got through

All of those were caught. Here's one that wasn't, which is the more instructive case.

Everyone working on the assistant (including the developers, including me) carried an assumption that its flow was linear: search, then quote, then book. One path, three steps, in order. Nothing in the acceptance criteria said that. Nothing said otherwise either. It was simply the shape everybody pictured when they read them, and because everybody pictured the same shape, it never came up.

The real flow branches. A user can go from search straight to booking without a quote in between. They can retrieve a saved quote and book from there, skipping search entirely. The linear version wasn't a simplification of the truth, it was a different structure, and code written for a pipeline doesn't become a graph by adjusting a parameter. Correcting it meant a significant refactor.

That's the ambiguity that actually costs you. The ones I found in review were visible because the words were doing obvious double duty: "stale" and "refresh" announce themselves as underspecified the moment you look at them properly. This one was invisible precisely because there was no disagreement to detect. A gap everyone fills the same way doesn't read as a gap at all. It reads as shared understanding, right up until the point where reality supplies a third path nobody's implementation accounts for.

I don't have a clean technique for catching those. The closest thing I've got is that when a flow is described in a sequence, it's worth asking explicitly whether it's the only sequence, not because the answer is usually no, but because the question is one nobody thinks to ask about a thing they already believe they understand.

Criteria that can't be written as Given/When/Then

There's a broader outcome from this project that I didn't expect, and it's changed how I read acceptance criteria generally.

Product writes the criteria. That hasn't changed. What has changed is that we now treat them as guidance rather than as criteria in the strict sense, because you can't hold an LLM-mediated flow to literal Given/When/Then scenarios.

Gherkin assumes a deterministic mapping: a given precondition and a given action produce a specified outcome, and the test is whether that outcome appeared. Put a language model in the middle and that assumption breaks. The same Given and the same When can produce a range of outcomes that are all correct: differently worded, differently ordered, differently complete. The useful question stops being "did it produce this output" and becomes "is this within the range of right answers."

Criteria written as strict scenarios then fail in one of two directions. Either acceptable behaviour quietly falsifies them, and you spend your time reconciling a passing feature with a failing scenario. Or you specify so narrowly that the only way to satisfy the letter of the criteria is to constrain the model until you've designed it out of the feature you were building.

Treating criteria as guidance sounds like a loosening, and it would be if nothing replaced the rigour. What replaces it is being much more explicit about the things that are deterministic (the data contract, the boundaries, what must never happen) and accepting that the conversational surface needs judgement rather than assertion. The precision moves. It doesn't disappear.

Writing architecture up the stack

Alongside the acceptance criteria work, I wrote an architecture and current-direction document for director-level engineering review. Not a system diagram, but a narrative: what the AI-assisted trip quoting system is doing, why the structure is what it is, and where it's heading.

In the interests of not overselling it: nothing came back that changed the direction. No question from that audience reframed the problem for me. The value was almost entirely in the writing, which I've come to think is the more reliable benefit anyway. You can't explain a system's direction clearly without having a clear model of it, and the places where the narrative wouldn't hold together were places where my thinking hadn't, either. Writing at that altitude forces different questions than implementation does, not "how does session state get passed between services?" but "is the AI layer appropriately separated from the booking services, and does that hold as the feature grows?" Getting no pushback isn't proof the answers were right, but the exercise had already paid for itself before anyone read it.

The product/architecture intersection

One thing that came out of this review: an AI assistant that builds and refines trip quotes occupies similar space to a basket that lets users save and compare quotes. If both features are built in parallel without deciding how they relate, you end up with two surfaces that duplicate most of their behaviour, neither as good as one coherent thing.

Getting alignment on this before implementation is cheap. Getting it after both features are built is expensive: by then both have users, both have engineering investment, and neither team wants to deprecate their work. The direction of travel is that the basket gets reduced in scope as the AI assistant matures. That's a consequence worth knowing before the basket is built as a full-featured alternative.

The invisible contribution

None of this showed up in a commit log. No pull requests, no features shipped. The output was documents, surfaced questions, and a clearer set of requirements for the team implementing the work.

The contribution is real, though. The lineage question would have become a schema decision made without full context. The staleness definition would have been interpreted one way when the product team intended another. The position of the basket relative to the AI assistant would have been left implicit until it became a conflict.

And the flow assumption is what the other side of the ledger looks like. One shared, unexamined belief about the shape of a user journey, held by everyone and written down by no one, cost more than every ambiguity I caught in review put together.

Designing the API for an AI consumer is one part of building an AI feature well. Designing the requirements so the thing that gets built is the right thing is another. The second part is less glamorous, but it's where a lot of the risk lives.

The demo is the test

Scott Mallinson — Mon, 17 Aug 2026 16:57:00 +0000

The AI assistant feature worked. I'd run it through enough scenarios to be confident in that. Natural language input, iterative refinement, quote output: the core flows held up under questioning, and the tests were green.

Then we started running proper stakeholder demos, and a different kind of work began.

Some of what surfaced was the feature misbehaving in a room full of people. Most of it wasn't. It was everything around the feature that needed resolving before you could call it shippable, plus a category of failure the test suite was never going to catch, not because the tests were bad, but because they were structurally unable to see it. A test suite checks that the system does what you told it to do. It has nothing to say about whether you told it the right thing, whether the data arriving matches what you assumed, or whether anything outside the code is ready.

Four kinds of blindness came out of those sessions. None of them was carelessness. Each is something you genuinely cannot assess until someone who knows what the answer should look like is sitting in front of the product.

Fixtures are complete; provider data isn't

The flow had been working cleanly in controlled testing (flight search, option comparison, quote generation, return-flight addition) and it held up through the demo. Mostly.

Partway through, the results came back ordered oddly. The cause was in the fare search service: a flight provider had returned results without a duration field on some flights. Nothing dramatic, and nothing consistent: the field was absent, not null, not zero. Just missing, on some responses and not others. The scoring and sorting logic downstream had been written assuming duration was always present.

In fairness to the demo, it recovered on its own. It was an intermittent blip of the sort that finds you specifically when there's an audience, and we moved on in the room. But the intermittency was the interesting part rather than a reason to shrug: a failure that appears on some provider responses and not others is one that will pass every run of a test suite and then find you in production instead.

A wrong value is something you can test for. A missing field requires a more fundamental decision: what does your code do when the field simply isn't there? The assumption in the original code was implicit. The scoring logic accessed duration directly, with no guard for absence. When a provider omitted the field, the sort broke.

The result wasn't a clear error. It was broken result ordering, which in an AI-assisted flow looks more like "the assistant is picking odd options" than "there's a bug in the fare search pipeline." That gap between where a failure happens and where it looks like a failure is what makes these worth tracing. Tracing it meant going through the provider response shape, the parsing layer, where the score was computed, and what order the output was sorted into. Every step had assumed complete data.

It's worth being precise about whose fault that is, because it changes the lesson. The provider publishes a schema. The responses don't reliably conform to it. That isn't an edge case we failed to imagine: it's a contract that isn't being honoured, which means the defensive work isn't optional politeness towards a well-behaved upstream. It's the actual cost of consuming that data at all, and it needs budgeting for as such.

The fix was to treat missing duration as a known input condition rather than an anomaly. Options without a duration are shuffled into the middle of the pack, so they sit neutrally when duration is taken into account in the ranking rather than being scored as though they were instantaneous or infinite. That choice is worth stating explicitly, because substituting anything into a sort has consequences: a neutral placement means a flight with no duration is never rewarded for the missing field, and never punished for it either. Deciding that consciously is the point. The original code decided it too. It just did so by accident, and badly.

The approach that holds up: at an API boundary, define explicitly what your code does when each field is missing. "Use a default" is valid. "Degrade gracefully" is valid. "Exclude the calculation for that result" is valid. Silent failure is not.

No fixture would have caught this, because fixtures are written by the same person who wrote the assumption. You don't fixture the shape you didn't know was possible.

What came out of it, beyond the fix, was better logging at that boundary: enough to tell whether a given failure traces to upstream data diverging from the provider's own specification or to something we did. When the answer to "is this ours?" takes an afternoon to establish, you end up debugging the wrong system on reflex.

Well-formed is not the same as correct

The second kind of blindness is subtler, and it produced the two findings I'd most want other people to steal.

We'd added a retrieve-saved-quotes flow to the assistant. A retrieve capability looks, from the outside, a lot like a search capability. You express what you want in natural language, and the assistant returns a list of matches. The input format is the same. The output format looks similar. But retrieval and search have different behavioural contracts, and designing one as though it were the other produces subtle failures.

Search is a narrowing operation. You have a corpus of options, you have a query, and you rank and filter until you have a manageable set of the most relevant results. The ideal search result is precise: the thing the user was looking for, surfaced confidently, with the noise stripped out.

Retrieval isn't like that. When a user asks "show me my saved quotes for the Johnson account", they don't want the system to guess which three quotes it thinks are most relevant. They want the list. The quotes are theirs; they know what they're looking for; the job of the assistant is to surface all of them and let the user decide.

The demo made this concrete in a way the specification hadn't. A user asked for quotes matching a fairly broad description and got back three results. There were twenty. The three were relevant in the sense that they matched the query's criteria, but returning three when twenty existed was the wrong behaviour. The user needed to scan a complete set, not a curated sample.

The fix required a behavioural specification: for retrieval queries that are broad but unambiguous (no specific price range, no specific date range, but a clearly-specified scope), return up to twenty results rather than filtering to a handful, and prompt the user to ask for more if they want to go past that. The cap isn't a judgement about relevance, which is the distinction that matters. It's a limit on how much to put in front of someone at once, with the remainder a request away rather than silently discarded.

The harder problem is making the distinction programmatic. When is a query a retrieval intent rather than a search intent? The heuristic: a query without specific narrowing criteria is retrieval; a query with explicit filters is search. "Show me my saved quotes" is retrieval. "Show me quotes under £5,000 for March travel" is search.

In practice that lives in the shape of the tooling rather than in prose. There are a number of structured tool definitions, each with an explicit contract, sitting under an overall system instruction that sets out when each should be called. The model can usually tell retrieval from search on its own, but "usually" is doing too much work if the distinction is left implicit: a model inferring behaviour from context will infer plausibly and consistently, which is precisely what makes a wrong inference hard to spot.

Because "usually" isn't good enough on its own, the prompts and the resulting decisions are logged, and we review them regularly to see where the classification is drifting and what needs refining. That review is the only thing standing between a misclassification and a silent wrong answer, since by construction the failure produces a well-formed response.

Legible enough to choose from

The second gap was in the data. When the assistant returned a retrieved quote, the response included a reference identifier and a small amount of metadata: enough for another system to load the full quote, not enough for a person to distinguish between quotes in a list.

In a system-to-system context, a quote reference is sufficient. The identifier points to the record; the receiving system loads the full object when it needs to. But when an AI assistant presents quotes to a user, the person needs to identify each one from what's visible. "Quote 1234" and "Quote 1235" are indistinguishable to anyone who doesn't already know their contents.

What the response needed was the customer's name, email address and phone number: enough for the person scanning the list to recognise which quote belongs to which conversation they've been having. Those fields existed in the underlying data model. The API response for referenced quotes had simply been built for system integration rather than human-facing presentation, so the response contract had to change.

This is a version of a broader pattern I've written about before: an API designed for one consumer type doesn't automatically work for another. For retrieval specifically, the data requirement is legibility: not just relevance, but enough information for a person to identify the item without opening it.

It also created a problem, which I'll come back to.

Why a test can't see either of these

Both issues share a shape: they look like success until you know what success is supposed to look like.

A response of three relevant quotes is a well-formed response. There's no error, no missing field, no failed assertion. The assistant returned matches for the query. The fact that it should have returned twenty is only visible if you already know there are twenty. A test can verify that a response contains results. It can't easily verify that it contains all the relevant results, because that requires knowing the ground truth.

The legibility problem has the same shape. "Quote 1234" is a valid response. It's only insufficient if you're the user who needs to choose between it and nineteen others. The specification said the response should contain a reference and some metadata. It did. The specification hadn't anticipated that a person would need to read that metadata to make a decision.

This is why retrieval features benefit from being tested with someone who actually knows the data. A developer testing against fixtures knows exactly what the fixture contains. A stakeholder testing against a real account knows what should be there and can tell immediately when something is missing or unreadable. The unit tests were green. The demo found both problems.

The failures that aren't in the code at all

The third category wasn't about the feature at all. It was everything around it, and the list turned out to be longer than I'd expected.

The first was infrastructure. The feature routes traffic through a CDN layer configured for a different set of traffic patterns: standard request-response cycles, not the longer-lived connections a conversational AI interface produces. That's not a code problem, and it's not something a development environment replicates. It surfaces when you start planning for real production traffic, often with another team who owns a different part of the stack. This kind of blocker is frustrating precisely because the feature itself is fine. Resolving it means coordinating with people outside the immediate team, making changes orthogonal to the feature, and accepting that your timeline depends on work you don't directly control.

The second was data governance. Using a cloud AI service means having clear answers to questions that don't arise with traditional backend services: where does the data go during inference, how is it classified, what are the compliance constraints, what logging or retention policies apply. These aren't hard questions in principle, but they require input from security, legal and platform teams, people who aren't usually in the room during feature development. Discovering that the process doesn't yet exist for your organisation's specific use case is something you want to find out before you have a launch date, not after.

The third was observability. In development, when something goes wrong, you attach a debugger, read the logs directly, and reproduce at will. In production you rely on instrumentation. AI features are harder to instrument than conventional ones: you're not just tracing a request through services, you're capturing enough context about what the model received, what it returned, and how that mapped to what the user intended, so you can diagnose problems after the fact. Finding out your telemetry isn't useful when you're trying to diagnose a live incident is worse than finding it out in a demo.

The fourth was commercial: pricing models, white-labelling requirements, and how the feature fits into customer contracts. None of that is engineering work, but it's coupled closely enough to implementation decisions (which endpoints are exposed, how usage is metered, what customisation is supported) that engineers end up in the conversation. Demos are good at surfacing this because they make the feature concrete enough for product and commercial stakeholders to engage seriously. Until they've seen it working, questions about pricing tiers stay abstract. Once they have, those questions become urgent.

What surprised me wasn't how many constraints surfaced. It was how useful the surfacing was. These are all still open, and all being worked rather than discovered, which is the entire difference between a constraint and a crisis.

Keeping the instrument honest

If the demo is doing this much work, it stops being an event and becomes a piece of test infrastructure. Which means it needs the same care you'd give any other test infrastructure.

For a feature with deterministic behaviour, the answer to "did the fix land?" is usually a failing test that now passes. For an AI-powered feature, correct behaviour is defined by human judgement. Twenty results when you asked for a broad range is correct; three is not. But there's no assertion for "enough." The only reliable oracle is someone who knows what the right answer looks like.

That makes the shared validation environment the primary feedback channel, not the test suite. And it makes the environment itself a variable you have to control, which is the part I underestimated.

We had people validating against an integration environment that wasn't stable enough for the job. Integration environments rarely are: they're shared, they carry whatever else is mid-flight, and their failures are nobody's in particular. The problem isn't the instability itself. It's that an unstable environment corrupts the one signal you're actually there to collect. When a stakeholder hits an error, the useful question is whether the feature is wrong, and in a flaky environment nobody can answer it. The feedback you get back is contaminated, and worse, it's contaminated in a direction that wastes your time: you go and investigate behaviour that was never yours.

The fix was to stand up an adjacent, more robust environment specifically for validation, and point people at that instead. It sounds like an infrastructure footnote. It isn't. It's the difference between a feedback channel and a noise generator.

The mechanics of getting builds there are automated, gated by pull request review, so this isn't a story about manual promotion bookkeeping: the machinery does what it's told. The judgement is upstream of the machinery: which environment do you ask a human being to form an opinion in, and does that environment's failure modes belong to you or to somebody else? Automating a promotion into an environment nobody can trust just gets you a reliable supply of unreliable answers.

The loop is: feedback, implement, promote, validate. How fast it closes depends less on the pipeline than on when the people whose judgement you need are actually available, which is worth saying plainly rather than claiming a turnaround time the calendar doesn't support. What you can control is that when they do look, they're looking at the right system.

The assumptions no demo will surface

For balance, one category the demo had nothing to say about.

In the trip planning view there was handling for a QUOTE_SAVED event that no longer needed to exist. The event had been there to trigger a UI refresh after a user saved a quote from the assistant. That flow had since changed: quotes are now created directly as referenced quotes rather than going through a save event. The UI coupling remained after the architecture changed, silently waiting to respond to an event that would never fire.

This is the same feature, and the same event, that I wrote about when it was first being built, where the problem was two components each having their own idea of what a saved quote was. It's a decent illustration of how these things age. The fix then was to converge on one shared event. The fix now was to delete it, because the flow it belonged to had moved on and nobody had told the handler.

Removing it was straightforward. Stale event handlers are a particular kind of noise: they look like intentional code, pass linting, don't cause test failures, and add confusion for anyone trying to trace how data flows. The test setup built around QUOTE_SAVED came out too.

The stakeholder review did surface some adjacent cleanup: quote option controls visible in flows where they no longer made sense, a focus restoration behaviour that needed work, a trigger path with more surface than it needed. But nobody in a demo was ever going to notice a handler waiting for an event that never fires. That one you find by tracing the save flow and noticing the orphan.

It's worth naming because it shares a root with the duration bug: code written to assumptions that had since diverged from reality. In one case an assumption about data completeness from a third party, in the other an assumption about an event still being fired. Neither was visible as a bug in normal operation. But only one of them was ever going to show up in a room. Demos catch the assumptions that produce visible wrong behaviour. They don't catch the ones that produce no behaviour at all.

The rules worth stealing

Each of these came out of something that had already gone wrong.

Define what happens when each field is absent, at the boundary. Not "handle errors". Specifically, for each field you read from an external system, decide now whether absence means a default, a graceful degradation, or an excluded calculation. The duration bug existed because that decision had never been made explicitly, so the code made it implicitly and badly.

Budget for the gap between a published schema and the data that arrives. If an upstream provider's responses don't reliably match their own specification, defensive handling isn't extra work you might get to. It's the cost of consuming that source, and pretending otherwise just relocates the work to whoever is on call.

State the intent you'd otherwise ask a model to infer. The retrieval-versus-search distinction was always in the designers' heads. It just wasn't in the tool definitions. And log the decisions, because a wrong inference here produces a well-formed answer that no assertion will flag.

Make responses legible to whoever actually reads them. A response shape is only correct relative to a consumer. "Quote 1234" is a complete answer to a booking engine and a useless one to a person choosing between twenty options.

Validate in an environment whose failures are yours. You are asking someone to judge whether your feature is right. If they're doing it somewhere flaky, you've asked them a question they can't answer, and you'll spend the next day investigating something that was never your bug.

Delete the coupling when the architecture moves. The QUOTE_SAVED handler survived the flow it existed for, passed every check, and cost the next person who traced that path an afternoon of confusion.

What's next, and what isn't resolved

The obvious objection to all of this is that "run more demos" is not a testing strategy. It doesn't scale, it depends on getting the right people in a room, and it catches problems late by definition: after the code is written, not before.

That's fair, and the honest answer is that the demo is currently doing a job no automated check is doing, which describes a gap rather than a design. Two of the three code-level failure categories are, in principle, mechanisable. Absent-field handling at a boundary is a property you could assert systematically rather than discovering one provider at a time. Ground truth for retrieval is harder, but "this account has twenty quotes, a broad query should return twenty" is a test you can write once you know to write it. What made both invisible wasn't that they were untestable. It was that nobody knew the assertion was needed until a person who knew the data ran the flow.

The concrete next step is less ambitious than automated evaluation and probably more useful in the short run: a runbook of the functionality worth testing and demonstrating, covering what's actually been built. Partly that's so a demo stops depending on whoever remembers the interesting paths. But writing it down is also how the implicit ground truth gets externalised, and every scenario in it that has a definite right answer is a candidate for an assertion later. The runbook is the cheap version of the eval set, and it has the advantage that it earns its keep immediately.

Then there's the thing that isn't resolved, which I'd rather state than let sit at the bottom of a backlog.

Making retrieved quotes legible meant putting customer names, email addresses and phone numbers into responses that flow through an AI assistant. That's personal data taking a new path through the system, and I've argued before that the logging layer should own that boundary rather than leaving it to each call site. Which is easy to write and harder to finish. The question of what gets retained, where, and for how long is acknowledged by everyone involved and settled by nobody yet, and it has to be resolved before this goes to production.

It's worth being clear that this is a consequence of a fix, not an oversight that predates it. The legibility change was correct: a list a person can't read is not a working feature. It just moved data somewhere it hadn't been, which is exactly the class of change that should trigger the question and often doesn't, because it arrives dressed as a usability improvement.

The feature isn't in production. It's going through internal testing, and all four categories of non-code blocker are still open work rather than closed decisions. So this isn't a story about a launch. It's a story about the gap between a feature working and a feature being ready, and how much of that gap only becomes visible when you put the thing in front of people who don't already know how it's supposed to behave.

Did the demos produce a better feature or just a better-documented one? Both, and I've stopped thinking those are separable. What changed most is my confidence in putting the thing in front of stakeholders at all, which turns out to be the same act as writing down what it does, performed in front of an audience that will tell you immediately when the answer is wrong.

Codebase health is a lagging indicator

Scott Mallinson — Mon, 10 Aug 2026 16:57:00 +0000

None of the work in this post shows up on a product roadmap. It's a tooling migration, some dependency bumps and a telemetry schema: the kind of thing that's easy to describe and hard to defend in sprint planning. It's also most of what decides whether a codebase is pleasant or miserable to work in three years from now.

Codebase health is a lagging indicator. You either pay the maintenance cost in small steady amounts, or you pay a bigger one later in debugging, security incidents and onboarding pain. What follows is the small steady version.

One toolchain instead of two

We moved a set of frontend services from ESLint and Prettier to Biome. ESLint and Prettier have been the default frontend linting and formatting stack for years, and they work fine. The friction is in running two separate tools: separate config files, separate plugin dependencies, the occasional conflict where Prettier reformats something ESLint just fixed, and a growing pile of wrapper packages and shared configs to keep consistent across repos.

Biome is one tool that does both. It's written in Rust, so it's noticeably faster than the ESLint and Prettier combination on larger codebases. It ships with sensible defaults, needs little configuration, and deliberately has no plugin system. For a team running several services, the draw isn't really the raw speed. It's the smaller configuration surface you have to keep consistent across all of them.

The migration was largely mechanical, and roughly the same in each repo:

Remove ESLint, Prettier, and their config files and plugins
Install Biome with a shared config inheriting from a common preset
Run the formatter over the codebase and commit the diff
Address the handful of lint rules where Biome's defaults differ from the old setup

In parallel, test tooling moved to a shared internal Jest preset, so repos that used to configure Jest themselves now inherit a standard setup. Less config to maintain, and a consistent test environment across services.

Giving up the plugin ecosystem is a real trade, and worth being honest that it's a trade rather than a free win: a plugin system is also the thing that lets you add exactly the rule your codebase needs. What we got back for it was fewer breaking changes to navigate and noticeably easier maintenance when it comes to updating dependency versions. On a stack spread across several repos, the tool you never have to think about is worth more than the rule you occasionally wish you had.

The formatting diffs were large on some repos, thousands of lines, but purely cosmetic. That's the usual worry with a mechanical formatting change: a noisy diff makes review harder and can bury real changes in whitespace churn. In practice it wasn't a problem. The diff views in GitHub and in the editor made the changes easy to read, especially where commits and pull requests stated plainly what they were for. Keeping formatting commits separate from functional ones is what makes that work: the tooling can only help you if you haven't mixed the two together in the first place.

The debt that came along for the ride

Touching these repos was also a chance to clear some accumulated dependency debt. The monorepo tooling in one repo was pinned to a version with known security advisories. Nothing actively exploited, just the sort of thing that keeps turning up in scans until you deal with it, so it went to the current stable release. A few repos had Node version pinning in CI sitting behind current LTS, updated at the same time.

None of this is interesting work. But leaving it to pile up means that six months from now, someone's investigating a security advisory under time pressure, or upgrading the monorepo tooling as an emergency instead of a planned change.

Saying where an error came from

The same instinct applies to observability, where most problems aren't really about the tools. They're about the data flowing into them being imprecise. Fixing that is duller than picking a dashboard, and it matters more.

Two services both handle PNR retrieval, the booking record at the heart of any reservation, and both had gaps. The first was missing structured telemetry logging entirely for its main flow. The fix was to add logging at each step, mapped to a defined schema. The schema is the whole point. Without one, every team reading the logs interprets freeform text its own way, and tooling can't aggregate anything reliably. Define it, and you can build dashboards, alerts and anomaly detection on top instead of reading log lines by hand.

The second was subtler. A service was surfacing errors from an upstream service layer without saying where they came from, so from the outside a failure looked like it originated in the retrieval service itself. The fix was to tag anything coming from upstream with the right source identifier in the telemetry, so an engineer debugging a failure can tell straight away whether the problem is theirs or upstream's.

The same idea turns up in static error classification, keeping an origin enum in sync, but here it's about what the logs say at runtime. In principle the two could disagree, the enum saying one thing and the telemetry another, though we've yet to see a case where they do.

The payoff is unglamorous and immediate: with honest, clearer logging we triage, identify and debug errors more quickly. Knowing where a failure actually came from costs nothing when everything's working, and is most of the job when it isn't.

The compounding bit

The value of standardisation work is almost invisible while you're doing it. Biome configured the same way everywhere, Jest set up identically, the advisories cleared, the logs saying which service a failure came from. Day to day, none of it feels like much.

It shows up later. Someone new moves between services without relearning the config. An upgrade goes through mechanically because everything follows one pattern. A lint rule catches a real bug that looser checking would have waved through. An engineer reads one log line and stops debugging the wrong service.

There's one piece of this I've left out, because it turned into a bigger subject than a section. Feature flags accumulate faster than they get cleaned up, and I've been building an AI-assisted flow to make removing them tractable. The automation was the easy part; deciding which flags it's allowed to touch was not. That's a subject of its own, and one I'll come back to shortly.

The trick with all of it is keeping this kind of work visible enough that it actually gets done, instead of being crowded out by features every sprint. It never argues for itself, because the incident it prevents is one you don't have and can't point at.

Designing a logging system for an AI service

Scott Mallinson — Mon, 03 Aug 2026 10:36:00 +0000

Logging is one of those things that accretes. Each developer who touches a service adds a line in the style they're used to, and over time you get a codebase where some errors are logged as exceptions, some as structured objects, some as plain strings, and some not at all. Test coverage is patchy in the same way. Nothing is obviously wrong with any single line, but the whole thing is harder to trust than it should be.

The work here was building a standardised logging system for an AI assistant service. It wasn't really about adding functionality. It was about coherence: defining how logs get created, what they contain, and how they're tested, then making that the path of least resistance so it stays consistent as more people add to the code.

What standardisation actually involves

In practice that meant a few things. A logging module that every part of the service imports. An interface developers use to emit logs instead of calling a logging library directly. Tests that check the shape and content of what gets logged. And enough documentation that the next person doesn't have to reverse-engineer the intent.

Going through the module instead of the library directly is about control. When all output passes through one layer, you can enforce a consistent structure, apply transformations uniformly, and change behaviour in one place rather than at every call site. It's the same instinct that has you wrap fetch in a single HTTP client rather than calling fetch all over a codebase.

The AI-specific problem

AI-adjacent services bring a logging problem that's easy to underestimate. You want enough detail to understand what happened: which input produced which output, where in a multi-step pipeline things went wrong, what the model was asked to do. But you have to be careful about what you keep.

User-provided input to an AI service can contain personal information. Names, travel details, payment data. Log request bodies indiscriminately and that data ends up in your log aggregation system, which probably has weaker access controls than your primary data stores and quite possibly keeps things longer than it should.

The right move is to make the logging layer responsible for this boundary, not the developers who call into it. You decide what's safe to log when you design the module, not at every call site. Adding a new instrumented line then doesn't depend on the developer remembering the rule. It works correctly by default. It's the same principle as any invariant the system relies on: enforce it once, at the boundary, instead of trusting every caller to get it right.

Health probes: the quiet dependency

A smaller change in the same spirit was correcting readiness and liveness probe URLs in a fare search service. The probes were pointing at endpoints that had been renamed at some point and never updated in the Kubernetes config.

Readiness and liveness probes are how the orchestration platform decides whether a pod is healthy and ready for traffic. Get them wrong and the platform either never sends traffic to a pod that's perfectly fine, or keeps restarting pods that don't need it. Both are confusing to diagnose unless you immediately suspect the probe config. It's a one-line fix. It's also the kind you'd much rather catch before production than after.

From merged pull requests to a performance review

Scott Mallinson — Mon, 27 Jul 2026 10:36:00 +0000

Every engineer I know is supposed to keep a work log, and almost none of them do. The intention is real. You start a running note, you fill it in diligently for about a week, and then a deadline lands and the note goes quiet. By the time the next performance review comes round you're staring at six months you can't account for, reconstructing your own year from commit history and half-remembered standups. The brag document everyone tells you to keep is empty, because keeping it is exactly the kind of low-urgency admin that loses every fight against actual work.

The thing is, I'm already producing a detailed record of what I do. It's just not in a note. It's in my pull requests.

The raw material is already there

A merged pull request is a surprisingly good primary source. It has a title, a description, a diff, the files it touched, a rough sense of scale, and a timestamp. String a week of them together and you've got a fairly honest account of where your effort went, written at the moment you did the work rather than reconstructed months later under duress. The problem was never a lack of data. It was that nobody wants to sit down on a Friday and turn a list of PRs into prose.

So I stopped doing that part by hand. A scheduled job reads my merged pull requests, hands them to an LLM with a fixed prompt, and gets back a short daily note: a one-line summary, a tidied list of what each PR actually changed and why it mattered, and a few tags for the domain and services involved. That note gets written into an Obsidian vault. Daily notes roll up into a weekly summary, and the weekly notes are what I actually read when a review comes round.

At work this runs through GitHub Copilot, inside my employer's approved tooling, so no code or PR data leaves the boundary it's meant to stay inside. The portable version I run at home does the same job through a self-hosted n8n workflow with a Claude doing the synthesis. The write-back into Obsidian is identical either way: the vault sits behind an MCP server fronting Obsidian's Local REST API plugin, so the model works through a small set of file operations (list, read, put, append, patch) rather than being handed a filesystem and trusted to behave. That constraint is doing more work than it looks like it is. A model that can only patch named regions of named files is a model that can't quietly reorganise your vault.

The whole pipeline is less elaborate than the description makes it sound:

See the original post at https://scottmallinson.com/from-merged-pull-requests-to-a-performance-review/ for the Mermaid diagram.

The prompt is the program

There's barely any code here. Each stage is a long, boring markdown file of numbered steps that gets handed to a model, and those files are the actual artifact: versioned, edited, argued with. One runs the daily loop over new pull requests. One does the weekly synthesis. One keeps a knowledge base of repo and service notes current against the organisation's GitHub.

The thing that surprised me is the ratio. The part of the daily prompt that says what to write is a handful of lines: summarise the change, note the components affected, judge the complexity, tag the domain. Almost all of the rest is defensive: rules about what the job must not do, must not assume, and must not conclude. Writing the summary was never the hard part. Not corrupting a vault you've been accumulating for years is the hard part, and a model that will cheerfully invent a plausible answer rather than admit a query failed is exactly the wrong tool to point at it unsupervised.

The bit that makes it usable

Pull requests only capture the work that turned into code. The review you unblocked, the design you argued for in a meeting, the demo you gave to stakeholders, the afternoon you spent helping someone else land their change. None of that shows up in a diff, and it's often the work that matters most when somebody is deciding whether you've grown into a more senior role.

So the generated sections are wrapped in marker comments, and anything I type outside those markers survives the next sync. The script only ever rewrites the region between its own start and end comments. I can drop a few bullets into a daily note about a meeting or a decision, and they sit there permanently while the PR-derived content underneath gets regenerated around them. The automation owns the parts it can see. I own the parts it can't.

That split turned out to be the whole trick. An automation that overwrites your manual notes is worse than useless, because you quietly stop trusting it with anything you care about. One that treats your own additions as load-bearing and works around them is something you'll actually leave running for years.

The rules worth stealing

If you build one of these, the defensive rules are the part to copy. Each of these earned its place by something going wrong first.

Use a real idempotency key. The PR URL is the key, and nothing gets appended to a daily note, a repo note or a career artifact until the vault has been searched for that exact URL. Re-runs then cost nothing and fix nothing twice. Without this you get a work log that quietly duplicates itself, which is a very slow way to discover you can't trust it.

Write to the merge date, not the run date. A pull request merged on Tuesday belongs in Tuesday's note even if the job runs on Thursday. It sounds obvious written down, and the naive version does the opposite by default. Everything lands in today's note, because today is the date the model has closest to hand. The merge timestamp from the API is the only date that's allowed to decide which file gets written.

Resolve the cutoff from the notes, not from the calendar. Each note carries its own sync metadata in frontmatter, including the last date the loop processed. Reading that back is what makes a same-day rerun safe. Take "today" as the cutoff just because today's note exists and you'll skip everything you merged before lunch.

Fail closed. This is the one I'd insist on. To a language model, a query that returned nothing and a query that failed look identical, and both invite the same tidy conclusion: no work today. So the prompt forbids writing "no new pull requests" unless the primary search and an independent cross-check written against a different API both come back empty, plus a backstop that re-scans the whole current week regardless of the incremental cutoff, and unions the results. An automation that reports "nothing happened" on a day you shipped is worse than one that crashes, because a crash gets fixed and a false negative gets believed.

Validate by re-reading. The last step before reporting success reopens every note the run touched and confirms the PR URLs are actually in them, then searches the vault for each URL to check the knowledge base and career entries landed too. Models will report success on writes that never happened. Verification against the vault, not against the model's own account of itself, is the only thing that catches it.

Leave a run record. Every run appends a short entry to a tracker note: the resolved identity it searched as, the cutoff it used, how many candidates each query returned, which notes it wrote, and any fallback path it took. When the output eventually looks wrong, that's what you debug from. Otherwise you're reverse-engineering the behaviour of a non-deterministic process from its output alone, which is as unpleasant as it sounds.

What the weekly pass actually consolidates

The weekly stage is where the pile of daily notes turns into something you'd willingly read, and its most important rule is a prohibition: it is not allowed to query GitHub at all. Daily notes are its only permitted input.

That constraint matters more than it looks. If the weekly pass could re-query pull requests, you'd have two sources of truth immediately drifting apart, and the manual context you carefully typed into Tuesday's note would get outranked by a diff the model finds more legible. The weekly note would slide back into being a list of PRs, which is the thing the whole exercise exists to escape. Each layer trusts the layer below it and nothing reaches back past its own input.

What consolidation actually means here is a transposition. The daily notes are organised by day; the weekly note is organised by kind of work. It loads every daily note in the ISO week, pulls the required sections (the non-PR context and the signal scores) and redistributes everything into four buckets that run across the week rather than down it: key activities, decisions and thinking, collaboration, and operational work. A decision recorded on Tuesday and the follow-through that landed on Thursday become a single thread instead of two entries you'd have to join up yourself. That joining is the entire value. Anyone can concatenate five daily notes.

Then it scores. Each day rates four dimensions (ownership, leadership, scope and execution) from nought to three, separately for PR and non-PR work, with the non-PR side weighted half again as heavy on the grounds that the work that doesn't show up in a diff is usually the work that's hardest to evidence later. The weekly pass sums those per dimension and bands the total: low, medium, or high. Crucially it then has to explain each band in prose, saying why scope was medium and what would have made it high, which forces the number to be defended rather than just displayed.

I treat the scores themselves with suspicion. They're useful as a nudge towards "this was a heavier week than it felt like at the time" and useless as anything resembling a measurement. A score of 10.5 labelled "medium" looks far more precise than it has any right to. What earns its keep is the sentence underneath it.

The weekly prompt also names its own failure modes and tells the model to regenerate if it hits one. The note is invalid if it's PR-only, if it ignores the daily notes, or if it drops manual content. Those are three specific, recurring ways that summarisation goes wrong: reverting to the most machine-legible input, skipping the source entirely, and quietly discarding the half that's harder to compress. Naming them explicitly catches far more than a general instruction to be thorough. The single line that does the most work in the whole file is synthesise, do not list.

Asking the vault questions

The daily loop doesn't stop at the daily note. It also maintains a small set of career artifacts as it goes: a brag document that only accepts high-signal items, a rolling performance review draft, an architecture log tracking systems touched and patterns introduced, and an influence log grouped by domain. All of them use the same marker-block discipline, so they accrete over months instead of being rewritten each run.

But the more interesting property is that none of this is a database. It's plain markdown with frontmatter and wikilinks, sitting in a folder. Which makes the whole vault a corpus you can point a model at and interrogate.

That matters because real performance reviews aren't free-form. You're assessed against a competency framework: a ladder with named criteria and level descriptors, and some notion of what "meets" versus "exceeds" looks like at your grade. So the useful question isn't "what did I do this year". It's the framework itself, handed over verbatim: go through the vault, take each criterion in turn, find the evidence, cite it, and tell me where the evidence is thin.

That's a very different task from summarisation, and the vault is shaped well for it. Criteria like architecture influence, cross-team collaboration or incident ownership map onto material that's already been extracted and tagged rather than buried in prose. The answer comes back per-criterion, with links down to the daily notes and, through them, to the pull requests underneath. And the gaps come back as gaps. An honest "no incident-ownership evidence recorded" is one of the more useful things it can tell you, because it's either a prompt to go and do that work or a prompt to start writing it down when you do.

The signal scores give you a crude ranking layer on top of that. Because the same four dimensions are recorded all the way from daily up through weekly, you can sort periods, spot the dimension that's been persistently thin, and check whether the story you're planning to tell about your year actually matches the trend in your own notes. It's directional, not quantitative, and I'd never put a score in front of a manager. But it's a decent check on self-narrative, which tends to be built out of the three weeks you remember most vividly.

Two caveats worth stating plainly. The vault only knows what you fed it, so a quarter you didn't write down looks exactly like a quarter you didn't work, and the model will happily paper over the difference if you let it. And the whole thing lives or dies on citation. A claim that traces back to a note that traces back to a merged PR is one you can defend in the room. A claim the model produced because it sounded like the right shape of achievement is one you should delete before anyone else reads it.

What's next

The weakest part of all this is exactly the part that matters most: the work that never turned into code. When I haven't typed manual context into a daily note, the job falls back on inferring it from the diff, producing something like "probably coordinated this across a few services". Occasionally that's a fair reading. Just as often it's the model writing plausible fiction about meetings I never had.

What makes that a good problem rather than a depressing one is that the fix isn't more clever prompting. It's more sources. Everything the diff can't see already exists, timestamped and structured, in systems I use all day: my calendar, the meetings themselves, my sent mail, the channels where the arguments actually happen. None of it needs inventing. It needs plumbing in.

Calendar is the obvious first source, and probably the highest value per unit of effort. Outlook exposes it through the Microsoft Graph API, and a day's events are already a structured record of where the time went and who it went with, before anyone summarises anything. Just enumerating them turns "inferred from PR evidence" into something factual. The metadata carries more signal than it looks: attendee lists across team boundaries are a scope indicator you cannot derive from a repository; the organiser field distinguishes meetings you were summoned to from the design review you called, which is close to a direct leadership measurement; recurring events separate standing ceremony from the things you convened deliberately. A daily note that opens with the meetings that genuinely happened is a much better prompt for everything downstream than one that opens with a guess.

Meetings themselves are the richer source, and the one that needs the most discipline. Both Teams and Webex will hand over transcripts and generated summaries through their APIs: Graph for Teams, the Webex REST API for meetings and recordings. The temptation is to drop a transcript into the daily note and let the weekly pass deal with it. That's the wrong shape. A transcript is thousands of words of low-signal conversation wrapped around a handful of load-bearing sentences, and burying those sentences in bulk text just moves the extraction problem downstream to a stage with less context. Far better to ask a narrow question at ingest (what was decided, what did I commit to, what did I argue for and did it land) and write only the answer into the note. Narrow questions against a transcript are a task models are genuinely good at. Open-ended summarisation of one is not.

Sent mail is underrated. The instinct is to point at the inbox, but the inbox is a record of what other people wanted from you. The sent folder is a record of your own output: the thread where you made the case for an approach, the long reply that unblocked someone, the escalation you decided was worth making. Graph will filter it by date and recipient, and the filtering needs to be aggressive. The useful fraction is small, but it's the fraction with your reasoning in it.

Slack, or whichever channel tool you live in, is the noisiest and the most tempting. Scoped to your own messages in public engineering channels, it catches something the other sources miss entirely: the design reasoning that gets written out properly in a thread and then never makes it into a document. Threads where you replied at length are review and unblocking work that leaves no other trace. Reply and reaction counts are a crude proxy for whether an argument actually landed. It's also the source most likely to manufacture significance from someone thinking out loud, so it's the one I'd add last and weight lowest.

The architectural point that ties these together is that each of them should be a source, not a summariser. Every one lands as evidence in the daily note, inside its own marker block, tagged with where it came from. The fail-closed rule extends to all of them too, because a calendar API that errors must never be allowed to mean a day with no meetings. Which leads to the change I'm most interested in: a trust ladder. Something I typed myself outranks something pulled from a structured source like a calendar entry or a pull request, which outranks something extracted from a transcript, which outranks anything inferred from a diff. Right now every one of those lands in the same typeface, which is precisely what makes the inference problem corrosive: it launders a guess into the appearance of a fact. Record the provenance and the review query can prefer the top of the ladder, treat the bottom as a prompt to go and check, and stop pretending the difference doesn't exist.

There's a boundary question that comes with all of this, and it deserves stating rather than discovering later. Transcripts and mail threads contain other people's words, and none of my colleagues signed up to appear in my work log. The rule I'd apply is the same one that already governs where this runs: stay inside approved tooling, extract my own contributions rather than archiving everyone else's, and keep the retained artifact at the level of "a decision was reached about X" rather than a verbatim record of who said what on the way there. That's not only the defensible position, it's the more useful one: the decision is the thing worth having in a review, and the argument that produced it usually isn't.

The smaller items are tractable too. The retrieval flakiness, where queries come back empty on days I'd clearly shipped, traces back to how my identity resolves across accounts, and the honest fix is an explicit identity map resolved once at the start of a run and asserted against, rather than the cross-check currently papering over it. The rollup stops at weekly, and the monthly layer above it is a folder and an intention; that layer gets much easier to justify once daily notes carry meetings and decisions as well as merges, because then there's an arc to see rather than a longer list. And the citation trail that the whole review use case rests on is still only half automated: the links exist from career artifact down to daily note down to pull request, but assembling them into per-claim evidence means walking the chain by hand. Carrying the evidence identifiers along at generation time, rather than reconstructing them a year later, is the change that finishes the loop.

Put together, that's a system where every claim in a review has a source, every source has a provenance, and the honest answer to a criterion I have no evidence for is that I have no evidence for it. That seems like a reasonable thing to build towards.

None of this makes me a better engineer. It just means that when I'm asked what I did last quarter, the answer is sitting in a folder instead of somewhere in my head, and I didn't have to keep a work log to get it there.

On console warnings and the things we don't remove

Scott Mallinson — Mon, 20 Jul 2026 11:40:00 +0000

Not all worthwhile work shows up in a changelog a user would read. Removing unused tooling and quieting a noisy console are two of those jobs. Neither is user-visible. Both are worth doing.

Removing tooling nobody was using

One service in the booking flow depended on a tool for generating API documentation from source comments. The pattern is familiar. Someone set it up, it worked, and at some point the docs stopped being generated or published. The tooling stayed.

This kind of residue is common in long-running codebases. It isn't broken and doesn't throw errors. It just accumulates: in package.json, in install time, in dependency-scan output, in the head of anyone who looks at the project setup and wonders what that config is for. Clearing it out took a small change. Drop the dependency, the config, and the npm scripts, then check nothing in CI was actually using it.

Is it worth a review cycle? I think so. It sets a norm: we remove what we don't use instead of letting it pile up. The review catches the case where someone was relying on it but hadn't said so. And the commit history records why it went, which helps if anyone ever wants to bring it back on purpose.

Defensive checks for prop types

The more interesting change was in a fare search interface. Some components were receiving props that could be undefined or null in edge cases they weren't built to handle. Nothing was breaking and the app kept working, but React was logging warnings to the console about the unexpected prop values. The fix was to handle the missing-or-malformed cases explicitly before passing props in, rather than letting them through and hoping the component survived.

Simple enough on the surface. The more interesting question is why console warnings pile up in the first place.

Console warnings as signal degradation

A clean console is a working signal. When a new warning appears, developers notice it, look into it, and decide whether it's real. When the console is already full of warnings, that signal degrades. New warnings blend in, and the bar for "something is wrong" creeps upward. It happens gradually. One or two warnings in a rarely-hit edge case seem fine, then a few more turn up elsewhere, and before long people are filtering console output by habit while a genuine error sits unnoticed in the noise.

So the real fix isn't "add null checks". It's treating the console as a meaningful output surface and keeping it clean on purpose: defensive prop handling so components don't get values they can't deal with, treating existing warnings as debt worth paying down, and refusing to merge new code that adds warnings. React's prop warnings are telling you something about the contract between a parent component and its children, namely that the parent is sending something the child didn't expect. Ignore them and you're throwing away information about your component interfaces that might matter during a later refactor or upgrade.

The value of maintenance

Neither change shipped a user-visible feature, and that's fine. A codebase that never gets this kind of attention fills up with clutter until it slows you down in measurable ways: noisy CI output, confusing structure, prop mismatches that graduate from warnings to real errors during an upgrade. Small maintenance compounds. So does neglect. You're choosing which.

Two kinds of correctness: currency bugs and ghost feature flags

Scott Mallinson — Mon, 13 Jul 2026 07:05:00 +0000

Some bugs are loud and obvious. Others sit quietly in a codebase doing exactly what the code says, which isn't quite what anyone intended. The quiet ones are the more interesting category, because finding them is closer to archaeology than debugging. A UI banner that wouldn't clear is the same shape of bug on the front end. These are its data-side cousins.

The currency display bug

In travel booking systems, taxes are complicated. There are base fares, carrier-imposed fees, and government taxes, each with its own rules about which currency it should display in. Domestic US travel adds specific airport facility charges and segment taxes with their own currency handling.

The bug was that these taxes showed up with the wrong currency symbol. The amounts were right and the calculation was fine. What wasn't being carried through to the display layer was the currency context. In the exchange flow, where an agent is repricing a ticket change and quoting the tax breakdown to a customer, that's exactly the kind of error that erodes confidence in the tool. The agent can't be sure which values to trust.

Fixing it meant tracing how those tax line items were assembled for rendering and making sure the right currency followed them through the pipeline. The root cause was a missing currency association at the point where the values were gathered into the display model. The fix itself was small once I found it. Tracing it was the bulk of the work, as usual.

The feature flag that didn't exist

The subtler one corrected a feature flag reference in the pricing details service: a flag name that had been sitting in the code but had never actually been created in the feature flagging service.

Every time the code evaluated that flag, it got a "not found" response, which the SDK treated as "off". Whatever the flag was meant to enable was never reachable. It wasn't failing loudly or throwing errors. It was silently defaulting to disabled, no matter what anyone had configured.

Flag names are strings. Nothing checks a flag name at compile time against what's registered in your flagging service. Create a flag under a slightly different name than the one in the code, and the mismatch is invisible, at build time and at runtime, unless you go looking for it. Removing flags that have outlived their purpose is its own discipline. This is the opposite failure: a flag that was never really there.

The passenger type that was never resolved

A third bug of the same quiet kind. The exchange fare search service generates fare options when a traveller changes a booked ticket, and that calculation depends on the passenger type (adult, child, infant, various discount categories) because each attracts different fares and tax treatment.

The fare generation helper was receiving a passenger type code, a short string like ADT or CHD, but it wasn't performing the lookup that translates that code into the full passenger type definition from the reference data service. For standard adult passengers it probably held up, because the code was making assumptions that happened to be true for the common case. For less common categories, the missing lookup produced wrong results without failing loudly. The fix was to resolve every code through the reference data before fare generation runs.

This is the kind of bug that's easy to wave through in review. The code looks plausible, the variable names suggest the right thing is happening, and the tests probably only cover the common cases. The tell is usually spotting that a code path receives something that looks like an identifier and treats it as if it were the thing itself.

Config drift and why it stays hidden

The flag and the passenger type are the same broader problem: drift. The code and the thing it references, a flag registry or a reference dataset, fall out of sync, and because neither side fails in a detectable way, the drift sticks around. The only symptom is behaviour that's quietly wrong, which is easy to blame on something else or miss entirely.

You usually find these by poking around a nearby area and noticing the mismatch. Validating that references resolve at startup is one mitigation. A naming convention and a clear "create it before you reference it" habit is usually more practical. The loud bugs get fixed because they announce themselves. The quiet ones only get fixed if someone goes looking.

The shape of shared libraries

Scott Mallinson — Mon, 06 Jul 2026 09:03:00 +0000

A shared library is defined less by what it does than by the shape it presents to everything that depends on it: its public surface, the contracts it implies, and the cost of changing either. Two bits of work from recently make that concrete. One was an export that was never declared. The other was a small change that quietly broke a numeric assumption.

Making implicit exports explicit

A couple of icon components from a shared plugin library were being used by a downstream service without being part of the library's declared public surface. The consumer reached in by path, something like import X from 'library/internal/path' rather than import X from 'library'.

That works, right up until it doesn't. Importing by internal path creates a hidden dependency on an implementation detail. Reorganise the library, move the internal path, and the consumer breaks. The library's authors have no way of knowing anyone relied on that path, because nothing declared the relationship.

The fix was to add the components to the library's declared exports. It's close to a one-line change, but it earns its keep: it makes the dependency visible to the maintainers, it brings the components under the same deprecation and versioning treatment as the rest of the public API, and it lets static analysis trace the import graph correctly. Libraries with undeclared consumers tend to become hard to refactor, because changes that look safe from the library's side turn out to break things elsewhere. Declaring the export brings the relationship into the open.

A breaking change disguised as a small one

The shared Node logging library is built on Winston, which ships a default set of severity levels (error, warn, info, and so on), each with a numeric priority where lower means more severe and error is level 0. The gap was that nothing sits above error. There's no way to say "this is worse than an error, this is on fire and needs a human now". critical plays that role in syslog and most structured logging conventions, but Winston doesn't include it by default.

Adding a custom critical level at priority 0 and shifting the existing levels up by one fixes that. The change is tiny in lines of code and deceptively large in blast radius. Levels are usually referenced by name, like logger.error(...), which is fine. But any code comparing levels numerically ("only process events with level <= 1") is now pointing at a different level than before, because error has moved from 0 to 1. Every numeric comparison that wasn't explicitly about critical is suddenly off by one.

That's why a change like this ships as a major version bump and asks consumers to update deliberately. The library exposes the new level and documents the shift; downstream services audit their numeric comparisons before upgrading. Writing the level was the easy part. The cost is rolling it through everything that depends on the library.

The shape is the contract

Both of these are the same lesson from different angles. A shared library's public surface is a contract whether or not you've written it down. An undeclared export honours that contract by accident. A numeric level shift changes it without telling anyone. Maintaining a library that lots of things depend on is mostly the work of keeping the contract explicit: declaring what's public, versioning what changes, and making the coordination visible instead of leaving consumers to find out when something breaks.

Three quiet bugs hiding in a cross-service feature

Scott Mallinson — Sun, 28 Jun 2026 08:05:00 +0000

I shipped a feature that looked simple from the outside: you could save a flight quote from an AI-powered assistant into a separate trip planning view. The user-facing part is simple enough. Getting there meant coordinating changes across a fare search service, an AI assistant backend, and two independent frontend components, each with a slightly different idea of what a "saved quote" was.

That kind of work is where a lot of the interesting coordination lives. It isn't algorithmically hard. It's demanding in a different way: you hold a complete picture of how the system behaves in your head while making changes in repositories that share no context with each other.

The silent configuration bug

Before any of the integration work could start, there was a bug to fix. AI assistant queries were returning fewer flight options than expected. No upsell options were coming back at all. Nothing in the system flagged it. Requests completed successfully and responses looked valid. You just got a narrower result set than you should have.

Tracing the fare search payload for AI assistant requests, I found a field that capped the maximum number of upsell results at zero. Zero maximum upsells means return none. It had probably been set when the AI assistant integration was first wired up, and since upsell results aren't always prominent in early testing, nobody had caught it.

The fix was a one-liner. Finding it was the work: tracing the full request chain to understand why AI assistant queries behaved differently from other consumers of the same API. That's how silent configuration drift goes. The value is technically valid, the system doesn't complain, and it quietly shapes what comes back.

When two components share an event

The core integration challenge was making sure that when a quote was saved from the AI assistant, two separate frontend components updated correctly: the assistant itself, to show the saved status, and the trip planning view, to refresh its basket with the new quote.

When I looked at how the trip planning component handled its refresh, it turned out to have its own local action for the job, a separate event doing the same thing as a shared action the AI assistant was already using. The duplication had grown gradually. The trip planning component came first, the AI assistant integration came later, and the shared event either didn't exist yet or wasn't visible when the local version was written.

The fix was to drop the local duplicate and have the trip planning component respond to the shared event directly. Small change, but it compounds. Both components now respond to the same contract. If the event shape changes, it changes once. If you want to know what triggers a basket refresh, there's one place to look instead of two.

I've written before about what adding an AI layer taught me about type ownership, and this was the same principle wearing different clothes. Shared contracts go beyond type definitions. They're about the system having one authoritative source for each concept. Separate copies that start identical will drift apart eventually, and by the time they do, it's rarely obvious which one is right.

Generating HTML carefully

One edge case in the rendering work. The saved-quote feature lets users copy a formatted version of a quote to the clipboard, with PDF export to follow. The copy content is rendered as HTML, so any string values drawn from API responses or user input need sanitising before they're embedded in the template.

It's easy to miss. When you're building a formatter that turns structured data into an HTML string, interpolating values directly feels natural. But if any of those values come from external sources, even indirectly through several layers of typed objects, you've got an injection path. A quote description field containing a <script> tag shouldn't end up executable in a clipboard payload.

The fix was simple: escape HTML entities in any user-visible string before it goes into the template. Not complicated, but it doesn't surface in happy-path tests or feature demos. You have to think about it on purpose, or you find out about it later in a less pleasant way.

What cross-service delivery actually involves

Cross-service feature work is its own skill. The mechanics of any single change are usually straightforward: a field added here, a schema extended there, an event handler updated in a third repository. The hard part is keeping a clear model of how the pieces connect while you move between codebases that share no context.

None of the things that had gone wrong here were individually complex. A suppressed search result, a duplicate event handler, an unsanitised string in a template. Each was a simple thing that had been allowed to exist because nobody had traced the full flow end to end. That tracing is most of what cross-service delivery actually is. You hold the whole picture, notice the gaps, and fix what you find before the feature ships with them baked in.

The scaffolding tax: getting a new service properly bootstrapped

Scott Mallinson — Wed, 24 Jun 2026 15:32:00 +0000

A lot of engineering work doesn't count as "building features" but has to happen before you can build features at any speed. Scaffolding a new service is squarely that: getting it from its initial template state into something actually deployable and maintainable.

From template to real service

We use an internal template to bootstrap new services. It gives you a working skeleton with the right structure, dependencies, and config patterns already in place. The catch comes afterwards. Once you've created a service from the template, there's a round of housekeeping to strip out the template's own identity and replace it with the new service's: config references, package names, internal identifiers, anything still pointing at the template rather than the service.

It's an hour's work and it matters. Leave a stale reference in the wrong place and you get subtle failures downstream. The wrong image gets pulled, metrics report under the wrong service name, alerts go nowhere because the routing rules don't recognise the identifier. Getting it right upfront is cheaper than tracking it down later.

Wiring into the deployment pipeline

A service isn't real until it's in the deployment pipeline. We use GitOps-managed configuration to define what runs in each environment, so adding a service means updating those definitions: registering it, updating the exclusion lists that gate which services deploy where, and making sure the non-production infrastructure knows it exists.

This looks like configuration editing, but it's closer to integration work. You're establishing the contracts between the service and the infrastructure that runs it. Get it in place and deployments become routine. Skip it and every deployment needs manual intervention.

Getting dependency management right from the start

We also set up the service's automated dependency updates properly from day one: finer-grained grouping aligned to upstream release cadences instead of one coarse batch, plus a first pass to clear the initial upgrade backlog before it builds up. The reasoning behind that grouping is worth a post of its own. The point here is that it's far cheaper to establish on a new service than to retrofit onto an old one.

Why scaffolding quality compounds

A service that's wired into the pipeline, has sensible dependency automation, and starts with clean configuration is one where future changes land quickly. One that's bootstrapped in a hurry, with stale references and manual deployment steps, picks up friction with every change. It's dull work. The alternative is treating it as someone else's problem to fix later, which usually means it never gets fixed at all.

Shipping a conversational search flow across services

Scott Mallinson — Sat, 20 Jun 2026 17:43:00 +0000

Some features are self-contained. Others cross enough systems that the changes have to land together, and the coordination becomes the hard part rather than any single change. Shipping an end-to-end conversational flight search was one of those. A user explores options through an AI assistant, selects one, and that selection saves into their booking basket. That single action spans three services and a frontend, and getting it working meant changing all of them roughly in parallel.

Building the flight options flow

The fare search service got improvements to how flight option data is structured in the conversational path. The previous structure had grown organically and was starting to show: handling and transforming it took more case-by-case logic than it should have. This pass cleaned up the data model, with consistent naming, clearer relationships between entities, and less special-casing at the edges, so downstream services and the frontend can work with it predictably.

The trip quote service got a new PATCH /quotes route for saving a selected option and merging it into an existing basket. It's a partial update. You're not replacing the basket, you're folding a selection into it, which is subtler than it sounds: the merge has to cope with a prior selection already existing, with new options conflicting with something already there, and with keeping the basket consistent throughout. What it accepts, what it returns, how it reports errors was most of the interesting work.

The frontend got updated type definitions to match. That sounds mechanical, but if the types don't reflect the actual shape of the data, you lose the compiler's ability to catch mismatches before they reach production.

Instrumenting the MCP server

A separate strand was adding observability to an MCP server, the component that sits between AI tooling and the backend services it calls, translating tool invocations into API calls and structuring the responses on the way back. The instrumentation covers APM tracing, metrics, and structured logging, so you can trace a tool call end to end: how long it took, whether it succeeded, which backend it hit, and where it failed.

The constraint worth flagging is what you don't log. Tool-call requests can carry user-provided context and identifying information, so the instrumentation records the shape and outcome of each call — trace IDs, durations, status codes, error types — without persisting the content. That boundary is a design problem in its own right, and one I've written about separately.

Context injection for AI coding assistants

A different thread again: a set of hooks for our GitHub Copilot configuration that inject context at different points in a development workflow, things like analytics context, test state, feature-flag configuration, and workspace information. Assistants are most useful when they understand the context they're working in. Without it they give generic answers that are technically correct and no use in your actual codebase.

The catch is that injecting too much backfires. Larger context costs tokens, and past a certain point the assistant spends its attention on the context instead of the problem. So the hooks are built around specificity. Each one fires at the moment its context is relevant: pre-chat hooks set up the initial picture, pre-tool-use hooks add context for the operation about to happen, post-tool-use hooks handle the follow-up. Getting it right is empirical. You find where the assistant gives unhelpful answers, work out what context would have helped, and add a hook there.

Naming as a form of maintenance

The feature had picked up two naming conventions as it evolved, one term in some places and another elsewhere. Neither was wrong, but having both meant reading the code required a constant mental translation. A codebase-wide rename pulled everything onto one vocabulary: service names, endpoint paths, function names, test descriptions.

It's the kind of change that's easy to defer, because it doesn't fix a bug or add a feature. The cost of deferring just compounds quietly. Every new developer has to learn the mapping, every review is a little harder, every search has to account for both terms. Doing it once, properly, is cheaper than living with the split.