DEV Community: Scott Mallinson

Release pipelines should be boring

Scott Mallinson — Tue, 16 Jun 2026 13:26:00 +0000

Release automation tends to get written in a hopeful mood. One pull request merges, one job runs, one tag gets created, everything lands cleanly. You assume only one thing happens at a time. The assumption isn't deliberate. It's just how you think when you're writing the happy path.

The trouble is that the happy path is a special case. As soon as a repository has busy automated dependency updates, several pull requests can merge within minutes of each other, and each merge fires its own release job. The jobs start from roughly the same point and then race each other to write back. We had a shared GitHub Actions template running releases across a set of repositories, and it turned out to be hiding three separate race conditions, each at a different stage of the same job.

Two jobs creating the same tag

The first race is at tag creation. Two jobs start at almost the same moment, both read the current version, both compute the next one, and both try to create the same version tag. One wins. The other fails with a tag conflict.

The fix is a concurrency group on the workflow. GitHub Actions supports this natively: you name a group, and a run that would join it while another is in flight either waits or gets cancelled. For releases you want it to wait. You're serialising, not skipping.

concurrency:
  group: release-${{ github.ref }}
  cancel-in-progress: false

With that in place, two release jobs in the same repo queue instead of colliding. The second runs cleanly once the first is done.

Two jobs pushing to the same branch

Serialising tag creation doesn't cover the last step: pushing the version-bump commit back to the branch. A job checks out the repo, bumps the version, commits, and pushes. If another job pushed in the gap between checkout and push, the working copy is now a commit behind, and git correctly refuses the non-fast-forward push.

There are two halves to fixing this. The first is to sync with the remote at the last possible moment before writing, so the bump lands on a current view of the branch:

- name: Sync with remote before versioning
  run: git pull --rebase origin ${{ github.ref_name }}

- name: Bump version
  run: npm version patch --no-git-tag-version

The ordering matters. Pull and rebase before the bump, so you're rebasing onto current remote state rather than dragging your version commit over changes that might conflict with it.

The second half is to make the push itself resilient. Even with a pre-push sync, two jobs can reach the push inside the same narrow window. So the push step retries: on a non-fast-forward rejection it pulls, rebases the bump onto the new tip, and tries again, bounded so it fails loudly instead of looping forever.

- name: Push Changes
  run: |
    for attempt in 1 2 3; do
      if git push origin HEAD:${{ github.ref_name }}; then
        break
      fi
      if [$attempt -lt 3]; then
        echo "Push failed, pulling and retrying..."
        git pull --rebase origin ${{ github.ref_name }}
      else
        echo "Push failed after $attempt attempts"
        exit 1
      fi
    done

Note that you recover after a failed attempt rather than pulling before every one. Failing first and then recovering avoids unnecessary work, and it avoids a window where a pre-emptive pull could rebase onto a conflicting state.

Cleaning up after a race that already happened

By the time the fixes land, a race can already have left a mess. A job writes a version-bump commit locally but fails to push it, so the version recorded in the package manifest is a step ahead of the tags actually published. Recovering means going through the CI logs to find the run that partially completed, working out what it left behind, and replaying that specific bump cleanly against the current branch. The archaeology takes longer than the fix.

That's the real cost of partial failures in automation. The failure itself is cheap. Reconstructing the state it leaves behind is what costs you. So you make release steps idempotent where you can: a step that's safe to re-run without doubling its effects turns a tense recovery into a re-run.

Why a flaky pipeline is worse than it looks

A release job that fails about half the time sits in an awkward blind spot. It's not bad enough to block anyone. You retry, it passes, you move on. The workaround is cheaper than the fix, so the failure gets normalised, and people stop reading it as a signal and start treating it as weather.

The cumulative cost is real, and it isn't only the wasted retries. An unreliable pipeline changes how people work. If every release risks a babysitting session, the rational move is to batch changes up, which is the opposite of the small, frequent pull requests that make review easier and rollbacks cheaper. A pipeline that runs quietly on every merge takes that disincentive away, and goes back to being something nobody has to think about.

Doing this in a shared template multiplies the payoff. The fixes land once, and every repository that inherits the template gets them without anyone rediscovering the problem repo by repo, one retry button at a time.

Granular Dependabot groups and getting error attribution right

Scott Mallinson — Sat, 13 Jun 2026 20:58:37 +0000

Dependabot has a default behaviour that doesn't scale well: one pull request per outdated dependency. For a repository with a hundred dependencies, a run generates dozens of individual PRs. Most are low-risk patch bumps a developer approves without reading closely — which means either they pile up unreviewed, or people start rubber-stamping them, which defeats the point of the review.

Why grouping matters

Dependency grouping addresses this. Instead of one PR per package, you define groups — "all AWS SDK packages", "all testing libraries", "everything from this upstream source" — and Dependabot combines the relevant updates into a single PR. The result is far fewer PRs, each giving a complete picture of a set of related changes.

The work was moving several repositories from ungrouped or coarsely grouped configs to ones with more granular, purposeful groups. The granularity matters: a group defined as "all dependencies" is barely better than no grouping — you still get one huge PR — while groups built around logical cohesion ("packages released together from the same upstream source") give you something you can actually review with confidence.

Applying it across different stacks

The interesting part of rolling this out is that each repo has a different dependency structure. A Node logging library has nothing in common with a .NET error-handling library or a React frontend. For the Node libraries, grouping around major upstream sources makes sense; for the .NET library, the groups follow NuGet namespaces; for the frontend, it's a mix of framework packages, tooling, and application libraries, each with their own natural groupings. You can copy the structure of the YAML across repos, but you still have to think about what actually belongs together in each — and that thinking is the part you can't automate away.

Error source attribution in a shared library

The other change was about correctness in error reporting. A shared .NET library classifies errors and warnings across several services — capturing not just the message but its origin: which part of the system produced it. The origin is an enum, and one category was missing: a group of trip-related services whose errors weren't covered by any existing value. When those services produced an error, it either failed to classify or fell into a catch-all, making it harder to route the alert and slower to find the responsible team.

Adding the value is straightforward. The more interesting question is why it was missing. This is a common pattern with shared classification types — the enum gets defined early, before all the consumers are known, and then isn't kept in sync as new services adopt the library. The fix doesn't just make reporting more accurate; it removes the ambiguity that wastes on-call time. "The origin is unknown" means more digging before anyone can act; "the origin is the trip services layer" means the right team gets paged immediately.

Platform hardening as a steady-state activity

This is worth naming for what it is: platform hardening. Not new features, not architecture changes, but the continuous work of making the infrastructure more reliable, maintainable, and legible to the people who depend on it. A Dependabot config sits in a file almost nobody reads until dependency updates go wrong; an error-source enum is invisible to end users. Both keep a system that dozens of engineers work in daily manageable over time. The return is long-tailed and largely invisible, which is exactly why it tends to get deprioritised — and doing it consistently, even when nothing more exciting is on the board, is how a platform stays manageable.

What adding an AI layer taught me about type ownership

Scott Mallinson — Tue, 09 Jun 2026 20:12:33 +0000

I've been working on an AI-powered trip planning assistant that sits on top of an existing set of booking microservices. The AI layer is genuinely interesting work — natural language input, iterative trip refinement, the whole thing. But the most valuable engineering work had nothing to do with the AI itself.

It was about types. Specifically, who owns them and what happens when they drift.

The problem with local schemas

The AI assistant service had its own validation schema for incoming flight search requests. It wasn't wrong, exactly — it reflected the shape of requests the service expected at the time it was written. But the canonical definition of what a valid flight search request looks like lives in the fare search service, and over time, subtle differences had crept in.

This is a fairly standard distributed systems problem. When two services independently define what they think the same thing looks like, they'll stay in sync right up until they don't. A field gets added somewhere. A constraint gets tightened. Someone updates one schema and not the other. Nothing breaks immediately — the tests pass, the service starts — but you've created a time bomb.

The fix was straightforward: pull the shared type out of the fare search package and use that directly in the AI assistant, removing the local definition entirely. One source of truth. When the API evolves, every consumer stays aligned automatically.
Simple to describe. Surprisingly easy to defer.

Why AI features make this urgent

Here's the thing about building an AI layer on top of existing services: it doesn't introduce new complexity so much as it surfaces the existing ambiguity.

A traditional integration between two services fails fast. Service A sends a request to Service B, B returns an error, A logs it, someone gets paged. The feedback loop is tight enough that schema drift tends to get caught reasonably early.

An AI assistant is different. The user is expressing intent in natural language. The assistant is interpreting that intent, deciding what to query, constructing requests, handling the responses. There are more layers of abstraction between the user's words and the actual API call. When something goes wrong, it might manifest as a confusing or unhelpful response rather than a clear error — which means it can go unnoticed for longer.

More importantly, the AI layer is making decisions about how to construct requests. If its understanding of what a valid request looks like is subtly out of sync with reality, those decisions will be subtly wrong. Not catastrophically wrong — just wrong enough to be annoying and hard to diagnose.

This is why schema consolidation that might have felt like a nice-to-have became genuinely urgent once an AI layer was involved. The model compounds every ambiguity downstream.

Session state is harder than it looks

The other significant change this week was enforcing a required session-tracking header throughout the conversation service. This one is less about types and more about correctness guarantees.

The header was already being passed in some code paths. The problem was "some" — in a service where every request needs to carry session context to maintain coherent conversation state, optional isn't good enough. A user iterating on a trip in natural language needs the system to remember where they are in the conversation. If that header gets dropped mid-flow, the session context is gone and the experience breaks in a way that's confusing rather than obvious.

Making it a validated requirement — something the service explicitly checks for and rejects requests without — is the kind of change that feels bureaucratic until the alternative happens in production.

The lesson isn't "validate everything". It's more specific: identify the invariants your system actually depends on and make them impossible to violate rather than relying on every caller to get it right.

If you're planning to add an AI layer on top of an existing set of services, the time to audit your type ownership is before you do it, not after.

It's not that the AI makes the type problems worse, exactly. It's that it makes them more consequential and harder to spot. A messy schema definition that was fine when the integration was service-to-service becomes a genuine liability when a model is making decisions based on it.

The boring foundational work — shared types, enforced invariants, single sources of truth — isn't glamorous. But it's what determines whether the interesting work on top of it actually holds up.

The quiet work of removing feature flags

Scott Mallinson — Fri, 29 May 2026 14:23:43 +0000

Feature flags are one of those tools that everyone agrees are useful right up until the point where they're not being used for anything — they're just there, silently complicating the codebase.

This was a quiet week in terms of new features. Two pull requests, both doing the same kind of thing: removing flags that had outlived their purpose. It's the sort of work that doesn't generate much excitement but, done consistently, makes a meaningful difference to how a codebase feels to work in.

What feature flags are actually for

The case for feature flags is well-understood. You decouple your deployment from your release — the code goes out, but the behaviour is gated behind a flag that you can flip remotely. This gives you the ability to run gradual rollouts, do A/B experiments, or kill a feature instantly without touching production. For a platform processing complex travel bookings, with lots of moving parts and no tolerance for downtime, that kind of control is genuinely valuable.

The problem is that flags don't clean themselves up. Once a feature has fully shipped, once the experiment has concluded and the "enabled" behaviour is the only behaviour you want, the flag becomes noise. It's a conditional in the code with only one meaningful branch. It's a name in a config file that nobody remembers adding. It's a test that exercises a code path that no longer exists in production.

Why this work is worth doing deliberately

The risk with flag sprawl isn't any one flag in particular. It's the accumulation. Once you have a dozen old flags in a codebase, each with their own conditional logic and test coverage, a new engineer reading the code has no easy way to know which flags are live, which have been retired, and which are permanent configuration that will never change. The code looks more complex than it actually is.

There's also a subtler problem: old flags can interact with new flags. Two conditionals that were each simple on their own can combine into behaviour that nobody anticipated and that no test covers. Flag cleanup isn't just tidiness — it reduces the combinatorial surface of the system.

The best time to remove a flag is shortly after the feature it was protecting has fully shipped and proven stable. The second best time is now. Keeping a habit of retiring flags when they're done means the codebase stays navigable, and the next engineer who needs to understand how the functionality works doesn't have to trace through a conditional that was only ever going one way.

Sometimes the most useful thing you can ship in a week is less code than you started with.