DEV Community

Kapil Singhal
Kapil Singhal

Posted on

IT Consulting vs a Scoped Pilot: How US/UK Teams Validate a Modernization Plan

Most modernization plans start as a slide deck: move the monolith to services, replace the batch jobs, get off the old database, add CI/CD. The deck is usually reasonable. The problem is that it is a hypothesis, and nobody knows which assumptions are wrong until someone writes code against the real system.

That is the choice many engineering leads in the US, UK, Canada and Australia face after an assessment: commission more IT consulting to refine the plan, or run a small, scoped pilot that tests it. This post is about when each one makes sense, and how to design a pilot that actually answers the questions the roadmap raised.

What each option is good at

IT consulting (architecture reviews, assessments, roadmaps) is good at breadth. A good consultant interviews the teams, reads the code and the incident history, maps dependencies and produces options with trade-offs. You need this when you don't yet know what the problems are.

A scoped pilot is good at depth. A small team takes one slice of the plan and builds it against your real systems, with written acceptance criteria and a fixed scope. You need this when you have a plan but don't know if it survives contact with production.

Question you're trying to answer Better tool
What are our biggest technical risks? Consulting / assessment
Which of three architectures should we pick? Consulting, then a pilot to test the favourite
Will the strangler pattern work on our billing module? Pilot
How long will migrating one service really take? Pilot
Can our deploy pipeline support independent releases? Pilot
Does the team understand the target architecture? Pilot (pair with them)

A rough rule: if the open questions start with "what", consult. If they start with "will it" or "how long", build something.

Signs your roadmap needs a pilot, not another workshop

  • The roadmap has an effort estimate for the whole programme, but nobody has migrated even one component yet.
  • The assessment lists "unknowns" like data quality, hidden coupling or undocumented integrations.
  • Two senior engineers disagree about the approach, and both arguments sound plausible.
  • Leadership wants a date, and the honest answer is "we'd be guessing".
  • You've had two rounds of review and the document is getting longer, not more certain.

At that point, more analysis tends to produce more confident-sounding guesses. A few weeks of real work produces evidence.

How to pick the pilot slice

The slice should be small enough to finish and representative enough that the result tells you something about the rest of the plan. We use three filters:

  1. It touches the hard part. If the risk is the shared database, the pilot must move a table or a read path, not build a greenfield service next to it. A pilot that avoids the risky area proves nothing.
  2. It ends in production, or as close as you can get. Behind a feature flag, on a shadow path, or for one internal team. "Works in a dev environment" leaves most questions open.
  3. It is reversible. You should be able to switch back to the old path in minutes if something goes wrong.

Good examples:

  • Extract one bounded context (say, notifications) behind an interface, route a percentage of traffic to the new path, and compare outputs.
  • Replace one nightly batch job with an event-driven consumer for a single event type, running both in parallel.
  • Move one service onto the new CI/CD pipeline and container platform end to end, including rollback.
  • Put one legacy endpoint behind an API gateway and add contract tests before any rewrite.

Bad examples: "set up the new platform", "migrate the user service and the order service", or anything with "phase 1 of 4" in the name.

Write the acceptance criteria before the code

A pilot without written success criteria turns into a demo. Agree these up front, in the repo, as a short PILOT.md:

## Pilot: extract notifications service

### Hypothesis
Notifications can be extracted without changes to the order flow,
and the new path can handle production traffic.

### In scope
- Notifications interface in the monolith
- New service + queue consumer
- Feature flag routing 0-100%
- Dashboards for latency, error rate, queue depth

### Out of scope
- Email template redesign
- Other bounded contexts

### Done when
- [ ] New path serves 100% of traffic for 7 days behind the flag
- [ ] Error rate and p95 latency no worse than the old path
- [ ] Rollback tested and timed
- [ ] Runbook and ADR merged
- [ ] One in-house engineer has deployed it unaided
Enter fullscreen mode Exit fullscreen mode

Notice what's on the list: rollback, a runbook, an architecture decision record and an in-house engineer deploying it. Those are what tell you whether the approach scales to the rest of the roadmap.

What to measure besides "did it work"

The deliverable is only half the result. The other half is what the pilot teaches you about the plan:

  • Estimate error. How far off was the original estimate for this slice? Apply that factor to the roadmap.
  • Unknowns found. List every surprise: hidden coupling, missing tests, data quirks. Each one probably appears elsewhere too.
  • Pipeline friction. How long from merge to production? How many manual steps?
  • Team readiness. Could your engineers explain and change the new code without the pilot team?

Feed these back into the roadmap before the next decision. Often the plan stays the same and only the sequencing changes. Sometimes the pilot kills an approach early, before you've committed to it for a year.

When to go back to consulting

Pilots aren't always the answer. Go back to assessment work if the pilot reveals that the problem is organisational (ownership, team boundaries, on-call) rather than technical, or if it shows the target architecture itself is wrong and you need to re-evaluate options. A good engagement moves between the two: assess, test the riskiest assumption, update the plan, repeat.

Summary

  • Use consulting to find out what to change; use a pilot to find out whether and how long.
  • Pick a slice that touches the hard part, reaches production and can be rolled back.
  • Write PILOT.md with in-scope, out-of-scope and "done when" before anyone writes code.
  • Measure estimate error, surprises, pipeline friction and team readiness, and update the roadmap with them.

How do you validate a modernization plan before committing to it? I'd like to hear what's worked, and what hasn't, in your teams.


Kapil Singhal, Director, Axentis Technologies

Axentis website: https://axentis.tech | Launch Pilot: https://axentis.tech/launch-pilot/

Aranyix (Axentis product): https://aranyix.tech

Axentis on LinkedIn: https://www.linkedin.com/company/axentis-technologies | Aranyix on LinkedIn: https://www.linkedin.com/showcase/aranyix/

Top comments (0)