Imagine reaching the comfortable part of a system design interview. You have drawn the API, chosen the database, and explained how the service handles more traffic. There is even a sensible reason for the cache.
Then the interviewer adds one condition: the system already exists, and you need to change it without breaking the customers using yesterday’s version.
Suddenly, a clean diagram is missing something important. It shows where you want to arrive, but says very little about the uncomfortable days when half the system has arrived and the other half has not.
That is the follow up I want in my preparation: How do we get from the current system to this design safely?
You can rehearse it without building an entire distributed system. You can take one small change, work through the versions that must coexist, and decide exactly when going back stops being simple.
Small change with a surprisingly large audience
Here is our practice scenario. A profile service stores a customer’s display name in a database column called display_name. A new contract calls the same value profile_name. The meaning has not changed; the name has.
Assume the rename is required by an agreed contract. In a real project, I would first challenge whether the benefit justified a migration at all.
The web app can deploy today. Some mobile clients will stay on an older version for weeks. A nightly import also updates profiles. All database writes go through service code we control, and both columns will live in the same transactional database during the transition.
These assumptions matter. If external jobs write directly to the table, or the new data lives in a separate database, this plan needs additional coordination.
The tempting answer is to rename the column, update the service, and deploy the new client. But the older mobile client still expects the old response field. An old server process can still be running during deployment. The import might be using a different release.
Migration has several clocks, even when the code change takes five minutes.
Start with who can still use the old contract
Netflix’s recent engineering post on the blast radius of library changes makes a useful distinction: detecting an API change and finding its consumers are different jobs. Its internal tool, Quasar, connects changed symbols with observed usages across repositories. The team also explains why an index with no observed references is not proof that no consumers exist.
My interview takeaway is simple like you describe the evidence behind your migration plan. Exercise below is my own small example, not Netflix’s architecture or a reported Netflix interview question.
Before drawing a new service, write down the consumers and writers:
- Web client: who can release it, and can the new behavior stay disabled initially?
- Mobile client: which supported versions still need the old API response?
- Profile service: can old and new instances overlap during deployment?
- Import worker: does its write path receive the same compatibility update?
A repository search helps find the worker. Request telemetry helps reveal old client versions. Neither proves complete coverage on its own, so record what you have not checked.
Draw the awkward middle, before the attractive ending
I prefer to sketch a tiny compatibility grid before deciding the deployment order.
An old client with an old server should continue working. An old client with the compatibility release should also work. A new client with that compatibility release should work. A new client reaching an old server is the combination we must prevent or explicitly support.
For this exercise, we prevent that last combination: deploy the compatibility release everywhere before enabling the new client contract. The new web release can exist with its new path switched off until that gate is satisfied.
This is more precise than saying the release is backward compatible. It names the combinations that are supported and the mechanism that keeps an unsupported combination out of production.
It also gives you a concrete answer when the interviewer asks what happens if deployments finish in the wrong order.
Rollout is part of the design
Our compatibility release acts as a bridge. It understands both API field names, preserves the old response contract, and can keep both database columns consistent. The old field remains the source of truth initially.
Add the new field without removing the old one
Add a nullable profile_name column. Keep the existing display_name column and behavior. Check the database’s actual migration and locking behavior before choosing how to apply this change on a large table.
The important property is that the old application can still operate with the expanded schema.
Upgrade every writer before trusting the new field
Deploy the bridge release to the profile service and import worker. For each name update, it writes both fields in the same database transaction. If a request contains both names with different values, reject it rather than silently choosing one.
During the rolling deployment, some old processes may still update only display_name. Therefore, keep reads on the old source of truth. Do not start treating the new field as authoritative just because the first bridge instance is healthy.
The gate is stronger: every active writer uses the bridge, older instances have drained, and scheduled writers have been accounted for. Then transactionally reconcile any differing non-null values using the still authoritative old field, under the same row locking discipline as live updates.
Backfill historical rows without undoing live edits
Copy missing names into the new column in bounded batches. Track progress so an interrupted run can resume, and monitor its effect on the database.
This dangerous implementation reads a row, holds the value in a worker, and writes it back later without checking whether anything changed.
Suppose the worker reads Maya. The customer changes the name to Mina, and the bridge correctly updates both columns. The worker then writes its old copy into profile_name. Both operations succeeded, but the fields now disagree.
For our single database exercise, read and fill the missing value atomically under the row’s transaction lock. Recheck that the destination is still missing; do not overwrite a populated field using a stale value held by a worker. The live update path must use the same database’s concurrency guarantees.
If live editing finishes first, the backfill finds nothing to fill. If the backfill finishes first, the later live update writes Mina to both fields. Explain both orderings rather than relying on the word atomic.
Switch reads only after the evidence is useful
Compare the fields before changing the read path. A missing new value, a disagreement, or an unaccounted-for writer is a reason to investigate.
Then enable new reads for a controlled cohort while retaining the bridge. Track that cohort separately so errors are not hidden inside healthy traffic from everyone else.
For this exercise, I can start with internal accounts, expand to a small stable set of users, and increase exposure after checks pass. The percentages and observation period belong to the risk and traffic pattern; they are not universal magic numbers.
Keep old API responses available for supported old clients. Switching an internal database read does not require every mobile customer to update that afternoon.
Rollback answer needs a specific destination
Saying we can roll back leaves a difficult question unanswered like roll back to which release, using which data?
While both fields are maintained, we can disable the new read path and return to the bridge behavior. Both columns still contain the current name. The recovery destination is a known compatible release, not any old binary someone finds in the deployment history.
If we return all the way to an older writer that updates only one field, the columns can diverge again. That needs reconciliation before another attempt to switch.
Eventually, we may stop maintaining the old column and later remove it. That is a separate decision. A binary that still queries the deleted column can no longer run, and reverting application code cannot recreate deleted values.
Before retiring the old path, account for supported clients, workers, old deployment versions, and the agreed recovery window. If an old mobile API must remain supported, keep an adapter that serves its response from the new storage.
Compatibility can stay at the boundary even after the internal schema changes.
Write the stop conditions before the launch
A dashboard is useful when it can change the decision. For this rehearsal, keep the checks close to the customer’s experience:
- Correctness: missing or conflicting names in a completed migration batch block progression until explained.
- Compatibility: exercise the supported client/server combinations, including the import worker and an older mobile response.
- Operational health: compare errors, latency, and database pressure with a baseline; define acceptable limits before exposure increases.
- Recovery: name the person or process that can disable the new path, and rehearse the return to the bridge release.
Our copy operation should preserve names exactly. A recommendation model migration would need different correctness checks. Do not carry one scenario’s acceptance rules into another without examining the meaning of the data.
Try the follow up on a question that is harder to hand-wave
Once the profile example feels clear, use monolithic wallet migration question. It is tagged Apple, and it asks for a phased migration with data ownership, consistency, and rollout decisions.
Start with one capability. Keep the rest of the wallet stable. Then add the uncomfortable follow up i.e. the new service has accepted writes, and now we want to send traffic back.
Where are those writes?
Can the old path see them?
What prevents two services from both becoming authoritative?
The easy rollback answer gets much less convincing when it cannot account for the latest customer action.
Use the question to produce a short migration note before reading further guidance:
- Name the current source of truth and every writer that can change it.
- Draw the old/new combinations that must coexist during release.
- Describe how historical data and live updates meet without losing either.
- Choose an observable gate for each increase in exposure.
- Identify the last reversible step and the recovery plan after it.
If one line is vague, that is the part to practice next. Another finished diagram will not answer it for you.
Answer I want to be able to defend
I prefer a migration plan that admits its awkward middle: we retain the old contract, upgrade every writer, reconcile and backfill the data, switch reads gradually, and retire the old path only when the evidence supports it.
That gives an interviewer something concrete to challenge. It also gives you a way to keep reasoning when a new requirement appears, instead of restarting the drawing.
Before your next practice session, take a system you already know and change one thing about it. Keep yesterday’s client alive while you do it.
Can you explain how the customer’s latest action survives every step?
Top comments (1)
Drawing the awkward middle before the attractive ending is the habit I need most, because my first instinct is still to rename the column and move on. Your Maya to Mina backfill example is the detail that sticks with me: a stale worker overwriting only profile_name while a live edit updates both fields is exactly how two sources of truth diverge quietly. When you gate on every writer using the bridge, how do you prove the older instances have actually drained before you switch reads?