DEV Community

OneLifeVibe
OneLifeVibe

Posted on

When the Incident Spans Two Clouds, the Timeline Is the Hard Part: Incident Response for Multi-Cloud Networks

The hardest part of a multi-cloud incident is rarely fixing the fault. It is working out, while the fault is still happening, what actually happened, in what order, and on which side of a boundary between clouds.

Inside one cloud, an incident has a rough shared reality. Logs use similar timestamps, metrics live in one place, and the team can usually agree on when things started. Add a second cloud and that shared reality splits. Each provider reports time, health, and errors in its own way. A request crosses a private link, a load balancer, and an identity service, and leaves fragments in three different systems. Reconstructing the story from those fragments, under pressure, is a design problem that should be solved before the incident, not during it.

Give every crossing a name the logs can share

A timeline can only be built if the pieces can be joined. For traffic that stays inside one service, a request ID is enough. For traffic that crosses clouds, the ID has to survive the crossing.

Pass a single correlation ID from the entry point through every hop, including asynchronous work started by the request. Log it on both sides of every cross-cloud call, in the same field name, so a search in either system finds the same journey. Where a managed service cannot carry a custom header, record the provider request ID alongside the correlation ID at the boundary, so the two can still be linked later.

This sounds basic, and it is the step most often missing. Without it, responders end up matching incidents by approximate time and guesswork, which works for a slow degradation and fails completely for a short spike that affected only some requests.

Keep the ID boring and consistent. One format, one header name, one log field, everywhere. Novelty in identifiers is a cost paid during the worst hour of the month.

Build one timeline, not two provider timelines

During an incident, each cloud console tells a plausible story on its own. Latency rose here. Errors rose there. Neither story, alone, says which came first, and cause and effect are easy to reverse when the stories are read separately.

Maintain a single incident timeline outside both consoles. In practice that can be a shared document or an incident tool, but the rule matters more than the tool: every entry gets a timestamp in one time zone, a source, and a short statement of what was observed, not what was concluded. Observations first, conclusions later, marked as conclusions.

Agree the time zone in advance. Coordinated Universal Time is the usual choice because provider logs, network devices, and managed service events do not all follow local time, and daylight saving changes have confused more than one post-incident review. Whatever the team chooses, dashboards, log queries, and the incident timeline should all use it during an incident, so nobody is converting times in their head while also trying to fix the fault.

Clock accuracy helps, but alignment of presentation helps more. Even with well synchronised clocks, a metric that aggregates in one minute buckets in one cloud and five minute buckets in another will appear to start at different times. Note the aggregation window next to any chart used in the timeline, or responders will argue about a five minute gap that is only a reporting artefact.

Decide ownership at the boundary before there is a fault

Cross-cloud incidents stall at boundaries. The team that owns the application sees errors. The team that owns the network sees a healthy link. The provider support team sees no fault on its side. All three can be correct, and the incident still has no owner for the space between them.

Name that owner in advance. For each cross-cloud path that matters, record who owns the path end to end, who can make changes to routing or traffic during an incident, and who is called when the fault is in neither cloud clearly. That record should be short, current, and findable in under a minute. A detailed architecture document that nobody can locate during an incident does not count.

The same applies to status communication. Decide who updates stakeholders, how often, and in what channel, before the incident. When the person fixing the fault is also the person being asked for updates every ten minutes, both jobs suffer.

Preserve evidence while fixing the fault

The fastest fix often destroys the evidence needed to prevent a repeat. Restarting a service clears its local logs. Changing a route removes the configuration that caused the problem. Scaling up hides the capacity limit that was reached.

Build preservation into the response, not as bureaucracy, but as a few quick habits. Export the relevant log window before restarting where that is practical. Save the effective routing and policy configuration as it was, not as it was intended to be. Note the exact time of each mitigation, so its effect can be separated from the original fault in the timeline.

This is also where readiness work pays off. If network changes, access changes, and configuration versions are already recorded as part of normal operations, responders do not have to reconstruct them from memory. Evidence collected routinely is calmer and more complete than evidence collected in a hurry.

Practise the joined up response, not only the parts

Most teams practise failover or recovery for a single service. Fewer practise the coordination layer: opening the shared timeline, joining logs across clouds with the correlation ID, contacting the boundary owner, and making a traffic decision based on incomplete information.

A short tabletop exercise exposes the gaps cheaply. Take a realistic scenario, for example intermittent errors on requests that cross from one cloud to a database in another, with healthy checks on both sides. Walk through who looks where first, what query they run, and what would convince the team to shift traffic, roll back a change, or wait. The exercise usually reveals missing log fields, unclear ownership, and dashboards that show each cloud separately but never the path between them. Those are fixable on a normal weekday.

After a real incident, update the same artefacts the exercise tests: the path ownership list, the log fields, the timeline template, and the runbook for that path. A review that produces only a narrative, with no change to those artefacts, leaves the next incident just as hard to reconstruct.

A short checklist for the next cross-cloud incident

  1. One correlation ID carried across every cloud boundary, logged under the same field name on both sides.
  2. One incident timeline, in one time zone, recording observations before conclusions.
  3. Aggregation windows noted on every chart used to judge start times.
  4. A named end to end owner for each important cross-cloud path, findable in under a minute.
  5. A named person for stakeholder updates, separate from the person fixing the fault where possible.
  6. Log windows and effective configurations preserved before restarts and route changes where practical.
  7. A tabletop exercise for a boundary fault, repeated after the ownership or paths change.

None of this makes a multi-cloud incident simple. It makes it legible. When responders can see one journey across both clouds, agree on the order of events, and know who owns the space between providers, the fix itself usually becomes the straightforward part. Teams that build that visibility into their private connectivity and operations, rather than assembling it during an outage, respond faster and learn more from each incident. That approach to consistent cross-cloud paths and shared visibility is described at cloudverg.com, where VergMesh and VergShield are used to keep multi-cloud estates observable and controlled as one estate.

The next incident that spans two clouds will still be stressful. It does not have to be confusing as well.

Top comments (0)