Google Cloud has published a customer-facing playbook for how to respond when its own services fall over. The workflow is a five-step loop, Verify, Investigate, Report, Resolve, Review, wrapped in a Prepare phase that runs before any of it. For on-call teams the operational read is simple. The vendor now has an opinion about which pane of glass you look at first, and the answer is Personalized Service Health.
The loop, at a glance
Verify that something is actually broken, then figure out who broke it: Google, you, or a third party. Investigate through the vendor's telemetry and your own change history. Report through the console at P1 or P2. Resolve, or at least communicate and fail over while Google works its side. Review with a blameless post-mortem, including the vendor's own incident report for major outages. The Prepare phase sits before all five and does the unglamorous work: automated response actions, replicated observability, a written playbook, drills.
What Prepare actually asks for
The list reads like a standard SRE prep checklist, which is the point of publishing it.
- Automated response actions on critical applications, designed in and reviewed for coverage.
- Observability data replicated to a redundant stack in a separate location from the systems being observed, with time stamps synced across streams.
- A written playbook that says out loud who does what, who to notify, how to reach them, which tools and data are available, how results are communicated, and how the pager hands over during long-running incidents.
- Simulated cross-team incident drills several times per year, each followed by a retrospective.
None of this is new. What is useful is having a vendor page you can point at when you ask for the budget to run the drill.
Verify: which console first
The guidance is explicit about the order. Check Personalized Service Health first. It surfaces Emerging Incidents, where Google has received an alert and on-callers are still investigating, and Confirmed Incidents, where customer impact has been established. Gemini Cloud Assist is integrated into the same view. The public Cloud Service Health dashboard is described as the fallback for broad, severe incidents. Known Issues live under Support then Cases, filtered by a resource selector in the console toolbar.
The skeptic's note: this is a vendor telling you to look at its own console first. That is not unreasonable when the question is "did they break it," but it does mean the first honest signal in an incident lives behind an authenticated Google Cloud UI.
Investigate: what to check if Google has not declared
If Personalized Service Health is silent, the playbook sends you back through three surfaces. Cloud Monitoring for spikes in 5xx error rates, elevated latency or traffic drops. Log Explorer for specific error strings like DEADLINE_EXCEEDED, SERVICE_UNAVAILABLE, or your own API errors. Quotas, in case you have hit a project ceiling on CPU or an API rate limit. Change history through Unified Maintenance Management in Cloud Hub is called out separately. The recommendation, when symptoms start right after a rollout, is to back out to the last known good configuration first and diagnose second.
Report and resolve
P1 is defined as a production service that is unusable or severely impacted with no workaround. P2 is significant impact or degradation with a possible workaround. A case needs project ID, affected region or zone, timestamps with an explicit timezone, error messages or log snippets, and the scope of impact. Premium and Enhanced Support customers get an Escalate button on the case in the console.
While waiting, the guidance is the usual short list: tell your stakeholders, fail over to a healthy region if you have one and if the disruption is at the infrastructure and not the workload level, watch the status dashboards for temporary workarounds, and mind your regulatory reporting deadlines.
Review: blameless, and read theirs too
Post-mortem with the four familiar questions: what went well, what could we have done better, where did we get lucky, where did we get unlucky. Google publishes a post-mortem or Incident Report for major outages, delivered through Personalized Service Health. Premium Support customers can request an Incident Summary for a Google-caused incident they were impacted by, provided they opened a P1 case.
The residual caveat
The playbook is honest about where its center of gravity sits. It assumes your first read of an incident happens inside Google's consoles. If your primary observability lives elsewhere, you still need a rehearsed path into the vendor's status surfaces, because "did they break it or did we" is answered there first. The Prepare list is the honest test of this document. Anyone can run a checklist during an incident. The teams that keep the playbook current are the ones that had a retrospective on a drill this year.
Top comments (0)