Ace Interviews

Posted on Oct 22 Originally published at github.com

10 Google SRE NALSD Prompts (System Design): Interview Scenarios for 2027+ Loops

#sre #devops #career #systemdesign

🔮 The 2027+ Google SRE NALSD Playbook: 10 Operational Frontier Simulations

"If you think System Design is about drawing a load balancer and a database, you are preparing for a 2022 interview."

Hyperscale infrastructure has shifted. The era of "Design a URL shortener" is dead. Modern Site Reliability Engineering at Google, Meta, and AWS requires sizing infrastructure for Agentic AI loops, power constraints, and post-quantum network fragmentation.

In modern Non-Abstract Large System Design (NALSD) rounds, drawing a global architecture without doing the physical bandwidth and IOPS math is an instant L3 (Junior) down-level. Hiring committees are now testing Operational Physics.

Below are 10 live interview simulations of the exact scenarios dominating senior hiring loops right now. To demonstrate how the Hiring Committee evaluates candidates, we have open-sourced the first 3 phases (The Math & Feasibility Checks) of these scenarios.

📑 Table of Contents

Part 1: The Computational & Data Core

Global LLM Inference Platform (KV-Cache Sizing, Prefill/Decode Disaggregation)
Multi-Tenant AI Accelerator Fleet (GPU Bin-Packing, N+1 Pod Loss)
Global RAG & Vector Retrieval Platform (10B Vectors, HNSW Index RAM vs. NVMe)
Token-Aware AI API Rate Limiter (Two-Phase Reservation, Overshoot Bounds)
Global Distributed Lock Service (Sharded Paxos, Speed-of-Light Latency Floors)

Part 2: The Operational Frontiers

Agentic “Infinite Loop” DDoS (Semantic Shedding, Queue Deadlines)
Multi-Modal Live Edge Streaming (Consistent-Hash GPU Pinning, Reconnect Buffers)
Post-Quantum Cryptography Fleet Rollout (Certificate Bloat, MTU Fragmentation)
Datacenter Power & Thermal Shedding (3 MW Breaker Trip, Hardware-Aware Suspension)
Sovereign Cloud Data Telemetry Sync (Air-Gapped Store-and-Forward, Aggregation)

🏗️ NALSD Scenario Simulation 1: Google SRE NALSD Round Mock Interview (Global LLM Inference Platform)

Why include this: This is our strongest flagship NALSD scenario for this entire Google SRE bundle. Google's public AI infrastructure direction makes it a credible forward-looking practice problem, although the exact interview prompt is unverified. Senior-level Google SRE-style NALSD mock interview for 2027+ preparation: global LLM inference capacity planning, token throughput, KV-cache sizing, accelerator provisioning, regional failure reserves, and infrastructure BOM.

The math that matters: Input/output token throughput, concurrent streams, accelerator throughput, KV-cache footprint, prefill/decode capacity, and regional failure reserves.

Interview simulation · Senior infrastructure engineering · Capacity sizing and BOM

We will treat this as a live, interviewer-led NALSD round. The candidate will reason aloud, calculate resource requirements, justify architectural decisions, and respond to follow-up questions as they arise.

One distinction upfront: this is a realistic simulated interview, not a transcript of an actual Google interview. The prompt and constraints are supplied; accelerator throughput, model configuration details, and hardware prices are not. Where those missing inputs affect the BOM, the candidate will state assumptions rather than invent benchmark results.

The objective is to demonstrate how a strong candidate approaches the problem in the interview—not to recite a prewritten architecture.

Phase 1 — Clarify the problem (Minutes 0–5)

INTERVIEWER

"Design a globally distributed LLM inference platform with hierarchical KV-cache management for Gemini-class models. We need 100,000 QPS globally, 4,000 input tokens on average, 500 output tokens, P99 TTFT under 300 milliseconds, P99 TPOT under 25 milliseconds, and 99.95% availability. How would you design it?"

CANDIDATE

"I would start by establishing the workload and the resource constraints. At this scale, the critical question is not just how many requests we can accept, but how much prefill computation, decode throughput, accelerator memory, and network capacity those requests consume.

I have three clarifications before sizing the system.

First, is 100,000 QPS the peak aggregate across all regions, and should I assume that traffic can be unevenly distributed?

Second, are the input and output token counts averages for initial and subsequent chat turns combined, or should I model separate workload classes?

Third, should I assume a particular accelerator and benchmarked model throughput, or can I use explicit planning assumptions and derive a conditional BOM?"

INTERVIEWER — SIMULATED RESPONSE

"Treat 100,000 QPS as peak global traffic. Traffic is uneven, and multi-turn sessions reuse prefixes. Use a representative accelerator configuration, but state your assumptions. We expect regional failure tolerance."

Candidate's interpretation: I will design for peak traffic rather than average daily traffic. I will distinguish logical conversation history from reusable KV-cache state. I will also avoid assuming that all requests consume the same amount of accelerator time.

The SLOs are requirements to validate, not performance guarantees that can be claimed before sizing and testing.

Phase 2 — Workload math and the first capacity boundary (Minutes 5–12)

The first calculations establish the amount of work entering the platform.

1. Input and output token throughput

Given:

Peak request rate: 100,000 requests/second.
Average input: 4,000 tokens/request.
Average output: 500 tokens/request.

Input-token rate:

100,000 × 4,000 = 400,000,000

That is 400 million input tokens per second before accounting for cache hits.

Output-token rate:

100,000 × 500 = 50,000,000

That is 50 million generated tokens per second.

Input throughput	Output throughput
400M tokens/second	50M tokens/second

These are logical workload rates before prefix-cache savings, not measured accelerator throughput.

2. Concurrent generation streams

If we assume an average request generates 500 tokens at an average of 25 milliseconds per token, the estimated generation duration is:

500 × 0.025 = 12.5 seconds

Using Little's Law, the approximate number of concurrent generation streams is:

100,000 × 12.5 = 1,250,000

So the initial planning estimate is 1.25 million concurrent generation streams.

But I would explicitly tell the interviewer that this estimate assumes the stated 25 milliseconds is representative of average token-generation time. The actual requirement is a P99 TPOT bound, so the average cannot be derived from the P99 alone. We need a measured or separately specified average to finalize concurrency.

3. The first interviewer challenge

INTERVIEWER

"You have calculated 1.25 million streams. Does that mean you need 1.25 million independent accelerator allocations?"

CANDIDATE

"No. That is the concurrency across the serving fleet, not the number of accelerators. Continuous batching allows an accelerator to serve multiple sequences concurrently, subject to memory capacity, memory bandwidth, and the latency target.

I need measured per-accelerator throughput and a validated batch configuration to translate these streams into an accelerator count. I will calculate memory demand next, because the KV cache can limit the feasible concurrency even when compute capacity appears sufficient."

Phase 3 — KV-cache sizing: the first major physical constraint (Minutes 12–20)

Candidate's whiteboard calculation

"I'll use an explicit example KV-cache configuration so the estimate is reproducible. Assume 64 transformer layers, 8 KV heads, a head dimension of 128, and one byte per KV value. These are illustrative assumptions, not a claim about Gemini's internal architecture."

For each token, the cache holds both keys and values:

KV_bytes = 2 × L × H_KV × D × B

= 2 × 64 × 8 × 128 × 1

= 131,072 bytes

That is 128 KiB per token before implementation-specific overhead.

For an active request containing approximately 4,250 tokens:

4,250 × 131,072 = 557,056,000 bytes

That is approximately 557 MB per request under this particular configuration.

This number is large, but it follows directly from the stated assumptions. It is not a universal KV-cache footprint.

Estimate aggregate cache demand

Using the illustrative concurrency estimate of 1.25 million active requests:

KV_total = 1.25 × 10⁶ × 557.056 MB

≈ 696 PB

Illustrative uncached aggregate KV footprint
≈ 696 PB

This is a workload-derived theoretical footprint, not a proposed HBM allocation.

The estimate assumes the example KV format and dimensions apply to all requests, all estimated token positions remain resident, and prefix sharing provides no reduction. Actual cache residency can differ substantially.

Interviewer challenge

INTERVIEWER

"You have just calculated hundreds of petabytes of KV state. Are you proposing to buy that much accelerator memory?"

CANDIDATE

"No. The calculation exposes why we cannot retain every request's full KV state in accelerator HBM. It also tells me that I must validate the workload assumptions before committing to the hardware budget.

I would separate three things: active decode state that must be immediately accessible, reusable prefix state that can be cached, and conversation history that can be persisted and recomputed when necessary.

I would also verify the actual model's KV dimensions and precision. If the interviewer expects a feasible production BOM, I need benchmarked per-accelerator throughput and a realistic cache-residency target. Otherwise, the aggregate figure is a useful upper-bound scenario, not a purchase order."

(Interviewer Note: This is an important NALSD moment. The candidate has used math to expose a physical constraint instead of jumping to a familiar architecture).

The interviewer pushes further

INTERVIEWER

"Fine. The platform must still support multi-turn conversations. How will you manage the cache, how many accelerator nodes do you need across prefill and decode, and what happens to our latency budget when a primary region goes dark?"

🔒 End of Preview: Sizing the Architecture

You have reached the end of the public preview (Phases 1–3).

Most candidates fail NALSD interviews at this exact moment: they freeze when faced with physical hardware ceilings like the 696 PB KV-cache footprint, or they throw arbitrary GPU counts at the board without deriving capacity. Passing an L5/L6 Google SRE round requires using mathematical constraints to dictate the architectural boundary before placing a single server.

🗺️ What’s Inside the Full Walkthrough: Phases 4 through 6

In the complete interview session included in our bundle, the candidate steps through the rest of the 45-minute loop:

Phase 4 — Compute, Accelerator, and Network Sizing (Minutes 20–30)
- Quantized Model Footprint: Sharding a 70B parameter model across 8-accelerator hosts (640 GB aggregate physical HBM) and calculating real usable capacity.
- Throughput Sizing Formulas: Deriving exact prefill (N_prefill) and decode (N_decode) node counts at target utilization (u = 60%).
- Network Fabric Math: Calculating the 1.6 Gbps+ outbound payload rate and isolating the prefill-to-decode KV transfer bandwidth to prevent congestion-driven tail latency.
Phase 5 — Architecture Derived from Sizing Constraints (Minutes 30–37)
- End-to-End Request Path: Designing the global routing layer, admission control, and prefix-aware scheduling across prefill and decode pools.
- Hierarchical Cache Retention: Engineering the HBM → Host DRAM → Durable conversation history storage pipeline.
- Regional Failover Math: Sizing surviving regions for the exact +33.3% load shift during an N-1 failure without exceeding TTFT/TPOT latency bounds.
Phase 6 — Final BOM, SLO Validation, and Conclusion (Minutes 37–45)
- The Complete Production BOM: Sizing criteria across accelerator hosts, HBM, host DRAM, regional gateways, and the control plane.
- Tail-Latency Validation Strategy: Empirical testing plans for P99 TTFT, inter-token generation latency distributions, and retry amplification safeguards.
- Scoring Breakdown: The exact rubrics Google interviewers use to distinguish an L5 (Senior) from an L6 (Staff) candidate.

🚀 Get the Full Walkthrough & Complete Interview Bundle

This scenario is just one part of the end-to-end curriculum designed to build your operational reflexes under real interview pressure.

👉 Unlock the Full Walkthrough in the Google SRE Interview Career Launchpad

What is inside the complete system:

Full-Length NALSD & Troubleshooting Transcripts: The complete, unabridged walkthroughs, mathematical proofs, and architectural blueprints for this multi-tenant accelerator fleet and other flagship scenarios.
20+ "War Room" Interview Simulations: Realistic mock scenarios equipped with the hidden Interviewer Scorecards, follow-up traps, and rubric criteria.
Linux Internals & Reliability Deep Dives: Practical drills covering D-state processes, TCP connection backlogs, kernel memory pressure, and distributed consensus.
70+ Production-Grade Coding Drills: Systems-focused Python & Go drills emphasizing concurrency, streaming, backpressure, and defensive design.
The Offer Maximizer: Negotiation frameworks and word-for-word scripts to secure top-of-band L5/L6 compensation packages.

Stop guessing the math under pressure. Master the simulation.

🏗️ Google SRE NALSD Scenario Simulation 2: Multi-Tenant AI Accelerator Fleet Capacity & Scheduling

Why include this: This scenario targets the core reality of enterprise AI infrastructure: managing a mixed, multi-tenant fleet of 2,000 heterogeneous workloads. Rather than assuming homogeneous single-model serving, this interview stresses multi-accelerator topology constraints, bin-packing fragmentation, gang scheduling, and pod-level correlated failure domains.

The math that matters: Accelerator-equivalent demand aggregation, compound utilization and bin-packing efficiency factors (0.70 × 0.85), N+1 pod failure derivation, physical host counts, and per-accelerator working memory limits.

Interview simulation · Senior infrastructure engineering · Fleet capacity, bin packing, and N+1 resilience

One distinction upfront: this is a realistic simulated interview, not a transcript of an actual Google interview. Hardware performance metrics, pod geometries, and utilization ceilings are explicit planning assumptions. The objective is to demonstrate how a strong candidate establishes capacity envelopes and prevents correlated outage cascades—not to guess vendor specs.

Phase 1 — Define the problem and workload dimensions

INTERVIEWER

"Design a shared accelerator fleet serving 2,000 enterprise AI workloads. Each workload has different throughput, latency, memory, and priority requirements. Calculate the accelerator count, memory capacity, scheduling overhead, and spare capacity needed to survive the loss of an entire accelerator pod."

CANDIDATE

"Before choosing the number of accelerators, I need to establish the workload distribution, the performance of each accelerator configuration, the memory footprint of each workload, and the failure domain.

Two thousand workloads do not necessarily mean two thousand simultaneously active workloads. Some may be continuously serving inference traffic, some may execute periodic batch jobs, and others may require accelerators only during training windows.

I will separate the problem into five dimensions:

Demand: How much accelerator compute does each workload class require?

Memory fit: Can the model, working state, and runtime fit on the available accelerator configuration?

Scheduling: How efficiently can heterogeneous workloads be packed onto hosts?

Latency and isolation: Can high-priority workloads meet their service-level objectives under contention?

Resilience: Can the fleet continue serving its required workload after losing an entire accelerator pod?

I will use explicit exercise assumptions first, then calculate the required capacity. Hardware performance figures will be assumptions for this design, not claims about any particular cloud provider's production fleet."

INTERVIEWER

"Good. Start by defining the fleet and its workload."

Phase 2 — Workload demand aggregation

INTERVIEWER

"You have 2,000 workloads. Give me a concrete demand model."

CANDIDATE

"I will divide the fleet into four workload classes. These are exercise assumptions that represent different resource profiles."

Workload class	Workloads	Peak accelerator demand per workload	Aggregate peak demand
Small inference	1,200	0.10 accelerator-equivalent	120
Medium inference	600	0.50 accelerator-equivalent	300
Large inference / fine-tuning	160	2.00 accelerator-equivalents	320
Large training / high-throughput AI	40	4.00 accelerator-equivalents	160
Total	2,000		900

"An accelerator-equivalent represents the compute capacity of one reference accelerator under our baseline benchmark assumptions. Fractional values represent aggregated demand across time-sliced or batched workloads; they do not mean a physical GPU can be divided arbitrarily without scheduling and memory isolation consequences.

The fleet's aggregate peak demand is:"

D = 120 + 300 + 320 + 160 = 900

"Therefore, the fleet must deliver 900 reference accelerator-equivalents of compute during the modeled peak."

INTERVIEWER

"Why not simply provision 900 accelerators?"

CANDIDATE

"Because 900 is the modeled compute demand, not the installed fleet size. We still need to account for operational utilization limits, fragmented capacity caused by bin packing, and the loss of an entire accelerator pod.

I also need to distinguish average demand from peak demand. If the workloads have synchronized peaks, aggregate demand approaches 900 accelerator-equivalents. If peaks are statistically independent, multiplexing may reduce the capacity required—but only after measuring demand distributions and acceptable queueing latency.

For this initial sizing, I will provision against the stated aggregate peak rather than assume that statistical multiplexing will work in our favor."

Phase 3 — Hardware baseline and capacity loss factors

INTERVIEWER

"What hardware are you designing around?"

CANDIDATE

"I will use a reference host containing eight accelerators, each with 80 GB of accelerator memory.

For the capacity calculation, I will apply these explicit assumptions:"

Parameter	Exercise assumption
Accelerators per host	8
Memory per accelerator	80 GB
Accelerators per pod	64 (8 hosts)
Operational utilization ceiling	70%
Effective bin-packing efficiency	85%
Target failure domain	1 complete pod (64 accelerators)

INTERVIEWER

"Explain the difference between those last two percentages."

CANDIDATE

"The utilization ceiling (70%) limits how heavily we plan to load capacity that can be allocated, leaving headroom for traffic bursts and latency-sensitive scheduling.

The bin-packing efficiency (85%) accounts for capacity that exists physically but cannot be assigned because of discrete workload shapes, memory limits, or topology placement fragmentation.

Because they describe completely different losses, I apply them multiplicatively to find the effective usable capacity:"

C_effective = N × 0.70 × 0.85 = N × 0.595

Phase 4 — Sizing the N+1 resilient fleet

INTERVIEWER

"How many accelerators do you need to survive losing one pod?"

CANDIDATE

"The surviving fleet must deliver at least 900 effective accelerator-equivalents after losing one complete 64-accelerator pod.

If the fleet has N installed accelerators, the surviving capacity is:"

N_surviving = N - 64

"Applying our combined efficiency factor (0.595):"

(N - 64) × 0.70 × 0.85 ≥ 900

(N - 64) × 0.595 ≥ 900

N - 64 ≥ 900 / 0.595 ≈ 1,512.61

N ≥ 1,576.61 accelerators

"Because we must provision full 64-accelerator pods, we round up to whole pods:"

N_pods = ⌈1,576.61 / 64⌉ = 25 pods

N = 25 × 64 = 1,600 accelerators

Installed Pods	Installed Accelerators	Surviving Accelerators (N-1 Pod)	Effective Post-Failure Capacity	Surplus Headroom
25	1,600	1,536	913.92 equiv	+13.92 equiv (+1.55%)

"After losing a 64-accelerator pod, 1,536 × 0.595 = 913.92 effective accelerators remain, clearing our 900-unit demand by 13.92 accelerator-equivalents.

However, a 1.55% surplus margin is razor-thin. I would not call this design production-ready without validating placement fragmentation, startup times, and rack-level topology."

Phase 5 — Translating into hosts and memory topology

INTERVIEWER

"Convert that accelerator count into physical hosts and memory capacity."

CANDIDATE

"At 8 accelerators per host, we need:"

N_hosts = 1,600 / 8 = 200 hosts

"The raw fleet memory is:"

M_fleet = 1,600 × 80 GB = 128,000 GB = 128 TB

M_surviving = 1,536 × 80 GB = 122.88 TB

Workload class	Working memory demand	Placement requirement
Small inference	16 GB	1 accelerator (fractional compute)
Medium inference	40 GB	1 accelerator
Large inference / fine-tuning	70 GB	1 accelerator
Large training / high-throughput AI	140 GB	2 accelerators (model-parallel sharding)

INTERVIEWER

"Does 128 TB of fleet memory prove that all 2,000 workloads will fit?"

CANDIDATE

"No. Aggregate memory capacity is an illusion.

A workload requiring 140 GB cannot fit on a single 80 GB device. It requires two accelerators that must reside on the same high-speed PCIe or NVLink domain. If the cluster has 200 GB of free memory scattered as 25 GB fragments across eight different machines, that 140 GB job cannot schedule.

Furthermore, for the 40 large training jobs requiring 4 accelerators each, we cannot allocate them piecemeal. We must enforce gang scheduling: all required accelerators, host memory, and fabric ports must be acquired simultaneously. Otherwise, partially allocated jobs will idle on scarce GPUs, creating cluster-wide resource deadlocks."

(Interviewer Note: This is an exceptional NALSD answer. The candidate immediately identifies that compute and memory cannot be treated as scalar pools without considering physical topology and deadlock risks).

The interviewer pushes further

INTERVIEWER

"You've proven the physical limits of bin packing and memory topology. Now, how do you actually design the scheduling system to orchestrate 2,000 competing workloads? How do you isolate latency-sensitive inference from batch training, manage control plane overhead, and orchestrate recovery without triggering a cold-start storm when an entire pod goes dark?"

🔒 End of Preview: Capacity Sizing & Topology Constraints

You have reached the end of the public preview (Phases 1–5).

Most candidates fail this interview by treating GPUs like interchangeable CPU cores—ignoring interconnect topologies, fragmentation losses, and correlated restart storms. In an L5/L6 Google SRE round, proving the physical failure boundary is only the first half of the problem.

🗺️ What’s Inside the Full Walkthrough: Phases 6 through 10

In the complete interview session included in our bundle, the candidate steps through the remaining 25 minutes of the mock loop:

Phase 6 — Heterogeneous Scheduler Architecture (Minutes 20–28)
- Control Plane vs. Data Plane Disaggregation: Designing the admission control, placement engine, and distributed agent runtime.
- Workload Isolation & Preemption: Implementing priority tiers, bounded queues, and checkpoint-aware preemption to protect latency-sensitive inference from batch monopolization.
- Multi-Dimensional Constraint Matching: Sizing placement engines that evaluate compute, memory, and topology (NVLink/RoCE) simultaneously.
Phase 7 — Control Plane Sizing & Queueing Dynamics (Minutes 28–34)
- Throughput Sizing: Sizing the scheduler for an initial burst (λ_submission ≈ 3.33 req/s) and steady-state lifecycle events (λ_events = 100 req/s).
- Latency Breakdown: Separating scheduling decision latency from resource queueing latency under heavy cluster contention.
- High-Availability Control Plane: Deploying a quorum-based 3-replica active scheduler fabric across distinct failure domains.
Phase 8 — Correlated Pod Failure & Cold-Start Storm Recovery (Minutes 34–39)
- Correlated Failure Mechanics: Handling the simultaneous loss of 8 hosts, 64 accelerators, and 5.12 TB of resident memory.
- Avoiding Recovery Cascades: Why instantaneous restarts trigger secondary failure storms through simultaneous model weights pulls and checkpoint reads.
- Staged Recovery Pipelines: Rate-limited warm-up queues, priority-tier restoration, and admission throttling.
Phase 9 — Final Production BOM & Infrastructure Sizing (Minutes 39–43)
- The Complete Infrastructure Footprint: Sizing 25 pods (1,600 accelerators, 200 hosts), high-bandwidth network fabric, and checkpoint storage throughput.
- Four-Step Validation Roadmap: Benchmark verification, simulated bin-packing audits, pod-loss drills, and thermal/power density planning.
Phase 10 — Final Interviewer Challenge & Scoring Rubric (Minutes 43–45)
- The Single Greatest Risk: Unpacking the danger of treating scalar capacity as fungible capacity.
- Official Interviewer Scorecard: The exact rubric Google interviewers use to grade workload sizing, failure domain awareness, and operational realism.

🚀 Get the Full Walkthrough & Complete Interview Bundle

This scenario is just one part of the end-to-end curriculum designed to build your operational reflexes under real interview pressure.

👉 Unlock the Full Walkthrough in the Google SRE Interview Career Launchpad

Stop guessing the math under pressure. Master the simulation.

🏗️ Google SRE NALSD Scenario Simulation 3: Global RAG and Vector Retrieval Platform

Why include this: Vector retrieval at the 10-billion scale is the foundational backbone of enterprise Generative AI and RAG architectures. This scenario tests whether a candidate understands that vector databases are memory-bandwidth-bound and scatter-gather-bound systems, rather than conventional disk-backed relational stores.

The math that matters: FP16 vs. FP32 representation, HNSW graph index overheads (+30%), metadata footprint, shard partition sizing, scatter-gather query amplification (naive 20M QPS vs. routed 1M QPS), network candidate bandwidth, and regional N+1 capacity sizing.

Interview simulation · Senior infrastructure engineering · Vector search, memory residency, and global latency

One distinction upfront: this is a realistic simulated interview, not a transcript of an actual Google interview. Data representations, graph overheads, and network margins are explicit planning assumptions. The objective is to demonstrate how a strong candidate establishes capacity envelopes and mitigates tail latency—not to guess vendor specs.

Phase 1 — Define the problem and operational scale

INTERVIEWER

"Design a globally distributed retrieval platform containing 10 billion embedding vectors, each with 1,536 dimensions. Serve 100,000 queries per second with P99 retrieval latency below 100 milliseconds. Calculate RAM, index overhead, storage, replication, network bandwidth, and the final Bill of Materials."

CANDIDATE

"Before choosing a vector database or deciding how many servers to provision, I need to establish the data representation, retrieval algorithm, query distribution, and geographic failure requirements.

Ten billion vectors is a massive dataset, but the vector count alone does not determine the infrastructure requirement. The result depends on bytes per dimension, index overhead, the number of candidates examined per query, the degree of memory residency, and whether retrieval requires one or multiple stages.

I will establish a quantitative model using these assumptions:

The dataset contains 10 billion vectors, each with 1,536 dimensions.

The global retrieval service must sustain 100,000 queries per second.

End-to-end retrieval latency must remain below 100 milliseconds at P99.

The primary index uses approximate nearest-neighbor search (ANN), with optional exact reranking.

The platform uses geographically distributed query-serving capacity and replicated data.

Capacity calculations will separate vector payload, index structures, metadata, replication, and operational headroom.

I will calculate the storage footprint first, derive the shard layout, size query throughput and network traffic, and then determine the compute capacity and final BOM."

INTERVIEWER

"Good. Start with raw storage. Do not choose a server count yet."

Phase 2 — Calculate raw vector storage: FP32 vs. FP16

INTERVIEWER

"Each vector has 1,536 dimensions. How much space do 10 billion vectors require?"

CANDIDATE

"The first decision is numeric precision. If every dimension uses standard 32-bit floating-point (FP32), each vector requires:"

1,536 × 4 bytes = 6,144 bytes

"For 10 billion vectors, that requires:"

10¹⁰ × 6,144 = 61.44 × 10¹² bytes ≈ 61.44 TB (55.9 TiB)

"For a more cost-effective and memory-bandwidth-efficient design, I will evaluate 16-bit floating-point (FP16) storage, provided offline recall and ranking-quality benchmarks show negligible degradation relative to FP32.

The FP16 vector size is:"

1,536 × 2 bytes = 3,072 bytes/vector

"The total raw payload becomes:"

10¹⁰ × 3,072 = 30.72 × 10¹² bytes = 30.72 TB

INTERVIEWER

"Does that mean the entire platform requires only 30.72 TB of RAM?"

CANDIDATE

"No. That number covers only the vector payload. The platform also needs the search index graph, vector identifiers, metadata, runtime buffers, and operational headroom.

I also need to distinguish storage capacity from query-serving capacity. A system may keep its searchable index in RAM while storing original vectors or document content on SSD. Alternatively, a fully memory-resident design may be justified if predictable P99 latency is more important than minimizing memory cost.

I will initially evaluate a memory-resident search index because the service has a strict P99 latency target of 100 ms. I will validate that decision with representative retrieval benchmarks rather than assume that all data must always reside in RAM."

Phase 3 — Index overhead, graph links, and metadata sizing

INTERVIEWER

"What index would you use, and how much additional memory does it require?"

CANDIDATE

"I would evaluate an HNSW-style (Hierarchical Navigable Small World) graph index because it provides an excellent recall-versus-latency curve for approximate nearest-neighbor search. However, the final algorithm depends on the required recall, update rate, ingestion pattern, and memory budget.

For the initial capacity model, I will use these explicit assumptions:"

Component	Assumption
Vector representation	FP16 (3,072 bytes/vector)
Raw vector payload	30.72 TB
Index structures and graph links	30% of raw vector payload
Identifiers and metadata	128 bytes per vector
Operational headroom	25%
Logical replication factor	3 copies

"The 30% index overhead is an illustrative planning assumption, representing graph connectivity (e.g., M = 16 or 32 links per node), implementation overhead, and indexing buffers.

The index memory estimate is:"

M_index = 30.72 TB × 0.30 = 9.216 TB

"The metadata estimate is:"

M_metadata = 10¹⁰ × 128 bytes = 1.28 × 10¹² bytes = 1.28 TB

"Therefore, the estimated working set for one complete dataset copy is:"

S_copy = 30.72 + 9.216 + 1.28 = 41.216 TB

Raw Vector Payload	HNSW Index Overhead	Metadata & IDs	Single-Copy Working Set
30.72 TB	9.216 TB	1.28 TB	41.216 TB

"This estimate excludes original raw document bodies, full-text inverted indexes, write-ahead logs, and external reranking models, which belong to separate downstream microservices."

Phase 4 — Replicated storage and memory residency targets

INTERVIEWER

"Calculate the storage requirement for three copies."

CANDIDATE

"With a baseline logical replication factor of 3:"

S_replicated = 41.216 TB × 3 = 123.648 TB

"Adding 25% operational headroom to account for memory fragmentation, runtime growth, and background index rebuilding:"

S_planned = 123.648 TB × 1.25 = 154.56 TB

INTERVIEWER

"How much RAM do you need?"

CANDIDATE

"If every replica keeps its vector payload, graph index, and metadata resident in memory, the aggregate usable memory target across all three copies is 154.56 TB.

This is usable memory, not total installed physical RAM. Physical servers require additional memory for OS kernel page caches, network socket buffers, query aggregations, and process overhead."

INTERVIEWER

"Is full memory residency always the right answer?"

CANDIDATE

"No. A hybrid architecture (such as DiskANN) can keep the high-level navigation graph in RAM while fetching the actual FP16 vectors from high-throughput NVMe SSDs via parallel asynchronous I/O.

That dramatically reduces RAM spend, but SSD read amplification and tail-latency jitter can jeopardize our strict P99 target of 100 ms. I will evaluate the memory-resident baseline first to guarantee tail latency, then benchmark hybrid NVMe offloading as an optimization."

Phase 5 — Shard layout and partitioning

INTERVIEWER

"Ten billion vectors cannot fit on one machine. How would you partition the dataset?"

CANDIDATE

"I will partition the dataset into 200 logical primary shards, each holding 50 million vectors:"

N_shards = 10,000,000,000 / 50,000,000 = 200 shards

"Let's calculate the working set per shard:

Raw vector payload per shard: 50,000,000 × 3,072 bytes = 153.6 GB

Index overhead per shard (30%): 153.6 × 0.30 = 46.08 GB

Metadata per shard: 50,000,000 × 128 bytes = 6.4 GB

The total working set per logical shard is:"

S_shard = 153.6 + 46.08 + 6.4 = 206.08 GB

"Across three copies, the cluster hosts 600 shard replicas (200 × 3)."

Primary Shards	Vectors / Shard	Working Set / Shard	Total Replicas (3×)	Cluster Working Set
200	50M	206.08 GB	600	123.65 TB

INTERVIEWER

"Does that mean every user query must visit all 200 primary shards?"

CANDIDATE

"Not necessarily. If every incoming query fans out to all 200 shards, the system suffers severe scatter-gather request amplification.

100,000 queries per second fanning out to 200 shards creates 20,000,000 internal shard searches per second. That creates massive CPU contention and network congestion.

Instead, I would introduce a two-tier routing architecture using a coarse centroid index (e.g., IVF clustering) or tenant/metadata partitioning to direct queries only to candidate shards.

For our initial model, I will assume queries route to an average of 10 candidate shards. However, this introduces an immediate tradeoff: aggressive pruning reduces query load, but poor routing drops relevant neighbors and degrades recall."

(Interviewer Note: This is a critical NALSD insight. The candidate connects database partitioning directly to network request amplification and retrieval recall).

The interviewer pushes further

INTERVIEWER

"You've highlighted the fan-out danger. Now prove how the network fabric handles candidate merging and reranking, show how tail latency behaves across parallel shard fan-outs, and size the compute fleet to survive the loss of an entire cloud region without breaching our 100 ms P99 SLO."

🔒 End of Preview: Vector Footprint & Shard Partitioning

You have reached the end of the public preview (Phases 1–5).

Most candidates fail this interview by treating vector search like a standard database—forgetting that scatter-gather tail latencies compound exponentially (P = pⁿ), and underestimating the network fabric needed to merge millions of candidate vectors per second.

🗺️ What’s Inside the Full Walkthrough: Phases 6 through 10

In the complete interview session included in our bundle, the candidate steps through the remaining 25 minutes of the mock loop:

Phase 6 — Query Throughput & Scatter-Gather Amplification (Minutes 18–24)
- Fan-Out Math: Contrasting naive broadcast (20M shard searches/s) with 10-way routed retrieval (1M shard searches/s).
- Per-Shard Load Balancing: Sizing each logical shard for 5,000 searches/s while handling embedding clustering hot spots.
Phase 7 — Network Bandwidth & Candidate Merge Sizing (Minutes 24–29)
- Ingress & Egress Traffic: Sizing the 400 MB/s user ingress against the 4 GB/s (32 Gbps) internal fan-out request volume.
- Candidate Merge Payload: Sizing 100M candidates/sec yielding 6.4 GB/s (51.2 Gbps) of candidate-response serialization across the coordinator layer.
Phase 8 — P99 Tail Latency Budget & The Multi-Shard Fan-Out Penalty (Minutes 29–34)
- End-to-End Latency Slicing: Allocating budgets across ingress (10ms), embedding (15ms), parallel search (40ms), rerank (15ms), and delivery (10ms).
- The Tail Probability Trap: Proving why independent 99th-percentile shards compound to only a 90.4% success rate over a 10-way fan-out (0.99¹⁰ ≈ 0.904), and designing hedging/speculative retry strategies.
Phase 9 — Multi-Region Replication & Disaster Recovery Math (Minutes 34–38)
- Regional Failover Load: Proving why surviving regions in a 3-region cluster must absorb a +50% load surge during a full regional outage.
- Rebuild Mechanics: Deriving the network transfer floor (55 minutes at 100 Gbps line rate) and preventing foreground query starvation during recovery.
Phase 10 — Compute Fleet Sizing & Final Bill of Materials (Minutes 38–45)
- Memory-Bound vs. Compute-Bound: Determining whether 500 throughput-sized hosts or 600 memory-placement hosts govern cluster sizing.
- The N+1 Regional Sizing: Scaling the fleet to 900 query-serving hosts (230.4 TB installed RAM) to survive a complete regional catastrophe.
- Official Interviewer Scorecard: The exact grading matrix Google interviewers use to evaluate vector database capacity, tail-latency resilience, and distributed system trade-offs.

🚀 Get the Full Walkthrough & Complete Interview Bundle

This scenario is just one part of the end-to-end curriculum designed to build your operational reflexes under real interview pressure.

👉 Unlock the Full Walkthrough in the Google SRE Interview Career Launchpad

Stop guessing the math under pressure. Master the simulation.

🏗️ Google SRE NALSD Scenario Simulation 4: Global Token-Aware AI API Rate Limiter

Why include this: Standard rate limiters count simple scalar requests (HTTP QPS). In modern GenAI platforms, request count is meaningless—a 100-token prompt and a 50,000-token multi-document prompt consume vastly different amounts of compute and financial budget. This scenario tests distributed admission control, two-phase token reservations, and cross-region coordination without synchronous WAN bottlenecks.

The math that matters: Weighted average token consumption (2,850 tokens/req), 14.25 billion tokens/sec peak accounting rate, in-memory tenant state vs. durable ledger volumes, RPC network payload (15.36 Gbps), bounded quota lease allocations, and regional N+1 capacity sizing (188 instances).

Interview simulation · Senior infrastructure engineering · Distributed admission control, token accounting, and fail-safe quotas

One distinction upfront: this is a realistic simulated interview, not a transcript of an actual Google interview. Workload distributions, benchmark numbers, and lease durations are explicit planning assumptions. The objective is to demonstrate how a strong candidate prevents multi-million-dollar billing overshoots during regional network partitions—not to memorize vendor tooling.

Phase 1 — Define the problem and operational scale

INTERVIEWER

"Design a global rate-limiting platform for 5 million AI API requests per second. Each request consumes a variable number of tokens. Enterprise customers have different per-second and monthly quotas. Calculate the state-storage capacity, check throughput, network requirements, replication overhead, and Bill of Materials while preventing material quota overshoot during regional failures."

CANDIDATE

"The first distinction I need to make is that request rate and resource consumption are not equivalent. A short classification request and a prompt processing tens of thousands of tokens each count as one API request, but their underlying infrastructure costs differ by orders of magnitude.

I will design a distributed admission-control service that evaluates both request quotas and token budgets before allowing work into the AI inference platform.

I need to establish five requirements:

Request enforcement: Enforce per-tenant requests-per-second (RPS) limits.

Token enforcement: Account for variable token consumption across both input and generated output tokens.

Monthly budgets: Prevent tenants from exceeding their contracted monthly financial allocations.

Global consistency: Enforce quotas consistently across regions without adding cross-continental round-trip latency to every API call.

Failure behavior: Strictly bound quota overshoot when a region, network link, or quota service becomes partitioned.

I will separate the low-latency request-decision path from the durable monthly accounting path. The first must handle millions of checks per second in memory; the second must preserve exact financial records for billing reconciliation."

INTERVIEWER

"Good. Start by quantifying the workload."

Phase 2 — Request and token workload math

INTERVIEWER

"You have 5 million requests per second. How many tokens must the platform account for?"

CANDIDATE

"Because request volume alone doesn't convey load, I will use this distribution across four workload classes:"

Request class	Traffic share	Average accounted tokens / request
Short requests	50%	500
Standard requests	35%	2,000
Long-context requests	14%	10,000
Large-context requests	1%	50,000
Total	100%

"The weighted average token consumption per incoming request is:"

T_avg = (0.50 × 500) + (0.35 × 2,000) + (0.14 × 10,000) + (0.01 × 50,000)

T_avg = 250 + 700 + 1,400 + 500 = 2,850 tokens/request

"At 5 million requests per second, the instantaneous token accounting rate is:"

T_rate = 5,000,000 × 2,850 = 14,250,000,000 tokens/second

T_rate = 14.25 billion tokens/s

Global Request Rate	Weighted Average Tokens/Req	Peak Token Accounting Rate	Peak Monthly Equivalent (30 Days)
5,000,000 req/s	2,850 tokens	14.25 B tokens/s	≈ 36.94 Quadrillion tokens

INTERVIEWER

"Do you need to store 36.9 quadrillion token records in a database?"

CANDIDATE

"No. We need to store aggregated counters and in-memory state to make admission decisions. We do not write a durable database row for every individual token.

The critical architectural boundary is between the hot request decision path (evaluating 5 million checks per second in sub-millisecond RAM) and the asynchronous usage ledger (recording aggregated consumption for billing and auditing)."

Phase 3 — The reservation-and-reconciliation quota model

INTERVIEWER

"How do you account for variable token consumption before model generation finishes?"

CANDIDATE

"I would use a two-phase reservation and reconciliation protocol.

At admission time, the exact output token count is unknown. If we only charge after generation finishes, malicious or bursty tenants could submit thousands of large requests simultaneously, blowing through their budget before the first token finishes generating.

The protocol functions as follows:

Phase 1 (Admission Reservation): The API gateway calculates exact input tokens and reserves a bounded upper-bound allowance for output tokens (e.g., max_tokens or a calibrated percentile). If the tenant's budget cannot cover the reservation, the request is throttled immediately.

Phase 2 (Streaming Tranches): For long-running streaming requests, output capacity is checked in bounded incremental tranches rather than locking an unlimited budget upfront.

Phase 3 (Post-Generation Reconciliation): Once generation finishes, the actual tokens consumed are reported. Unused reserved tokens are immediately released back to the tenant's local budget, and actual usage is queued for ledger aggregation.

This guarantees that in-flight requests can never silently overspend available capacity."

Phase 4 — In-memory state and tenant storage capacity

INTERVIEWER

"How many enterprise customers are you designing for, and how much state must the quota service maintain?"

CANDIDATE

"I will assume 100,000 active enterprise tenants. Each tenant maintains a request-rate counter, a token-rate bucket, monthly usage counters, active reservation tracking, and policy version flags.

Assuming an average of 1 KB of active in-memory state per tenant:

The raw working state is:"

S_tenant = 100,000 × 1 KB = 100 MB

"Replicating across three nodes for local high availability:"

S_replicated = 100 MB × 3 = 300 MB

"Adding 100% operational headroom for allocator fragmentation, indexes, and bursts:"

S_planned = 300 MB × 2 = 600 MB

INTERVIEWER

"600 MB is tiny. Does that mean state storage is trivial?"

CANDIDATE

"Storage capacity is trivial; concurrency and write contention are not.

A single shared counter per tenant becomes an intolerable hot key under 5 million requests per second. If a single enterprise customer drives 20% of global traffic (1M QPS), thousands of gateway instances would attempt to execute atomic compare-and-swap operations against the same memory address across regions.

That is why we must decouple global policy storage, local distributed counter state, and durable accounting ledgers."

Phase 5 — Quota check throughput and instance sizing

INTERVIEWER

"Five million requests per second arrive globally. How many quota check instances do you need?"

CANDIDATE

"Assuming atomic local evaluation (checking request rate, token rate, and monthly budget in a single operation), the service must sustain 5,000,000 admission decisions/second.

Let's assume an optimized in-memory quota service instance can sustain 50,000 admission decisions per second while meeting sub-millisecond P99 latency and staying under CPU saturation.

The baseline compute requirement is:"

N_base = 5,000,000 / 50,000 = 100 instances

"Adding 25% operational headroom for traffic spikes, garbage collection, and rolling deployments:"

N_planned = ⌈100 × 1.25⌉ = 125 instances

INTERVIEWER

"Would you put a globally consistent database call on every request?"

CANDIDATE

"Absolutely not. At 5 million QPS, a synchronous cross-region transaction (e.g., Spanner or Raft consensus across WAN links) would destroy our latency budget. Every API call would incur 80–120 milliseconds of speed-of-light network transit time just to check a rate limiter.

Instead, the global coordinator must distribute bounded quota authority.

A centralized coordinator grants local token leases (e.g., 5-second sub-budgets) to regional clusters. Regional gateways make 100% local, sub-millisecond admission decisions against their granted lease.

But this introduces the most critical failure mode in distributed systems: what happens when a region gets partitioned?"

(Interviewer Note: This is the pivotal moment of the interview. The candidate avoids the distributed consensus trap and frames the architecture around lease-based authority).

The interviewer pushes further

INTERVIEWER

"That brings us to the core challenge: If you distribute bounded sub-budgets across three regions, what happens when a network partition cuts off a region? If you fail open, you risk massive financial overshoot; if you fail closed, you kill availability. How do you mathematically guarantee that global overshoot remains strictly bounded during a major regional outage?"

🔒 End of Preview: Workload Sizing & Lease-Based Architecture

You have reached the end of the public preview (Phases 1–5).

Most candidates fail rate-limiting interviews by proposing naive global Redis clusters or distributed locks that collapse under multi-region WAN latency. In an L5/L6 Google SRE round, proving how to distribute quota authority and mathematically bound financial overshoot during partitions is what separates a Senior engineer from a Junior developer.

🗺️ What’s Inside the Full Walkthrough: Phases 6 through 12

In the complete interview session included in our bundle, the candidate steps through the remaining 25 minutes of the mock loop:

Phase 6 — Network RPC Payload & Internal Traffic Sizing (Minutes 18–23)
- Ingress/Egress RPC Math: Sizing 256-byte request payloads and 128-byte responses to uncover 1.92 GB/s (15.36 Gbps) of sustained internal admission traffic.
- Control-Plane Lease Traffic: Calculating renewal frequencies and sizing control fabrics to handle worst-case recovery bursts.
Phase 7 — Bounding Quota Overshoot During Regional Partitions (Minutes 23–28)
- The Multi-Region Overshoot Proof: Proving why independent regional quotas multiply customer spend by 3×, and mathematically distributing bounded sub-allocations (40% / 35% / 25%).
- Fencing & Reclamation: Establishing safe lease reclamation protocols to prevent double-spending without stranding customer quota.
Phase 8 — Monthly Budget Enforcement Without Synchronous WAN Calls (Minutes 28–32)
- Hierarchical Token Leases: Enforcing 100-billion-token enterprise monthly contracts across regional boundaries using bounded drawdown blocks.
- Idempotent Reconciliation: Designing two-phase settlement pipelines to release unspent reservation buffers and absorb delayed regional records.
Phase 9 — Fail-Open vs. Fail-Closed Decision Matrix (Minutes 32–36)
- Operational Triage: A precise condition-by-condition matrix governing token exhaustion, lease expiry, and coordinator isolation.
- The 10,000-Token Emergency Allowance: Enforcing strict, finite financial blast radiuses rather than unbounded fail-open policies.
Phase 10 — Sizing the Durable Usage Accounting Ledger (Minutes 36–39)
- Aggregated Storage Footprint: Deriving 1.44 billion daily records into 288 GB/day (8.64 TB/month) of durable ledger data.
- Auditability & Dispute Resolution: Balancing compact aggregated tables against raw event stream retention.
Phase 11 — Regional N+1 Compute Sizing (Minutes 39–42)
- Surviving Regional Outages: Calculating the +50% regional traffic surge during an N-1 failure.
- Final Fleet Sizing: Scaling from 100 baseline instances to 188 planned instances (distributed 63/63/62 across regions) to preserve both full throughput and 25% headroom during disaster recovery.
Phase 12 — Final Bill of Materials & Interviewer Rubric (Minutes 42–45)
- Complete Infrastructure BOM: Instances, memory targets, RPC payloads, ledger storage, and coordination fabrics.
- Four Mandatory Production Acceptance Tests: Throughput, double-spend correctness, partition isolation, and reconciliation integrity.
- Official Interviewer Scorecard: How Google interviewers evaluate distributed authority, financial risk mitigation, and operational sequencing.

🚀 Get the Full Walkthrough & Complete Interview Bundle

This scenario is just one part of the end-to-end curriculum designed to build your operational reflexes under real interview pressure.

👉 Unlock the Full Walkthrough in the Google SRE Interview Career Launchpad

Stop guessing the math under pressure. Master the simulation.

🏗️ Google SRE NALSD Scenario Simulation 5: Global Distributed Lock and Coordination Service

Why include this: Distributed consensus is the hardest problem in distributed systems. At 500,000 operations per second across three continents, naive single-Raft-group designs collapse instantly under log serialization and speed-of-light cross-WAN latencies. This scenario tests whether a candidate understands the difference between local availability-zone quorums and globally synchronous consensus, and why a lease without a fencing token is an architectural illusion.

The math that matters: 450,000 consensus-backed mutations/sec, Raft majority quorum size (Q = ⌊N/2⌋ + 1), critical-path commit latency budget (23 ms budget vs. 100 ms P99 SLO), 100 sharded consensus groups (300 voters), 7-day replicated write-ahead log growth (209 TB), and degraded N-1 AZ failure capacity (600,000 mutations/sec headroom).

Interview simulation · Senior infrastructure engineering · Consensus, quorum latency, fencing tokens, and split-brain safety

One distinction upfront: this is a realistic simulated interview, not a transcript of an actual Google interview. Consensus throughput baselines, WAL entry sizes, and network allowances are explicit planning assumptions. The objective is to demonstrate how a strong candidate establishes capacity envelopes and enforces mathematical linearizability—not to guess vendor specs.

Phase 1 — Define the problem and consistency boundaries

INTERVIEWER

"Design a globally deployed distributed coordination service that supports 500,000 lock operations per second across North America, Europe, and Asia.

The service must prevent conflicting lock ownership, tolerate one availability-zone failure, preserve acknowledged operations, and meet a defined commit-latency SLO. Calculate quorum latency, storage growth, network requirements, replication capacity, failure reserves, and the final bill of materials."

CANDIDATE

"I would start by separating correctness requirements from capacity requirements. A lock service can process hundreds of thousands of requests per second and still be fundamentally broken if two clients believe they own the same lock simultaneously.

I need to establish five design constraints before sizing anything:

Linearizable Consistency: Lock ownership changes must be strictly linearizable within the defined ownership domain.

Durable Commitment: Once a successful mutation is acknowledged to a client, a single availability-zone (AZ) failure must never erase or roll back that write.

Availability: The service must continue operating after one complete AZ failure, provided a majority quorum remains online.

Latency Boundary: The P99 commit latency must have an explicit target and measurement boundary. For this exercise, I will budget a P99 commit-latency SLO of 100 milliseconds, measured from server-side admission of a valid mutation to durable quorum acknowledgment.

Global Behavior: Clients in North America, Europe, and Asia must reach the correct authority without allowing a wide-area network partition to create split-brain competing owners.

I will assume the 500,000 operations per second represents a peak sustained workload, not a momentary burst."

INTERVIEWER

"Why not start by calculating the number of servers?"

CANDIDATE

"Because server count depends entirely on the consistency protocol and the physical placement of voting replicas.

If I assume the wrong quorum topology (such as requiring synchronous intercontinental WAN round trips on every write), the resulting server count may look mathematically reasonable on paper while physical speed-of-light network latency makes the 100 ms P99 SLO physically impossible to achieve.

I will first define the protocol and calculate its latency floor, then derive server capacity."

Phase 2 — Workload composition and mutation breakdown

INTERVIEWER

"What exactly counts as a lock operation?"

CANDIDATE

"I distinguish acquisition attempts, lease renewals, explicit releases, and status reads. These operations have radically different performance costs.

I will use this illustrative planning mix:"

Operation	Traffic share	Operations/second	Requires Replicated State Mutation?
Acquire / contend for lock	30%	150,000	Yes (consensus write)
Renew lease	40%	200,000	Yes (consensus write)
Release lock	20%	100,000	Yes (consensus write)
Read lock state	10%	50,000	No (safe read protocol)
Total	100%	500,000

"For a conservative baseline, acquisitions, renewals, and releases all require an append to the replicated Write-Ahead Log (WAL).

The system must sustain:"

500,000 × 90% = 450,000 consensus-backed mutations/second

"and 50,000 reads/second."

INTERVIEWER

"Why does contention matter in your capacity model?"

CANDIDATE

"Because when multiple clients contend for the exact same lock key, adding more servers does not increase throughput.

Two clients racing for a single lock cannot both succeed; that operation is strictly serialized. If 500,000 operations target 500,000 distinct distributed locks, they can be sharded and executed in parallel. But if 100,000 clients contend for a single 'hot lock', that key becomes an absolute serialization bottleneck.

For our initial sizing, I assume the keyspace is sufficiently uniform across independent locks to allow parallel partition sharding."

Phase 3 — Global consistency architecture: Home-Region Authority

INTERVIEWER

"Would you place all consensus replicas in one region or distribute them across North America, Europe, and Asia?"

CANDIDATE

"Neither choice should be made without evaluating the speed-of-light network budget.

If we configure a single global Raft group with voters split across continents, every single write requires a synchronous trans-Atlantic or trans-Pacific round trip (e.g., London to Virginia is ~70 ms RTT, Tokyo to Virginia is ~150 ms RTT). Under high load, tail latency and packet jitter will blow past our 100 ms P99 SLO.

Instead, I will use a Home-Region Authority Model:

Each lock key is assigned to a specific consensus group hosted in an authoritative home region (e.g., Region A, B, or C).

The consensus group maintains three voting replicas placed across three independent Availability Zones (AZs) within that home region.

Global clients from any continent can connect to their nearest edge gateway, which routes the request to the lock's authoritative home region.

This confines the synchronous quorum write to ultra-low-latency intra-region fiber (1–2 ms RTT), guaranteeing that normal commit operations easily meet the 100 ms target."

Phase 4 — Quorum math and commit-latency budget

INTERVIEWER

"Calculate the quorum. How many replicas must acknowledge a write?"

CANDIDATE

"For a consensus group with N = 3 voting replicas:"

Q = ⌊N / 2⌋ + 1 = ⌊3 / 2⌋ + 1 = 2 replicas

"The leader must persist the log entry locally and obtain durable acknowledgment from at least one follower before committing the write.

By distributing the three voters across AZ-1, AZ-2, and AZ-3, any single AZ can go completely dark and the surviving two voters still form a valid majority (2/3) to commit writes without interruption."

INTERVIEWER

"Now calculate the commit latency floor."

CANDIDATE

"Let's establish the synchronous critical path within the home region:"

Processing Stage	Assumed P99 Budget
Client ingress & request validation	2 ms
Leader log append & durable fsync	4 ms
Intra-region network round trip to follower	2 ms
Follower durable fsync	4 ms
Commit processing & state-machine apply	3 ms
Queueing and scheduling allowance	10 ms
Total Planning Budget	25 ms

"The synchronous commit critical path is:"

T_commit ≈ T_leader_persist + T_follower_RTT + T_follower_persist + T_processing

T_commit ≈ 4 ms + 2 ms + 4 ms + 3 ms = 13 ms

"Adding our 10 ms queueing allowance gives an estimated 23–25 ms service-side P99 commit time, leaving massive headroom under our 100 ms SLO."

Phase 5 — Lock ownership, fencing tokens, and split-brain safety

INTERVIEWER

"A client acquires a lock, experiences a 30-second GC pause, and wakes up after its lease has expired. Another client has already acquired the lock. How do you prevent the first client from corrupting the protected database or storage bucket?"

CANDIDATE

"A time-based lease alone cannot guarantee mutual exclusion in distributed systems. A client that pauses for 30 seconds due to garbage collection, VM migration, or network paging will resume with stale state and blindly perform its write.

To guarantee linearizability, the consensus state machine must issue a monotonically increasing fencing token on every successful ownership acquisition.

Consider this sequence:

Client A acquires Lock X and receives Fencing Token 104.

Client A pauses (GC freeze); its lease expires after 10 seconds.

Client B acquires Lock X and receives Fencing Token 105.

Client B successfully writes to the downstream storage bucket with Token 105.

Client A wakes up from its GC freeze and attempts to execute its write using Token 104.

The downstream resource rejects Client A's write because it has already committed a write with Token 105 (104 < 105).

The protected resource must enforce fencing token ordering. Without downstream fencing, no distributed lock service on earth can prevent data corruption."

(Interviewer Note: This is the definitive Staff-level distinction. The candidate proves that mutual exclusion is an end-to-end guarantee between the lock service and the downstream protected resource, not a magic property of timers).

The interviewer pushes further

INTERVIEWER

"Now that correctness is locked down, let's size the physical fleet. You need to sustain 450,000 consensus mutations per second. How many consensus groups do you create, how much WAL storage grows per day, and what happens to your throughput when AZ-1 suffers a catastrophic power outage?"

🔒 End of Preview: Consistency Architecture & Fencing Semantics

You have reached the end of the public preview (Phases 1–5).

Most candidates fail this interview by assuming a single consensus cluster can scale to 500,000 writes/sec, or by assuming that network time synchronization (NTP) can safely replace fencing tokens. In an L5/L6 Google SRE round, proving how to partition consensus groups and maintain degraded throughput during failure is what earns a Strong Hire.

🗺️ What’s Inside the Full Walkthrough: Phases 6 through 16

In the complete interview session included in our bundle, the candidate steps through the remaining 25 minutes of the mock loop:

Phase 6 — Sharding the Namespace Across Consensus Groups (Minutes 18–22)
- Group Throughput Sizing: Sizing individual Raft groups at 10,000 mutations/sec baseline (6,000 mut/s at 60% target utilization).
- Horizontal Group Layout: Deriving 75 baseline groups, adding 25% headroom, and establishing 100 independent consensus groups (4,500 mutations/s average load).
- Hot-Key Contention Analysis: Modeling why sharding protects aggregate throughput but leaves individual contested locks strictly serialized.
Phase 7 — Replica Placement Across Availability Zones (Minutes 22–26)
- Physical Voter Footprint: Sizing 100 groups × 3 voters = 300 voting instances.
- Fault-Domain Isolation: Distributing exactly 100 voters in AZ-1, 100 in AZ-2, and 100 in AZ-3 with balanced leader placement.
Phase 8 — Durable WAL Sizing & Compaction Lifecycle (Minutes 26–30)
- Write-Ahead Log Ingestion: Calculating 115.2 MB/s of raw mutation data (450,000 × 256 bytes).
- 7-Day Replicated Storage Math: Sizing 9.95 TB/day logical growth into 209.1 TB of 7-day replicated WAL storage across 3 voting copies.
- Snapshot Compaction & Active State: Sizing 100 million active lock records into 51.2 GB logical (153.6 GB replicated) in-memory state.
Phase 9 — Network Replication Bandwidth & Catch-Up Bursts (Minutes 30–33)
- Follower Fan-Out Payload: Deriving 1.84 Gbps baseline replication bandwidth and budgeting 3.7 Gbps (2× allowance) for protocol framing.
- Snapshot Catch-Up Throttling: Preventing replica recovery transfers from starving consensus heartbeats or triggering spurious leader elections.
Phase 10 — Degraded-Mode Capacity During an AZ Failure (Minutes 33–36)
- The Surviving 2-Voter Capacity Proof: Sizing surviving groups at 6,000 mutations/sec in 2-node degraded mode (100 × 6,000 = 600,000 mut/s).
- Proving +33.3% Degraded Headroom: Demonstrating that the cluster easily clears the 450,000 mutation requirement after losing an entire availability zone.
Phase 11 — Safe Regional Failover & Split-Brain Prevention (Minutes 36–39)
- The 7-Step Authority Transfer Protocol: Monotonically advancing authority generations, fencing old regions, and resolving ambiguous in-flight mutations.
- Preventing Split-Brain: Proving why an unreachable region must never be usurped without verifiable quorum fencing.
Phase 12 — Protecting Against Lease-Expiration Stampedes (Minutes 39–41)
- Mitigating Recovery Storms: Implementing randomized renewal jitter, priority queuing for existing leaseholders, and admission backpressure.
Phase 13 — High-Throughput Linearizable vs. Stale Reads (Minutes 41–43)
- Read-Index & Lease-Read Protocols: Sizing the 50,000 read operations/sec without paying full WAL append penalties.
Phase 14 through 16 — Production BOM, 10 Acceptance Tests & Rubric (Minutes 43–45)
- The Synthesized BOM: 300 voter instances, 100 groups, 209 TB storage, and 3.7 Gbps replication fabric.
- The 10 Mandatory Acceptance Tests: Fault injection, split-brain isolation, GC pause tolerance, and cold-start verification.
- Official Interviewer Scorecard: The exact criteria Google interviewers use to grade consensus design, failure isolation, and operational maturity.

🚀 Get the Full Walkthrough & Complete Interview Bundle

This scenario is just one part of the end-to-end curriculum designed to build your operational reflexes under real interview pressure.

👉 Unlock the Full Walkthrough in the Google SRE Interview Career Launchpad

Stop guessing the math under pressure. Master the simulation.

🏗️ Google SRE NALSD Scenario Simulation 6: Agentic “Infinite Loop” DDoS & Workload Isolation

Why include this: Autonomous AI agents are fundamentally different from human API clients. When a reasoning engine or multi-agent tool loop bugs out, it doesn't back off—it loops every 100 milliseconds across thousands of concurrent instances. This scenario tests queueing theory under hyper-amplification, strict multi-lane capacity isolation, deadline propagation, and how to prevent self-inflicted cancellation storms from taking down production databases.

The math that matters: Workload step amplification (50× surge to 500,000 steps/sec), offered database load (1,020,000 ops/s), net backlog growth rates (900,000 ops/s excess), Little's Law concurrency derivations (L = λW), and mathematically bounding queue capacities (B_max = μ_agent × T_queue = 30,000 ops) to enforce strict 500 ms wait-time limits.

Interview simulation · Senior infrastructure engineering · Admission control, queueing theory, Little's Law, and backpressure

One distinction upfront: this is a realistic simulated interview, not a transcript of an actual Google interview. Database capacities, step amplification ratios, and service times are explicit planning assumptions. The objective is to demonstrate how a strong candidate prevents cascading collapse and enforces mathematical boundaries under runaway traffic—not to recite canned architectures.

Phase 1 — Define the problem and failure domain

INTERVIEWER

"Design an infrastructure platform that supports 50,000 autonomous AI agents running across multiple enterprise tenants. A faulty agent workflow enters a tight loop, repeatedly calling tools, internal APIs, and downstream services. Traffic surges, database latency spikes, and human-facing web requests begin timing out.

Your system must contain runaway workloads, preserve interactive user traffic, calculate backlog growth and maximum queue residence time, and determine the capacity required to survive the storm."

CANDIDATE

"I would treat this as a workload-isolation and capacity-engineering problem, rather than a standard network traffic-filtering problem. The agents are authenticated, authorized workloads, but their buggy loop behavior creates an internal Denial of Service.

I need to answer five quantitative questions:

Workload Generation Rate: How quickly does the storm generate work into the platform?

Sustainable Service Rate: What is the actual, benchmarked throughput capacity of each constrained shared resource?

Backlog & Deadline Physics: How quickly does the queue accumulate, and at what timestamp will queued requests exceed their client deadlines?

Protected Capacity Allocation: How much backend database and compute capacity must be strictly reserved for interactive human traffic?

Containment Mechanics: How do we terminate runaway agent workflows without allowing their retries, compensations, or cancellation hooks to trigger a secondary failure cascade?

A separate agent message queue alone cannot protect human users if those agents still monopolize database connection pools, memory, disk I/O, or CPU."

INTERVIEWER

"Good. Give me a concrete workload model. Do not start with architecture boxes."

Phase 2 — Workload amplification math: Normal vs. Runaway storm

INTERVIEWER

"Give me the traffic numbers."

CANDIDATE

"We have 50,000 active agents. Under normal steady-state operation, each agent initiates an average of one workflow step every 5 seconds:

The baseline arrival rate is:"

λ_normal = 50,000 / 5 = 10,000 workflow steps/second

"Now, consider a bug where affected agents enter an infinite retry or unconstrained reasoning loop, firing a new workflow step every 100 milliseconds (0.1s).

If all 50,000 agents enter this runaway state, the offered arrival rate becomes:"

λ_storm = 50,000 / 0.1 = 500,000 workflow steps/second

"The runaway loop generates a 50× traffic surge."

INTERVIEWER

"Is a workflow step the same as a database operation?"

CANDIDATE

"No. A single agent step reads state, executes tools, and writes memory. I will assume an average of 2 database operations per workflow step.

In parallel, let's assume interactive human users generate 20,000 database operations/second under normal conditions."

Workload	Workflow steps/sec	Database operations/sec
Normal agent traffic	10,000	20,000
Runaway agent storm	500,000	1,000,000
Protected human traffic	N/A	20,000

"Before containment, the total offered load hitting the database tier is:"

λ_DB = 1,000,000 + 20,000 = 1,020,000 operations/second

"This is offered load—not completed work. We must now evaluate how the database responds when offered load exceeds service capacity by an order of magnitude."

Phase 3 — Database capacity boundaries and human reservation

INTERVIEWER

"How much database capacity do you actually have?"

CANDIDATE

"Let's establish a benchmarked sustainable service rate for the database cluster at our target latency SLO:

Total Sustainable DB Capacity: 120,000 operations/second

Protected Human Reservation: 40,000 operations/second (2× normal human peak)

Allocated Agent Budget: 60,000 operations/second

Operational Headroom: 20,000 operations/second (bursts, vacuuming, failover margin)

Notice that the agent budget (60,000 ops/s) is deliberately constrained. We do not let agents consume all remaining capacity simply because the database is idle."

Database Fleet Allocation	Operations/sec	Share of Total	Purpose
Protected Human Workload	40,000	33.3%	Guarantees 2× human traffic headroom
Agent Workload Budget	60,000	50.0%	Hard admission ceiling for all agent jobs
Operational Headroom	20,000	16.7%	Absorbs transaction latency variance
Total Fleet Capacity	120,000	100%	Measured sustainable throughput ceiling

INTERVIEWER

"Why reserve 40,000 ops/s for humans when normal human traffic is only 20,000?"

CANDIDATE

"Because reservations must absorb peak variance, not just median traffic.

However, an architectural reservation cannot exist solely in documentation. If agent queries and human queries share the same connection pool, agent queries will consume all socket descriptors within milliseconds.

We must enforce physical isolation: separate connection pools, distinct DB worker threads, separate query queue priorities, and hard transaction timeouts."

Phase 4 — Backlog growth and queue collapse physics

INTERVIEWER

"The runaway storm offers 1,000,000 ops/second. Your database can sustain 120,000 ops/second. Calculate queue accumulation."

CANDIDATE

"If the platform naively admits all incoming agent traffic without an admission cap, the total offered load is 1,020,000 ops/s.

With a sustainable service capacity of μ = 120,000 ops/s, the net backlog growth rate is:"

λ - μ = 1,020,000 - 120,000 = 900,000 operations/second

"The queue expands by 900,000 pending operations every single second.

After just 10 seconds of uncontained storm traffic:"

B(10) = 900,000 × 10 = 9,000,000 operations

INTERVIEWER

"How long until the queue reaches 10 million operations?"

CANDIDATE

"Under steady-state overload:"

t = 10,000,000 / 900,000 ≈ 11.1 seconds

"In 11 seconds, the system accumulates 10 million pending database operations.

In reality, it collapses even faster: as queues explode, OS memory pressure triggers GC pauses, connection pools exhaust, locks hold open longer, and downstream timeouts trigger aggressive client retries—driving effective database throughput toward zero."

Phase 5 — Queue residence time: Why large queues are lethal

INTERVIEWER

"Suppose you allocate a massive message queue buffer capable of holding 5 million operations. Is that sufficient to absorb the burst?"

CANDIDATE

"A large buffer does not solve an overload problem; it converts an availability problem into a latency catastrophe.

If the queue holds 5 million operations and agents are budgeted at μ_agent = 60,000 ops/s, the time required just to drain the queue is:"

T_drain = 5,000,000 / 60,000 ≈ 83.3 seconds

"A request entering the back of that queue will sit waiting for 83.3 seconds before being touched. But agent workflow steps carry client-side timeouts (e.g., 2 seconds).

That means 100% of the queued work will expire and time out while waiting in line. The worker fleet would spend all its CPU processing requests that the client has already abandoned!

We must mathematically derive the maximum allowable queue size (B_max) from the maximum acceptable queue wait time (T_queue = 500 ms):"

B_max = μ_agent × T_queue

B_max = 60,000 ops/s × 0.5 seconds = 30,000 operations

Parameter	Unbounded / Large Queue	Mathematically Bounded Queue
Queue Depth	5,000,000 operations	30,000 operations
Queue Residence Time	83.3 seconds (100% dead on arrival)	≤ 500 milliseconds (Fresh)
System State	Memory exhaustion & cascading timeout loops	Fast-fail backpressure; zero stale work

"We must cap the agent queue at 30,000 operations. Once that limit is hit, the admission controller must immediately reject incoming agent steps with HTTP 429 / backpressure signals, rather than buffering dead work."

(Interviewer Note: This is an elite NALSD response. The candidate connects queueing theory directly to request deadlines, proving why big buffers destroy system survivability).

The interviewer pushes further

INTERVIEWER

"You've proven that 94% of the storm traffic (470,000 steps/s) must be rejected at ingress. Now, how do you size the worker fleet to process the surviving 30,000 steps/second? How do you isolate resources at the kernel and database connection levels, and what happens when your cancellation logic fires across 50,000 agents simultaneously without triggering a secondary DDoS?"

🔒 End of Preview: Queueing Dynamics & Backlog Mathematics

You have reached the end of the public preview (Phases 1–5).

Most candidates fail this interview by suggesting bigger queues, autoscaling the database, or setting an arbitrary 70% CPU alert. In an L5/L6 Google SRE round, proving how to enforce multi-lane isolation, calculate queue residence bounds, and prevent cancellation storms is what earns a Strong Hire.

🗺️ What’s Inside the Full Walkthrough: Phases 6 through 14

In the complete interview session included in our bundle, the candidate steps through the remaining 25 minutes of the mock loop:

Phase 6 — Multi-Level Admission Control Architecture (Minutes 18–22)
- Layered Defense: Implementing per-tenant rate buckets, per-workflow step quotas, global agent concurrency bounds, and strict deadline propagation.
- Beyond Simple RPS: Why transaction lock durations and expensive tool executions bypass scalar rate limits unless query-level budgets are enforced.
Phase 7 — Physical Backend Isolation for Interactive Traffic (Minutes 22–26)
- Connection Pool Disaggregation: Isolating the 40,000 ops/s human reservation into physically segregated connection pools and dedicated worker threads.
- Avoiding Shared Bottlenecks: Preventing shared lock tables, transaction log contention, and disk I/O saturation from bridging the isolation boundary.
Phase 8 — Storm Containment & 94% Traffic Shedding Math (Minutes 26–29)
- The 470,000 Steps/sec Rejection Protocol: Implementing fast-fail drop policies, client retry suppression, and payload deduplication at the edge.
Phase 9 — Sizing the Agent Worker Fleet (Minutes 29–33)
- Worker-Equivalent Math: Deriving the 500 worker-equivalent fleet (30,000 steps/s ÷ 60 planned steps/worker).
- Translating to Physical Hosts: Accounting for serialization overhead, runtime memory footprints, and heavy tool-call execution.
Phase 10 — Little’s Law Concurrency & Downstream Latency Jitter (Minutes 33–36)
- In-Flight Step Math: Applying Little's Law (L = λW) to calculate nominal concurrency (1,500 active steps at 50 ms latency).
- The Latency Shock Scenario: Modeling what happens when downstream latency spikes to 200 ms (6,000 concurrent step explosion) and capping execution slots.
Phase 11 — Multidimensional Saturation Detection Beyond CPU (Minutes 36–39)
- Why CPU Thresholds Lie: Detecting hidden queueing cascades via connection exhaustion, lock contention, and I/O wait while CPU remains at 40%.
- Hysteresis Recovery Curves: Designing dual-threshold recovery curves to prevent oscillation between full drop and full admission.
Phase 12 — Safe Cancellation Pipelines & Preventing Secondary Storms (Minutes 39–41)
- The Cancellation Storm Trap: Why bulk canceling 50,000 fanned-out agents triggers secondary API storms.
- Bounded Cleanup: Implementing propagating cancellation tokens, bounded cleanup budgets, and uninterruptible call limits.
Phase 13 through 14 — Production BOM, 10 Acceptance Tests & Rubric (Minutes 41–45)
- The Synthesized BOM: 120k ops/s DB fleet, 500 worker-equivalents, 30k bounded queue, and dedicated interactive reserves.
- The 10 Fault-Injection Tests: Validating runaway containment, tenant monopolization, connection exhaustion, and gradual recovery.
- Official Interviewer Scorecard: The exact rubrics Google interviewers use to grade overload containment, queueing discipline, and operational safety.

🚀 Get the Full Walkthrough & Complete Interview Bundle

This scenario is just one part of the end-to-end curriculum designed to build your operational reflexes under real interview pressure.

👉 Unlock the Full Walkthrough in the Google SRE Interview Career Launchpad

Stop guessing the math under pressure. Master the simulation.

🏗️ Google SRE NALSD Scenario Simulation 7: Multi-Modal AI Inference at the Edge

Why include this: Edge AI is where raw physical media transport collides with transformer memory architecture. Streaming 1,000,000 concurrent voice-and-video sessions requires balancing multi-terabit WAN ingress against raw frame decoding pipelines and massive GPU KV-cache footprints. This scenario tests whether a candidate can disaggregate media networking from accelerator memory, avoid naive full-frame copies, and prevent reconnect storms during rack-level hardware failure.

The math that matters: Encoded media ingress (4.064 Tbps payload / 5.3 Tbps provisioned), 2M sampled frames/sec (12.44 TB/s raw pixel processing), multi-modal KV cache (1.95 GiB/session), memory-bound vs. throughput-bound GPU sizing (2,500 GPUs memory vs. 1,429 GPUs compute), 4.4 Tbps rack-recovery bandwidth spikes, and 2-second circular replay buffers.

Interview simulation · Senior infrastructure engineering · Bandwidth engineering, GPU memory footprints, session state recovery, and edge topology

One distinction upfront: this is a realistic simulated interview, not a transcript of an actual Google interview. Bitrates, frame sampling rates, and GPU memory ceilings are explicit planning assumptions. The objective is to demonstrate how a strong candidate establishes capacity envelopes and mitigates tail latency—not to guess vendor specs.

Phase 1 — Define the problem and operational scale

INTERVIEWER

"Design a globally distributed, edge-based multimodal inference platform supporting 1,000,000 concurrent voice-and-video sessions. Users expect interactive, low-latency conversational responses while streaming continuous video and audio to nearby inference infrastructure.

The platform must survive GPU-node and complete rack failures, recover sessions without losing essential conversation context, and calculate ingress bandwidth, GPU memory, inference capacity, recovery traffic, and failover headroom."

CANDIDATE

"I would begin by separating the incoming media stream from the model's internal transformer state.

The network carries compressed, encoded media; the media pipeline decodes, crops, and samples frames; and the inference engine maintains session context and potentially massive GPU-resident Key-Value (KV) caches.

Those are three fundamentally different physical resources with completely different scaling curves. I will explicitly model:

Encoded Media Ingress: Ingress network bandwidth for compressed video and audio streams.

Decoded Frame Pipeline: Frame sampling rate and raw pixel data movement.

GPU Inference Fleet: Active conversational sessions, token generation throughput, and HBM KV-cache footprints.

Session Placement: Dynamic routing across edge points of presence (PoPs) vs. centralized compute clusters.

Failure & Reconnect Dynamics: Recovery bandwidth, context reconstruction, and prefill headroom after a GPU node or rack disappears.

I will also distinguish a session's recoverable conversation state from its ephemeral GPU memory. A consistent-hashing ring can route a reconnected user to a healthy node, but it cannot magically resurrect GPU HBM that vanished during a node crash."

INTERVIEWER

"Good. Establish the workload parameters."

Phase 2 — Workload specification & media baseline

INTERVIEWER

"What are your baseline media assumptions?"

CANDIDATE

"I will establish the following planning baseline for our 1,000,000 concurrent sessions:"

Parameter	Planning Assumption	Rationale
Concurrent Sessions	1,000,000	Target peak active connections
Encoded Video Stream	4 Mbps (H.264 / 1080p)	Standard high-definition client uplink
Camera Frame Rate	30 frames/second	Source client capture rate
Model Frame Sampling	2 frames/second	Sampled keyframes for visual transformer input
Encoded Audio Stream	64 kbps (Opus / AAC)	Low-latency bi-directional voice stream
Retained Context Horizon	120 seconds	Active sliding conversational window
Failover Recovery Target	≤ 5 seconds	Target latency to resume output after node failure

INTERVIEWER

"Why distinguish the client source frame rate (30 fps) from the inference frame rate (2 fps)?"

CANDIDATE

"Because receiving 30 frames per second over the network does not mean the visual model should execute transformer prefill on all 30 frames.

Processing 30 full 1080p frames per second across 1,000,000 users would require 30 million visual encodings per second—exploding accelerator cost into the billions of dollars. Sampling 2 frames per second (or triggering on scene-change embeddings) captures visual context while cutting model compute demand by 93.3%.

The client bitrate dictates network ingress capacity; the sampled frame rate dictates decode and vision-encoder compute; and the conversational horizon dictates GPU KV-cache memory. Conflating these three dimensions leads to disastrously wrong sizing."

Phase 3 — Encoded network ingress bandwidth

INTERVIEWER

"Calculate the aggregate network ingress requirement for one million sessions."

CANDIDATE

"Let's calculate the raw encoded media payload:

For the video streams:"

B_video = 1,000,000 × 4 Mbps = 4,000,000 Mbps = 4 Tbps

"For the audio streams:"

B_audio = 1,000,000 × 64 kbps = 64,000,000 kbps = 64 Gbps

"The aggregate encoded media payload is:"

B_payload = 4 Tbps + 0.064 Tbps = 4.064 Tbps

Stream Component	Per-Session Rate	1,000,000 Sessions Payload
Video (1080p H.264)	4 Mbps	4,000 Gbps (4.000 Tbps)
Audio (Opus/Speech)	64 kbps	64 Gbps (0.064 Tbps)
Total Media Payload	4.064 Mbps	4.064 Tbps

INTERVIEWER

"Is 4.064 Tbps the physical network capacity we need to provision?"

CANDIDATE

"No. 4.064 Tbps is pure unpacketized payload.

We must add at least 30% operational headroom to account for WebRTC/RTP encapsulation, IP/UDP header overhead, TLS/SRTP crypto expansion, packet retransmissions (NACKs), and traffic burstiness:"

B_planned = 4.064 Tbps × 1.30 ≈ 5.2832 Tbps

"The platform requires approximately 5.3 Tbps of aggregate ingress capacity.

This bandwidth must be geographically distributed across regional Edge PoPs close to users. We do not haul 5.3 Tbps of raw consumer video across transcontinental backbone links."

Phase 4 — Decoded frame processing rate: The raw pixel explosion

INTERVIEWER

"You receive 30 fps but sample 2 fps for model ingestion. What is the processing rate through the visual decoding pipeline?"

CANDIDATE

"At 2 sampled frames per second across 1,000,000 concurrent sessions:"

F_sampled = 1,000,000 × 2 = 2,000,000 frames/second

"The media decode fleet must decode and preprocess 2 million frames every second.

To understand the data movement risk, consider the raw pixel footprint of an uncompressed 1080p (1920 × 1080), 8-bit, 3-channel RGB image:"

S_frame = 1920 × 1080 × 3 bytes = 6,220,800 bytes ≈ 6.22 MB

"If the system naively copied uncompressed raw frames between processes or across the network, the data rate would be:"

2,000,000 frames/s × 6.22 MB ≈ 12.44 TB/s (99.5 Tbps)

INTERVIEWER

"What does that 12.44 TB/s figure dictate architecturally?"

CANDIDATE

"It proves that raw decoded frames must never traverse the network.

Video decoding, cropping, resizing to model input dimensions (e.g., 384 × 384), and normalization must occur on the ingress media host or via hardware video decoders (NVDEC) located in the same PCIe bus or NUMA domain as the vision encoder.

The pipeline must transform raw video frames into compact vision tokens or embeddings at ingress, discarding raw pixels immediately."

Phase 5 — Session placement and edge distribution

INTERVIEWER

"How would you distribute one million sessions across the edge?"

CANDIDATE

"I would deploy a two-tier hierarchy: Edge Ingress PoPs (for TLS termination, media decoding, and audio streaming) and Regional Compute Hubs (hosting high-density GPU clusters).

For an initial planning baseline, assume 20 edge clusters globally. Under a uniform distribution:"

S_cluster = 1,000,000 / 20 = 50,000 sessions/cluster

"Each edge cluster terminates an average of:"

50,000 × 4.064 Mbps = 203.2 Gbps ingress payload

INTERVIEWER

"Would you use consistent hashing for session placement?"

CANDIDATE

"Consistent hashing is an effective routing and placement mechanism—it minimizes session reassignment when worker nodes scale up or down.

But consistent hashing is not a state recovery protocol.

When a GPU host crashes, the hash ring will immediately redirect the user to a surviving worker node. But that surviving node has zero knowledge of the past 120 seconds of conversation. Its GPU memory does not contain the session's KV cache.

If the replacement node is forced to rebuild context by re-reading and re-encoding two minutes of historical video, that single user failover will trigger massive prefill spikes, crushing cluster throughput.

We must explicitly decouple routing placement from state persistence and recovery."

(Interviewer Note: This is an exceptional NALSD distinction. The candidate refuses to let a routing algorithm substitute for an end-to-end failure recovery design).

The interviewer pushes further

INTERVIEWER

"Now let's tackle the core bottleneck: GPU memory and recovery. What is the KV-cache footprint per session, how many GPUs do we provision, and how do you recover 50,000 sessions within 5 seconds when an entire rack loses power without melting down the surviving GPU cluster?"

🔒 End of Preview: Ingress Bandwidth & Media Pipeline Sizing

You have reached the end of the public preview (Phases 1–5).

Most candidates fail edge multimodal interviews by confusing network bitrates with GPU memory footprints, or assuming that edge nodes can afford to store uncompressed video frames in memory. In an L5/L6 Google SRE round, proving the physical boundary between media decode, accelerator HBM, and failover recovery traffic is what earns a Strong Hire.

🗺️ What’s Inside the Full Walkthrough: Phases 6 through 15

In the complete interview session included in our bundle, the candidate steps through the remaining 25 minutes of the mock loop:

Phase 6 — Multi-Modal KV-Cache Memory per Session (Minutes 18–22)
- Layer-by-Layer Derivation: Sizing a 64-layer, 8-head, 128-dim model at 2 bytes/element over 8,000 context tokens.
- The 1.95 GiB/Session Ceiling: Proving why each session consumes exactly 1.953 GiB (256 KiB/token) of dedicated GPU memory.
Phase 7 — Aggregate Memory & The 10% Active Generation Duty Cycle (Minutes 22–26)
- The 2.05 PB Trap: Why keeping all 1,000,000 sessions resident in GPU HBM simultaneously is an unaffordable financial mistake.
- Duty-Cycle Disaggregation: Sizing the fleet for 100,000 actively generating sessions (195.3 TiB active KV memory) while offloading idle sessions.
Phase 8 — Throughput-Bound vs. Memory-Bound GPU Sizing (Minutes 26–30)
- Compute Sizing: Deriving the 1,429 GPU decode requirement (2,000,000 tokens/s ÷ 1,400 planned tokens/GPU).
- Memory Sizing: Calculating the 2,500 GPU placement requirement (80 GiB HBM ÷ 1.953 GiB = 40 sessions/GPU).
- The Structural Winner: Demonstrating why memory capacity (2,500 GPUs) completely dominates compute throughput (1,429 GPUs).
Phase 9 — Four Session-State Strategies Compared (Minutes 30–33)
- Trade-off Analysis: Evaluating Cache Pinning, Replicated Session Metadata, Synchronous KV Replication, and Selective Context Replay.
- The Optimal Architecture: Combining cache pinning with lightweight metadata checkpoints and deterministic prefill reconstruction.
Phase 10 — The 4.4 Tbps Rack-Level Recovery Shock (Minutes 33–36)
- The Single-Node Failure: Sizing the 78.1 GiB cache transfer over 100 Gbps (6.7 seconds), proving direct live cache transfer breaches the 5s SLO.
- The 32-Node Rack Failure: Proving why synchronous transfer of 2.44 TiB across a failed rack demands an impossible 4.4 Tbps burst network.
Phase 11 — Selective Replay & 2-Second Media Buffers (Minutes 36–39)
- Buffer Footprint: Sizing a circular 2-second encoded media buffer (1.016 MB/session) into a compact 1.016 TB aggregate cluster buffer.
- Decoupling Media Reconnect from Full History: Balancing immediate media stream resumption with background conversation context reconstruction.
Phase 12 — Rack Failure N+1 Headroom & Sizing (Minutes 39–41)
- Rack Topology: Sizing 79 racks of 32 GPUs each and adding N+1 spare capacity (2,532 installed GPUs).
- Preventing Spatial Imbalance: Ensuring surviving racks have sufficient unfragmented HBM slots to absorb the 1,280 displaced sessions.
Phase 13 through 15 — Reconnect Storm Admission, BOM & Rubric (Minutes 41–45)
- Staged Recovery Pipeline: Rate-limiting 50,000 reconnects to 1,667 reconnects/sec to prevent prefill collapse.
- The Production BOM: 5.3 Tbps ingress, 2,532 GPUs, 195 TiB active HBM, 1.016 TB ring buffer, and redundant edge routing.
- Official Interviewer Scorecard: The exact rubrics Google interviewers use to grade bandwidth engineering, accelerator utilization, and recovery sequencing.

🚀 Get the Full Walkthrough & Complete Interview Bundle

This scenario is just one part of the end-to-end curriculum designed to build your operational reflexes under real interview pressure.

👉 Unlock the Full Walkthrough in the Google SRE Interview Career Launchpad

Stop guessing the math under pressure. Master the simulation.

🏗️ Google SRE NALSD Scenario Simulation 8: Post-Quantum Cryptography Fleet Migration

Why include this: Post-quantum cryptography (PQC) is no longer a theoretical exercise—NIST's standardized ML-KEM (FIPS 203) key encapsulation is being rolled out globally to prevent "Harvest Now, Decrypt Later" attacks. Migrating 100,000 microservices across shared service meshes and edge proxies introduces larger public keys, cipher expansion, higher CPU cycle consumption, and packet fragmentation. This scenario tests whether a candidate can model cryptographic capacity, connection churn, and canary blast radiuses under strict latency SLOs.

The math that matters: 40,000 peak handshakes/second, baseline vs. hybrid core throughput (250 vs. 200 handshakes/core/sec yielding a +25% CPU tax), 286 baseline cores scaled to 429 cores for regional N+1 survivability, 3,000 extra wire bytes/handshake (0.96 Gbps payload overhead), Little's Law in-flight buffer memory, and synchronized rollout wave churn (33,333 handshakes/s shock load).

Interview simulation · Senior infrastructure engineering · Cryptographic engineering, TLS handshake capacity, CPU overhead, and safe fleet rollout

One distinction upfront: this is a realistic simulated interview, not a transcript of an actual Google interview. Handshake benchmarks, wire payload sizes, and migration velocity are explicit planning assumptions. The objective is to demonstrate how a strong candidate establishes capacity envelopes and executes zero-downtime cryptographic cutovers—not to guess vendor specs.

Phase 1 — Define the problem and migration scope

INTERVIEWER

"Design a fleet-wide migration of 100,000 microservices from conventional public-key cryptography (RSA / ECDH) to post-quantum cryptography (ML-KEM). The migration must preserve service availability, satisfy existing connection-establishment latency SLOs, absorb increased handshake compute costs, and prevent overloading shared envoy proxies and edge ingress during rollout.

Calculate handshake throughput, CPU requirements, network overhead, connection setup latency, rollout rate, and the final Bill of Materials."

CANDIDATE

"I would treat this as both a cryptographic compatibility migration and an infrastructure capacity-engineering exercise.

The most critical architectural distinction is that post-quantum cryptography does not impose one uniform overhead across the fleet. The actual cost depends on:

The specific key encapsulation mechanism (KEM) chosen.

The negotiated TLS cipher suite (e.g., hybrid X25519 + ML-KEM-768).

Certificate chain depth and signature algorithms.

Connection reuse efficiency (HTTP/2 and gRPC connection pooling vs. short-lived HTTP/1.1 connection churn).

Whether TLS terminates at shared edge/sidecar proxies or within the application runtimes.

I will size the fleet from explicit benchmark assumptions across five core dimensions:

Handshake Throughput: Total connection setup rates, including persistent connection churn, autoscaling bursts, and client retries.

Cryptographic Compute: CPU cores required for asymmetric key decapsulation and signature verification.

Transport & Network Overhead: Wire byte increases, MTU-induced packet fragmentation, and proxy memory buffering.

Rollout Velocity & Blast Radius: Pacing migration cohorts to prevent self-inflicted connection storms during rolling restarts.

Safe Rollback Strategy: Automated rollback mechanics without stranding incompatible client/server cryptographic handshakes."

INTERVIEWER

"Good. Define the workload parameters."

Phase 2 — Fleet scale and handshake workload baseline

INTERVIEWER

"Give me a concrete workload model."

CANDIDATE

"I will establish the following planning baseline for the 100,000 microservices:"

Parameter	Planning Assumption	Rationale
Microservices in Scope	100,000	Fleet-wide service count
Migration Window	14 days	Target operational transition window
Baseline TLS Handshakes	20,000/second	Measured steady-state connection establishment
Peak Handshake Multiplier	2× average	Peak diurnal traffic and pod scaling bursts
Target Peak Handshake Rate	40,000/second	Fleet-wide sizing requirement
Target Cryptographic Suite	Hybrid Key Exchange (e.g., X25519 + ML-KEM-768)	NIST FIPS 203 compatibility transition
Service Availability SLO	99.99% availability	Zero-downtime requirement during cutover
Rollback Capability	Granular per-cohort rollback	Instant config reversion upon SLO breach

INTERVIEWER

"Why is the raw microservice count (100,000) insufficient to size the hardware?"

CANDIDATE

"Because 100,000 microservices using persistent gRPC connection pooling with long-lived HTTP/2 streams might generate almost zero new handshakes per second. Conversely, 1,000 poorly configured legacy microservices establishing a fresh TCP+TLS handshake for every outbound REST call could generate 100,000 handshakes per second.

Handshake load is a function of connection lifetime and churn, not service count.

We must explicitly measure new TLS handshakes per second, distinguishing between full asymmetric handshakes, session resumption (TLS session tickets / PSK), and failed handshakes. For our capacity design, we will size for 40,000 full handshakes per second at peak."

Phase 3 — Cryptographic transition strategy: Why hybrid key exchange?

INTERVIEWER

"Which post-quantum algorithm would you deploy, and why not replace the entire cryptographic stack overnight?"

CANDIDATE

"For key establishment, I would deploy ML-KEM (Module-Lattice-Based Key-Encapsulation Mechanism, standardized in NIST FIPS 203).

However, an overnight 'big bang' migration across 100,000 microservices is catastrophic for two reasons:

Cryptographic Fragility: If a novel quantum-resistant algorithm is found to have an unforeseen algebraic flaw, a pure post-quantum deployment leaves the entire fleet vulnerable.

Ecosystem Incompatibility: Heterogeneous clients, legacy Java runtimes, middleboxes, and hardware load balancers will choke on unrecognized TLS extensions and jumbo ClientHello packets.

Therefore, we must deploy a Hybrid Key Exchange (combining classical elliptic-curve Diffie-Hellman like X25519 with ML-KEM-768). The shared secret is derived by hashing both outputs together (SS = KDF(SS_classic || SS_pq)).

An attacker must break both algorithms simultaneously to decrypt the session. This provides quantum resistance against future capture while preserving classical security if the post-quantum math falters.

Furthermore, we keep classical ECDSA/RSA certificates initially, migrating key exchange first and certificates second. This isolates the CPU and payload blast radius."

Phase 4 — Sizing handshake compute capacity

INTERVIEWER

"How many CPU cores must the fleet allocate to handle 40,000 handshakes per second?"

CANDIDATE

"Let's establish a benchmarked throughput capacity for our target hybrid configuration on our reference server hardware.

Assume benchmark load testing proves that a single modern CPU core running our optimized TLS library (e.g., BoringSSL with AVX2/AVX-512 acceleration) sustains 200 full hybrid handshakes per core per second while staying within our P99 latency budget.

At a planned operational ceiling of 70% CPU utilization (leaving headroom for burstiness and scheduling variance):"

C_core = 200 × 0.70 = 140 handshakes/second/core

"To sustain the 40,000 peak handshake requirement, the dedicated core requirement is:"

N_cores = ⌈40,000 / 140⌉ = ⌈285.71⌉ = 286 CPU cores

Peak Handshake Rate	Benchmarked Core Rate	Planned Core Rate (70% Util)	Handshake Compute Fleet
40,000 / sec	200 handshakes/s	140 handshakes/s	286 CPU Cores

INTERVIEWER

"Does that 286-core count represent the total CPU needed for the proxy fleet?"

CANDIDATE

"No. That sizes only the asymmetric cryptographic handshake budget.

Symmetric data encryption (AES-GCM / ChaCha20-Poly1305), proxy HTTP parsing, routing filters, access logging, and service-mesh sidecar overhead must be sized on top of this.

286 cores is strictly the cryptographic tax for connection establishment."

Phase 5 — Quantifying the cryptographic CPU tax

INTERVIEWER

"Suppose the conventional TLS configuration benchmarks at 250 handshakes per core per second, while the hybrid configuration drops to 200. What is the net increase in compute demand?"

CANDIDATE

"Let's calculate the baseline conventional compute demand under the same 70% utilization ceiling:

Planned throughput per conventional core:"

C_classic = 250 × 0.70 = 175 handshakes/second/core

"Cores required for conventional TLS at 40,000 handshakes/second:"

N_classic = ⌈40,000 / 175⌉ = 229 CPU cores

"For the hybrid configuration, we require 286 cores. The absolute deficit is:"

ΔN = 286 - 229 = 57 additional CPU cores

"The relative compute overhead of enabling post-quantum hybrid key exchange is:"

Overhead = (250 / 200) - 1 = 1.25 - 1 = +25%

Cryptographic Configuration	Benchmarked Core Throughput	Cores Needed (40k Peak)	Net Compute Delta
Conventional TLS (ECDH)	250 handshakes/s	229 Cores	Baseline
Hybrid TLS (ECDH + ML-KEM)	200 handshakes/s	286 Cores	+57 Cores (+25.0%)

INTERVIEWER

"Why not simply add an arbitrary 30% buffer across the entire fleet and call it done?"

CANDIDATE

"Because an arbitrary buffer treats symptoms instead of physics.

If a service uses persistent connections with 99% reuse, adding 30% CPU to the application servers is a massive waste of capital. But if a shared edge proxy handles high-churn external mobile clients, a generic 30% reserve might completely fail during morning traffic spikes when connection establishment concentrates on specific ingress nodes.

Capacity planning must isolate the exact point of termination (edge vs. ingress vs. sidecar mesh) and size for the measured connection churn."

(Interviewer Note: This is an elite NALSD response. The candidate connects cryptographic math directly to infrastructure topology rather than relying on lazy fleet-wide padding).

The interviewer pushes further

INTERVIEWER

"Now let's examine the network and deployment physics. ML-KEM keys are orders of magnitude larger than classical elliptic curves. How many additional gigabits per second does this inject into our network fabric, how does it affect proxy buffer memory, and what happens when your rollout restarts 10,000 microservices simultaneously and creates a massive 33,000 handshake/second connection storm?"

🔒 End of Preview: Handshake Workload & Cryptographic CPU Sizing

You have reached the end of the public preview (Phases 1–5).

Most candidates fail cryptographic migration interviews by assuming post-quantum upgrades are just a library update—ignoring packet fragmentation, connection-churn shock waves during rolling restarts, and proxy buffer exhaustion. In an L5/L6 Google SRE round, proving how to model the physical wire overhead, scale for regional N+1 survivability, and pace canary rollout cohorts is what earns a Strong Hire.

🗺️ What’s Inside the Full Walkthrough: Phases 6 through 14

In the complete interview session included in our bundle, the candidate steps through the remaining 25 minutes of the mock loop:

Phase 6 — Network Wire Overhead & Cipher Expansion (Minutes 18–22)
- Payload Breakdown: Sizing ML-KEM-768 public keys and ciphertexts (1,200 bytes client-to-server, 1,800 bytes server-to-client = 3,000 bytes extra wire payload).
- Network Bandwidth Impact: Proving that 40,000 handshakes/sec generates an extra 120 MB/s (0.96 Gbps sustained payload) and evaluating Path MTU (PMTU) packet fragmentation risks.
Phase 7 — Proxy Buffer Memory via Little’s Law (Minutes 22–26)
- In-Flight Handshake Math: Applying Little's Law (L = λW) across a 50 ms mean handshake duration to uncover 2,000 concurrent in-progress handshakes.
- Incremental Memory Footprint: Sizing 16 KiB of temporary crypto allocations per handshake into 31.25 MiB of dynamic buffer memory, and evaluating tail-latency buffer bloat.
Phase 8 — Quantifying Connection Setup Latency & Tail Impact (Minutes 26–29)
- Latency Benchmarks: Comparing conventional (18 ms P50 / 45 ms P99) against hybrid TLS (20 ms P50 / 52 ms P99).
- SLO Budget Enforcement: Analyzing why a +7 ms (+15.6%) tail latency shift breaches aggressive 50 ms connection-establishment SLOs and how to mitigate it via session resumption.
Phase 9 — Fleet Rollout Velocity & Schedule Math (Minutes 29–32)
- Pacing the Migration: Deriving 7,143 services/day (298 services/hour) across the 14-day migration window.
- Canary Cohort Architecture: Designing an automated 4-tier rollout (100 → 1,000 → 10,000 → 100,000 services) with automated SLO pause thresholds.
Phase 10 — The 33,000 Handshakes/sec Restart Storm (Minutes 32–35)
- The Rollout Churn Trap: Modeling how a 10,000-service deployment wave restarting simultaneously creates 2,000,000 new handshakes in 60 seconds (33,333 handshakes/sec shock load).
- Preventing Proxy Meltdown: Implementing graceful connection draining, randomized restart jitter, and mesh-level handshake rate limiters.
Phase 11 — Cryptographic Compatibility Matrix & Rollback Pipelines (Minutes 35–38)
- Automated Rollback Triggers: Pre-programming rollback conditions for negotiation drops, proxy CPU saturation, and elevated TCP retransmissions.
- Bidirectional Compatibility: Ensuring clients and servers gracefully negotiate fallback without downgrading security or creating split-brain configurations.
Phase 12 — Sizing the Regional N+1 Resilient Fleet (Minutes 38–41)
- Surviving Regional Proxy Loss: Sizing 3 symmetric regions where the loss of one region leaves the remaining two absorbing 100% of the 40,000 handshake peak.
- The N+1 Core Derivation: Scaling the 286 baseline cores to 429 total installed CPU cores (143 cores per region).
Phase 13 through 14 — Production BOM, Acceptance Tests & Rubric (Minutes 41–45)
- The Synthesized BOM: 429 CPU cores, 0.96 Gbps network allowance, 31.25 MiB buffer memory, and automated canary controllers.
- The 10 Mandatory Production Verification Tests: Testing PMTU limits, restart storms, canary rollbacks, and crypto fallback attacks.
- Official Interviewer Scorecard: How Google interviewers evaluate cryptographic capacity planning, failure-domain engineering, and operational risk mitigation.

🚀 Get the Full Walkthrough & Complete Interview Bundle

This scenario is just one part of the end-to-end curriculum designed to build your operational reflexes under real interview pressure.

👉 Unlock the Full Walkthrough in the Google SRE Interview Career Launchpad

Stop guessing the math under pressure. Master the simulation.

🏗️ Google SRE NALSD Scenario Simulation 9: Rack-Level Emergency Power Shedding

Why include this: Hyperscale data centers operate at physical limits where electrical grid substation drops, cooling compressor trips, or utility brownouts require shedding massive amounts of IT power within minutes. Software engineers routinely fail this interview by assuming that sending SIGSTOP or terminating pods drops electrical draw—unaware that idle servers with fully powered GPUs and spin-up fans still consume up to 60% of their peak wattage. This scenario tests closed-loop telemetry, hardware-enforced power caps, and emergency control plane deadlines.

The math that matters: Shedding 3 MW of a 12 MW IT footprint (-25%), calculating a +20% curtailment reserve (P_target = 3.6 MW), 100-rack inventory reconciliation, multi-tier action portfolios (1.6 MW GPU caps + 1.2 MW batch pause + 1.0 MW server off = 3.8 MW nominal), 10% measurement uncertainty margin (3.42 MW net), and the 175-second critical-path control budget against a 300-second (5-minute) hard deadline.

Interview simulation · Senior infrastructure engineering · Physical constraints, hardware power capping, closed-loop telemetry, and emergency control systems

One distinction upfront: this is a realistic simulated interview, not a transcript of an actual Google interview. Facility capacities, rack power densities, and telemetry latency bounds are explicit planning assumptions. The objective is to demonstrate how a strong candidate enforces physical electrical boundaries and prevents catastrophic breaker trips—not to guess vendor specs.

Phase 1 — Define the physical constraint and failure boundaries

INTERVIEWER

"A data center experiences a sudden electrical utility constraint. The facility operator mandates that we reduce IT power consumption by 3 megawatts within five minutes while strictly protecting critical online services.

The facility hosts GPU inference, batch training, interactive web services, distributed storage, networking fabric, and control plane services. Design a power-shedding system that identifies which workloads can be curtailed, calculates the available electrical reduction, and guarantees that the emergency control path acts within the five-minute deadline.

Calculate rack-level power budgets, curtailment capacity, control-plane latency, failure reserves, and the final Bill of Materials."

CANDIDATE

"I would treat this as a physical capacity-control problem governed by electrical physics, rather than an application autoscaling task.

Standard autoscaling or Kubernetes pod eviction cannot be the primary emergency response—it is far too slow, and terminating or pausing a process does not guarantee a predictable or immediate drop in electrical draw from the wall.

I begin by distinguishing three distinct electrical quantities:

Measured IT Power: The real-time electrical draw currently consumed by compute hosts, accelerators, storage, networking switches, and in-row cooling within the facility boundary.

Curtailable Power: The empirical, benchmarked wattage reduction demonstrated by triggering a specific hardware or workload control action.

Guaranteed Emergency Reduction: The net curtailment that remains guaranteed after discounting telemetry error, incomplete commands, rack communication drops, and the strict five-minute deadline.

My design will rely on preconfigured hardware-level power caps, selective workload suspension/termination, edge admission control, and an isolated, high-priority emergency controller.

The system must never assume that sending a software signal to an application process directly controls electrical current."

INTERVIEWER

"Good. Establish the facility inventory and workload priorities."

Phase 2 — Facility power inventory & workload prioritization

INTERVIEWER

"What assumptions will you use for the 12 MW facility?"

CANDIDATE

"Let's assume the data center currently consumes 12 MW of total IT load.

A mandatory reduction of 3 MW represents a sudden 25% reduction in total facility power draw.

I will classify the data center's 12 MW into four distinct priority tiers:"

Workload Tier	Initial IT Power	Priority Status	Curtailment Eligibility
Critical Interactive Inference	3.0 MW	Highest	Protected (Zero Shedding)
Essential Storage, Network & Control Plane	2.0 MW	Highest	Protected (Zero Shedding)
Flexible GPU Training & Batch Analytics	5.0 MW	Curtailable	Primary Target for Shedding
Deferrable Compute & Noncritical Batch	2.0 MW	Curtailable	Secondary Target for Shedding
Total Facility IT Load	12.0 MW		7.0 MW Eligible Pool

INTERVIEWER

"Why classify workloads before picking your shedding mechanism?"

CANDIDATE

"Because our objective is not simply to drop power indiscriminately—it is to drop power without collapsing services that must remain available.

We have 7.0 MW of nominally curtailable workload to extract 3.0 MW of reduction. However, a flexible batch job might physically reside in the same rack as a critical networking leaf switch or shared storage head.

If we blindly cut power to that rack PDU, we take down the storage cluster with it. We must maintain a real-time mapping between logical workloads and physical electrical topology (Racks, Busways, and Power Distribution Units)."

Phase 3 — Calculating the curtailment requirement and uncertainty reserve

INTERVIEWER

"You need 3 MW. How much workload must you target to guarantee that reduction?"

CANDIDATE

"Let's first examine the naive math: if a discrete curtailable workload unit sheds an average of 300 kW, the theoretical unit count is:"

N_naive = 3,000 kW / 300 kW = 10 workload units

"However, targeting exactly 3.0 MW is an operational failure. In a physical emergency:

Some servers will fail to acknowledge IPMI/Redfish commands.

Network jitter will delay command arrival to certain rack controllers.

Thermal inertia in server fans can temporarily keep power elevated.

Telemetry sensors carry calibration margins of error (± 2–5%).

To guarantee that we clear the 3 MW threshold before facility breakers trip, I will provision an explicit 20% curtailment reserve:"

P_target = 3 MW × 1.20 = 3.6 MW

"At 300 kW per workload unit, the planned target count becomes:"

N_planned = 3,600 kW / 300 kW = 12 workload units

Target Reduction	Safety Reserve (+20%)	Planned Reduction Target	Target Workload Units (300 kW ea.)
3.0 MW	+0.6 MW	3.6 MW	12 Units

"We must target at least 3.6 MW of nominal reduction to ensure that even with partial command execution failures, we comfortably exceed the 3.0 MW facility requirement."

Phase 4 — Reconciling rack-level power budgets

INTERVIEWER

"How do you translate that facility plan down to the physical rack level?"

CANDIDATE

"Let's assume the data center contains 100 physical racks, averaging 120 kW per rack:"

100 racks × 120 kW = 12,000 kW = 12 MW

"However, treating racks as uniform 120 kW blocks is dangerous. Actual power density varies wildly depending on hardware generation and accelerator density:"

Rack Category	Example Count	Average Measured Power	Curtailment Policy
Critical Service Racks	20 racks	80 kW (1.6 MW total)	Preserved: Zero shedding permitted
Flexible High-Density GPU Racks	40 racks	180 kW (7.2 MW total)	Primary: Apply hardware GPU power caps
Mixed Compute Racks	25 racks	100 kW (2.5 MW total)	Secondary: Suspend validated batch jobs
Storage & Fabric Racks	15 racks	100 kW (1.5 MW total)	Protected: Maintain quorum & heartbeat paths
Total Facility	100 racks	—	Reconciled to 12 MW Total

INTERVIEWER

"Why is real-time telemetry reconciliation critical before issuing shedding commands?"

CANDIDATE

"Because an emergency controller must never act on stale or desynchronized telemetry.

If a rack controller reports a cached reading of 180 kW, but the batch job finished 10 minutes ago and the rack is already idling at 70 kW, commanding that rack to shed 80 kW will produce near-zero physical reduction.

The central controller will believe it shed load while the facility's main electrical meter shows the deficit remains. The system must continuously reconcile intelligent PDU meters with out-of-band server telemetry."

Phase 5 — Physical power shedding mechanisms: Dispelling software myths

INTERVIEWER

"Which actions will you actually execute to drop power within five minutes?"

CANDIDATE

"I establish a strict hierarchy based on how deterministic and rapid the resulting electrical drop is:

Tier 1: Hardware-Enforced Power Caps (Immediate / ≤ 10 seconds) Directly invoking hardware power-clamping registers via out-of-band BMC/Redfish or GPU driver APIs (e.g., Nvidia nvidia-smi -pl or TPU power governors). Clamping an 8-GPU host from 700W to 350W per accelerator immediately drops wattage at the wall while keeping the host online.

Tier 2: Workload Suspension & Rapid Checkpointing (≤ 60 seconds) Suspending batch jobs whose underlying nodes can drop into deep sleep or idle states.

Tier 3: Ingress Admission Control (Immediate) Immediately rejecting incoming training and noncritical analytics jobs to prevent idle capacity from refilling.

Tier 4: Graceful Node Power-Off (≤ 120 seconds) Issuing ACPI power-downs to noncritical bare-metal compute hosts."

INTERVIEWER

"Why not simply send SIGSTOP or SIGTERM to every batch process via Kubernetes?"

CANDIDATE

"Because SIGSTOP is a userspace operating system signal—it does not govern electrical physics.

When you pause a process with SIGSTOP:

High-bandwidth GPU memory remains fully allocated and powered.

PCIe lanes and memory controllers remain in high-power performance states.

Server power supplies and cooling fans continue spinning at high RPM.

A paused server often continues drawing 50% to 70% of its peak electrical power.

Relying on userspace process signals to solve an electrical grid emergency is a fatal architectural mistake. True power reduction requires hardware-level power clamping or cutting AC draw to the power supply."

(Interviewer Note: This is an exceptional Senior/Staff-level distinction. The candidate proves that software processes and electrical current operate across different physical layers).

The interviewer pushes further

INTERVIEWER

"Now let's build the complete emergency timeline. You have exactly 300 seconds (5 minutes). What is the exact numerical curtailment plan, how do you verify delivered wattage via closed-loop telemetry, and what happens when an entire row of rack controllers drops offline halfway through the emergency?"

🔒 End of Preview: Facility Workload & Power Mechanics

You have reached the end of the public preview (Phases 1–5).

Most candidates fail power-shedding interviews by treating electrical current like CPU cycles—assuming software signals can drop physical wattage, or blowing past the 5-minute deadline due to slow metric collection. In an L5/L6 Google SRE round, proving how to enforce hardware power caps, calculate telemetry latencies, and guarantee emergency margins is what earns a Strong Hire.

🗺️ What’s Inside the Full Walkthrough: Phases 6 through 14

In the complete interview session included in our bundle, the candidate steps through the remaining 25 minutes of the mock loop:

Phase 6 — Numerical Action Portfolio & Uncertainty Math (Minutes 18–22)
- The 3.8 MW Action Portfolio: Sizing 20 GPU groups (1.6 MW via hardware caps) + 8 batch suspension groups (1.2 MW) + 4 shutdown groups (1.0 MW).
- The 10% Shortfall Stress Test: Proving that under a 10% non-additive shortfall, the portfolio still yields 3.42 MW of confirmed drop (+420 kW buffer over the 3 MW target).
Phase 7 — Control-Path Latency Budget & The 175-Second Critical Path (Minutes 22–26)
- Step-by-Step Timing: Allocating budgets across Detection (10s), Selection (5s), Dispatch (10s), Execution (60s), Verification (30s), and Fallback (60s).
- The 125-Second Safety Buffer: Proving that the total planned response (175 seconds) leaves over 2 minutes of slack before the 300-second facility deadline.
Phase 8 — Redundant Emergency Control Plane Architecture (Minutes 26–30)
- Disaggregating Control from Compute: Isolating emergency control networks from the shared production network.
- Dual-Layer Decision Fabric: Logically centralized planning with pre-authorized, autonomous rack-level agents that execute local emergency clamping if the central brain becomes partitioned.
Phase 9 — Idempotency, Generation Fencing & Race Condition Prevention (Minutes 30–33)
- Preventing Double-Shedding: Engineering unique operation IDs, generation counters, and durable action state logs (Planned → Dispatched → Verified).
- State Re-Synchronization: Handling central controller failover without blindly re-executing already applied power commands.
Phase 10 — Shared Infrastructure & Blast-Radius Mapping (Minutes 33–36)
- Cross-Dependency Traps: Preventing batch compute shutdowns from de-powering racks hosting critical storage metadata nodes or BGP border routers.
- Escalation Protocol: Handling the failure mode where eligible curtailable load cannot mathematically clear 3 MW without breaking safety SLAs.
Phase 11 — Handling Controller Failure Mid-Flight (Minutes 36–39)
- Three Partition Scenarios: Controller drops pre-dispatch, post-dispatch, and partial rack agent isolation.
- Closed-Loop Verification: Refusing to credit issued commands as delivered reductions until branch PDU telemetry confirms current drop.
Phase 12 through 14 — Production BOM, 10 Acceptance Tests & Rubric (Minutes 39–45)
- The Complete Emergency BOM: 3 MW target, 3.8 MW portfolio, 175s timeline, intelligent PDU telemetry, and isolated control fabrics.
- The 10 Mandatory Emergency Drill Tests: Fault-injecting slow sensors, unresponsive BMCs, partial shortfalls, and cold restarts.
- Official Interviewer Scorecard: How Google interviewers evaluate physical constraints, closed-loop telemetry, and operational sequencing under crisis.

🚀 Get the Full Walkthrough & Complete Interview Bundle

This scenario is just one part of the end-to-end curriculum designed to build your operational reflexes under real interview pressure.

👉 Unlock the Full Walkthrough in the Google SRE Interview Career Launchpad

Stop guessing the math under pressure. Master the simulation.

🏗️ Google SRE NALSD Scenario Simulation 10: Sovereign-Cloud Telemetry Synchronization

Why include this: Sovereign-cloud boundaries (such as Germany's BSI C5 / EU GDPR data residency requirements) strictly forbid exporting raw application traces, user metadata, or database query logs across international borders. Simultaneously, central engineering platforms require operational visibility across 100,000 microservices over intermittent WAN links subjected to recurring 8-hour network outages. This scenario tests data residency allowlisting, durable local spooling, and the fluid dynamics of draining multi-gigabyte backlogs while live production traffic continues to arrive.

The math that matters: 100,000 microservices producing 1,667 summaries/sec (3.41 MB/s / 27.3 Mbps steady-state), 8-hour outage backlog (48 million summaries = 98.3 GB), 72-hour local retention with 2-copy and 3-copy storage sizing (2.65 TB vs. 3.98 TB provisioned), and the live backlog drain formula (T_drain = B / (C - R) = 1.94 hours over an effective 140 Mbps link).

Interview simulation · Senior infrastructure engineering · Data residency compliance, durable spooling, backlog drain physics, and network partitions

One distinction upfront: this is a realistic simulated interview, not a transcript of an actual Google interview. Payload encodings, network outage durations, and link bandwidths are explicit planning assumptions. The objective is to demonstrate how a strong candidate establishes capacity envelopes and enforces international compliance boundaries—not to recite vendor tooling.

Phase 1 — Define requirements, data residency, and baseline workload

INTERVIEWER

"We operate 100,000 microservices in a German sovereign-cloud environment. We need to synchronize operational telemetry with a central global analytics platform outside Germany.

Regulations strictly mandate that raw telemetry and user-identifiable data remain within Germany, while only approved aggregate summaries may be exported. Furthermore, the cross-border network link can be completely unavailable for eight hours per day, and outages may recur before the previous backlog has finished draining.

Design the system and size its storage, bandwidth, and compute capacity."

CANDIDATE

"Before choosing a database or transport protocol, I must establish the mathematical data contract, steady-state production rate, outage backlog, and net catch-up velocity.

A statement like '480 payloads accumulate during an outage' is a common candidate trap: with 100,000 services, the generation rate is orders of magnitude larger.

I will establish this planning baseline:"

Parameter	Planning Assumption	Rationale
Microservices in Scope	100,000	Fleet-wide service count
Summary Emission Rate	1 summary / service / minute	Periodic aggregation window
Average Encoded Summary	2 KiB	Compressed metric/histogram payload
Network Outage Window	8 hours (480 minutes)	Target recurring partition horizon
Nominal WAN Link Bandwidth	200 Mbps	Dedicated cross-border pipe
Effective Link Utilization	70% (140 Mbps)	Transport framing, TCP congestion, packet loss
Local Retention Horizon	72 hours	Durability buffer before upload confirmation
Storage Overhead & Headroom	20% index + 25% safety margin	Filesystem buffers and compaction headroom
Data Residency Boundary	Germany (BSI / GDPR)	Zero raw traces or PII exported

INTERVIEWER

"Why not assume one summary per minute across the entire environment?"

CANDIDATE

"Because telemetry is generated independently by each microservice.

Across 100,000 services emitting once every 60 seconds, the aggregate production rate is:"

λ_summary = (100,000 services × 1 summary) / 60 seconds ≈ 1,666.7 summaries/second

"That is 100,000 summaries per minute.

At an average of 2 KiB per encoded summary, the steady-state data production rate is:"

R = 1,666.7 × 2 KiB ≈ 3.33 MiB/second ≈ 3.41 MB/second

"Converting this to bits per second for network sizing:"

R_bits = 3.41 MB/s × 8 = 27.28 Mbps

"The German sovereign cluster generates a sustained 3.41 MB/s (27.3 Mbps) of compressed, approved summary data before transport protocol overhead."

Phase 2 — Calculating the 8-hour outage backlog and 72-hour retention storage

INTERVIEWER

"Calculate the backlog produced when the cross-border link drops for eight hours."

CANDIDATE

"Eight hours contains 8 × 60 = 480 minutes (or 28,800 seconds).

The total number of pending summaries generated during an 8-hour outage is:"

N_outage = 100,000 services × 480 minutes = 48,000,000 summaries

"The accumulated payload volume is:"

B_outage = 48,000,000 × 2 KiB = 96,000,000 KiB ≈ 98.3 GB

"Verifying via our byte generation rate (3.41 MB/s × 28,800 s ≈ 98.3 GB), both calculations align.

The 8-hour outage produces an uncompressed payload backlog of 98.3 GB."

INTERVIEWER

"Is 98.3 GB the local disk capacity we need to provision?"

CANDIDATE

"No. 98.3 GB is only the raw delta for a single 8-hour drop.

The platform must satisfy a 72-hour local retention policy to guarantee data survival across three consecutive days of severe outages, disk failures, or remote consumer ingestion stalls.

Over 72 hours, the total payload volume generated is:"

N_72h = 100,000 × 60 × 72 = 432,000,000 summaries

S_payload = 432,000,000 × 2 KiB ≈ 884.7 GB

"Now, we must apply local replication, indexing overhead, and filesystem safety margins to derive provisioned capacity:"

Storage Component	Sizing Formula	Usable Capacity
72-Hour Raw Payload	432M summaries × 2 KiB	884.7 GB
Local Replica (2-Copy)	1 additional full copy	884.7 GB
Replicated Payload Subtotal	884.7 GB × 2	1.769 TB
Storage / Index Overhead (20%)	1.769 TB × 0.20	353.9 GB
Subtotal (Data + Index)	1.769 TB + 353.9 GB	2.123 TB
Safety Headroom (+25%)	2.123 TB × 1.25	2.65 TB

"We must provision at least 2.7 TB of usable local disk capacity across independent failure domains to guarantee that a 72-hour backlog survives a local disk failure."

Phase 3 — Local durable architecture & sovereign schema allowlists

INTERVIEWER

"How do you architect the local buffer, and how do you guarantee that detailed PII or raw logs never cross the border?"

CANDIDATE

"I structure the platform into five isolated tiers:

In-Region Telemetry Collectors: Ingest high-volume raw spans, metrics, and logs from microservices and aggregate them into 1-minute summaries entirely in German RAM.

Sovereign Export Gateway: Enforces an explicit schema allowlist (service category, minute bucket, p50/p90/p99 latency histograms, error rates, schema version).

Rejects all fields outside the allowlist by default.

Strictly drops raw query strings, IP addresses, stack traces, and tenant identifiers. Pseudonymization alone is rejected as insufficient for compliance.

Durable Local Spool: Writes immutable, sequence-numbered batches to an append-only replicated WAL across two independent nodes.

Synchronization Workers: Drain and upload batches over the external link when connectivity is healthy, applying bandwidth pacing and exponential backoff.

Remote Ingestion Endpoint: Performs idempotent deduplication based on unique batch IDs and emits durable acknowledgments."

INTERVIEWER

"Would you use SQLite with WAL mode for the local spool?"

CANDIDATE

"SQLite with WAL can work for small single-node prototypes, but at 1,667 writes/second, a single-file SQLite database becomes an fsync concurrency bottleneck.

Furthermore, a single SQLite file hosted on one VM fails our failure-domain requirement: if the hypervisor or NVMe drive crashes during the 8-hour outage, the entire backlog is lost.

We need a distributed, partitioned append-only commit log (such as Kafka, Apache Pulsar, or a multi-worker replicated queue) distributed across two availability zones within the Frankfurt sovereign region."

(Interviewer Note: This is a critical Staff-level distinction. The candidate rejects lightweight single-node toys and enforces failure-domain survivability during the outage).

The interviewer pushes further

INTERVIEWER

"Now let's tackle the mathematical core of the problem: catch-up throughput. When the 200 Mbps link comes back online after an 8-hour blackout, how long does it take to drain the 98.3 GB backlog while 3.41 MB/s of fresh production data continues to pour in? What happens if outages recur before the backlog hits zero?"

🔒 End of Preview: Workload Sizing & Data Residency Architecture

You have reached the end of the public preview (Phases 1–3).

Most candidates fail telemetry synchronization interviews by making the classic queueing error: dividing the backlog by link bandwidth (T = B / C)—completely ignoring that new data continues to generate while the pipe is draining. In an L5/L6 Google SRE round, proving the physics of the net drain rate (C - R) and modeling repeated outage duty cycles is what earns a Strong Hire.

🗺️ What’s Inside the Full Walkthrough: Phases 4 through 9

In the complete interview session included in our bundle, the candidate steps through the remaining 25 minutes of the mock loop:

Phase 4 — Catch-Up Throughput & The Live Backlog Drain Formula (Minutes 18–24)
- The Fatal Calculation Error: Proving why naive division (98.3 GB ÷ 17.5 MB/s = 1.56 hours) fails because 19.1 GB of fresh data generates during synchronization.
- The Governing Drain Equation: Deriving the net drain rate (C - R = 17.5 MB/s - 3.41 MB/s = 14.09 MB/s) to reveal the real recovery time: 1.94 hours.
- Link Capacity Verification: Proving that draining within a strict 2-hour window requires at least 195.4 Mbps nominal bandwidth—revealing that our 200 Mbps link operates on a razor-thin margin.
Phase 5 — Repeated Outages & Long-Term Duty Cycle Stability (Minutes 24–28)
- The 10-Hour Duty Cycle Trap: Modeling what happens when the link recovers for only 2 hours before failing for another 8 hours (20% availability window).
- Mathematical Recovery Stability: Proving why the link must sustain C ≥ R × (10 / 2) = 17.05 MB/s to prevent runaway backlog accumulation across cyclic brownouts.
Phase 6 — Idempotent Transport, Checksums & Reconciliation (Minutes 28–33)
- At-Least-Once Delivery Semantics: Engineering unique 64-bit batch IDs, monotonic sequence ranges, and canonical SHA-256 payload checksums.
- Safe Local Deletion Boundary: Why TCP acks and HTTP 200s are insufficient; requiring durable commit acknowledgments from the central storage engine before purging local spool blocks.
- Handling Spool Exhaustion: Implementing explicit backpressure, dynamic sampling reductions, and emergency alerting instead of silently overwriting historical data.
Phase 7 — Multi-Node Failure Domains & 3-Copy Storage Sizing (Minutes 33–37)
- Surviving Concurrent Hardware Faults: Sizing storage requirements if compliance demands surviving two concurrent storage node crashes.
- The 3-Copy Expansion: Scaling provisioned local storage from 2.65 TB (2-copy) up to 3.98 TB (3-copy) including indexing and 25% safety headroom.
Phase 8 — Cryptographic & Architectural Data Residency Controls (Minutes 37–40)
- Zero-Leakage Enforcement: Isolating internal VPC endpoints, egress firewall allowlists, and mutual TLS (mTLS) client-certificate pinning.
- Schema Versioning & Quarantine: Rejecting unapproved new microservice metric fields at the gateway rather than dropping the entire batch.
Phase 9 — Final Bill of Materials & Acceptance Criteria (Minutes 40–45)
- The Complete Production BOM: 1,667 summaries/sec collectors, 2.7 TB/4.0 TB replicated spool nodes, 17.1 MB/s sync workers, and 200 Mbps WAN links.
- Four Mandatory Operational Drills: 8-hour blackout recovery, repeated 8h/2h outage duty cycles, disk-loss failovers, and synthetic PII leakage detection.
- Official Interviewer Scorecard: The exact rubrics Google interviewers use to evaluate sovereign compliance, queueing mathematics, and recovery resilience.

🚀 Get the Full Walkthrough & Complete Interview Bundle

This scenario is just one part of the end-to-end curriculum designed to build your operational reflexes under real interview pressure.

👉 Unlock the Full Walkthrough in the Google SRE Interview Career Launchpad

What is inside the complete system:

Full-Length NALSD & Troubleshooting Transcripts: The complete, unabridged walkthroughs, mathematical proofs, and architectural blueprints for this multi-tenant accelerator fleet and other flagship scenarios.
20+ "War Room" Interview Simulations: Realistic mock scenarios equipped with the hidden Interviewer Scorecards, follow-up traps, and rubric criteria.
Linux Internals & Reliability Deep Dives: Practical drills covering D-state processes, TCP connection backlogs, kernel memory pressure, and distributed consensus.
70+ Production-Grade Coding Drills: Systems-focused Python & Go drills emphasizing concurrency, streaming, backpressure, and defensive design.
The Offer Maximizer: Negotiation frameworks and word-for-word scripts to secure top-of-band L5/L6 compensation packages.

Stop guessing the math under pressure. Master the simulation.

Top comments (1)

Ace Interviews • Oct 22

"Curious to hear from practicing SREs and Production Engineers: Which of these 10 scenarios do you think candidates struggle with most under a 45-minute timer? My bet is on the Datacenter Power Shedding (3 MW limit) or the Multi-Modal Edge KV-cache memory math."