DEV Community: Prateek Srivastava

GCP Is Running Out of N2 Machines in us-central1 — And It's Silently Breaking Production Clusters

Prateek Srivastava — Thu, 27 Aug 2026 17:42:53 +0000

🔴 Ongoing · Reports active since April 2026 · No official GCP acknowledgement

Engineers running workloads on Google Cloud Platform's us-central1 region are encountering a growing problem: GCP cannot provision new n2 VMs on demand. The autoscaler requests the machines. GCP says no. Your pods stay Pending. And the GCP status dashboard sits on green the entire time.

Google officially defines this error as occurring when "the zone does not have enough resources available to fulfill the request" — distinguishing it from quota limits, which are a separate class of problem. The error, when you eventually find it buried in GKE cluster-level logs, looks like this:

Node scale up in zones us-central1-f associated with this pod failed:
GCE out of resources. Pod is at risk of not being scheduled.

The issue has been documented across Google's own developer forums since April 2026, with fresh reports appearing through August. It has affected n1, n2, and e2 instance types across all availability zones in us-central1 — zones a, b, c, and f — making the standard "try a different zone" advice ineffective when the shortage is regional.

What Engineers Are Reporting on Google's Own Forums

The reports started quietly in spring and have grown louder through summer. These are direct quotes from Google Developer Community discussion threads:

"For many years I've been deploying Compute Engine VMs to us-central-1 region but for the past couple days I've constantly received error ZONE_RESOURCE_POOL_EXHAUSTED."
— Kevin_Koehler, April 13, 2026 · Google Developer Community

"Assumed it was a temporary resource shortage but it's been happening consistently for the past week, which has been very frustrating."
— nju, May 18, 2026 · Google Developer Community

"I tried zones a, b, c, and f. Same issue. I tried e2, n1, n2. Same issue... I guess GCP doesn't want my money anymore 🤷"
— standev, August 13, 2026 · Google Developer Community

⚠️ No Official Response. As of the time of writing, no Google employee has responded to these forum threads, and no incident has been posted to the GCP Status Dashboard for this issue. The gap between what users experience and what Google publicly acknowledges is the defining characteristic of this problem.

The Failure Mode That Looks Like Slowness

What makes this particularly dangerous for production systems is how the failure presents itself. When GCP cannot provision a requested machine type, it does not return an error to your application. The GKE cluster autoscaler keeps retrying silently. Your worker pods stay in Pending state. Your job scheduler marks tasks as still running. Your monitoring dashboards show activity. Nothing appears broken.

The only place the truth surfaces is in GKE cluster-level logs — not in your application logs, not in your job scheduler UI. Most engineers only look there when something is visibly broken, which by then is hours later.

In distributed batch workloads on GKE, a common pattern is a lightweight driver pod that coordinates the job and schedules work across executor (worker) pods that do the actual computation. When this capacity issue strikes, the driver pod starts successfully — it's small and fits on an existing node. The executor pods, which need the new n2 VMs the autoscaler is trying to provision, never arrive. From the outside, the job appears to be running. Internally, it's doing nothing.

Time	What Happens
T + 0 min	Driver pod created, job shows as RUNNING. Everything looks normal.
T + ongoing	Autoscaler requests new n2 nodes — GCP cannot fulfil. Worker pods remain Pending. No error surfaced to the app layer.
T + hours later	GCP eventually finds capacity — workers schedule, job completes. Having waited hours to do minutes of work.

"The driver node got created around 19:30 UTC but the executors were allocated only around 00:30 UTC — 5 hours later. This is the case with all the tasks."
— Platform engineer's documented incident report

Google's own GKE documentation acknowledges this scenario, stating the resolution is to "try a different zone, or try again later." When the shortage is regional and persistent — spanning all zones for hours — neither of those options helps in the moment.

Which Machine Types Are Affected

Based on community forum reports, the issue is not limited to a single machine family:

Machine Family	Types Reported	Zones (us-central1)	Community Status
n2	n2-standard, n2-highmem (all sizes)	a, b, c, f	❌ Most frequently reported
n1	n1-standard, n1-highmem	a, b, c, f	❌ Also reported
e2	e2-medium, e2-standard	Some zones	⚠️ Intermittent reports
n2d	n2d-highmem, n2d-standard	a, b, c, f	⚠️ Some teams found better availability
c4	c4-highmem, c4-standard	—	✅ No shortage reports found (Aug 2026)

Three Actions You Can Take Today

1. Stop autoscaling from zero on SLA-critical node pools

When your minimum node count is zero, a capacity crunch becomes an indefinite stall — the autoscaler cannot get new nodes, and no existing nodes are available either. Raising the minimum floor means your workloads always have somewhere to land.

# Before — zero floor, entirely dependent on autoscaler from scratch
min_node_count = 0
max_node_count = 10

# After — warm baseline; autoscaler supplements from here
min_node_count = 4
max_node_count = 10

Always commit this change through your IaC tooling (Terraform / Terragrunt). A manual GCP Console change is not reflected in state and will be silently overwritten on the next apply.

2. Try a different machine type

Google's official troubleshooting documentation explicitly lists "selecting a different machine series or type" as a recommended workaround for ZONE_RESOURCE_POOL_EXHAUSTED. Some teams have had better provisioning success with n2d compared to n2 during this period.

gcloud beta container node-pools update YOUR-NODEPOOL-NAME \
  --project      YOUR-PROJECT-ID \
  --cluster      YOUR-CLUSTER-NAME \
  --region       us-central1 \
  --machine-type n2d-highmem-16

Plan for approximately 30 minutes of node pool downtime. Schedule this during a quiet window and validate the new machine type on a non-production cluster first.

3. Alert on pod pending time — not just job failure

This failure mode presents as slowness, not an error. Your job scheduler will not raise an alert — it sees the job as running. Set a Kubernetes-level alert on pod pending duration.

# List all Pending pods across namespaces
kubectl get pods -A --field-selector=status.phase=Pending

# Inspect a specific pod — look in Events for the root cause
kubectl describe pod <pod-name> -n <namespace>
# GCE out of resources  →  capacity problem (this article)
# Insufficient cpu/memory  →  resource request too large for available nodes

The Long-Term Answer: Consider Migrating to C4

The C4 machine series — powered by Intel Emerald Rapids processors and Google's Titanium offload chips — is Google's current-generation compute offering, and no documented community reports of ZONE_RESOURCE_POOL_EXHAUSTED for C4 have been found as of August 2026.

Spec	n2-highmem	c4-highmem
Processor	Intel Cascade Lake / Ice Lake	Intel Emerald Rapids (5th Gen Xeon)
Max vCPUs	128	192
Max memory	864 GB	1.5 TB DDR5
Network bandwidth	Up to 100 Gbps	Up to 200 Gbps
Price-performance	Baseline	Up to 25% better vs C3

💡 Migration approach: The safest path is to create a new C4 node pool alongside your existing n2 pool, validate your workloads on the new pool, then drain and delete the old one. This avoids any single-pool downtime and lets you roll back cleanly if something doesn't behave as expected.

Before migrating, validate on a pre-production cluster first. C4 uses Hyperdisk as its storage backend by default rather than standard persistent disks — confirm compatibility before rolling to production.

Key Takeaways

1. The GCP status page is not your monitoring.
ZONE_RESOURCE_POOL_EXHAUSTED does not appear as a GCP incident. Monitor your cluster directly — pod pending duration is a more reliable early-warning signal than any external status page.

2. Minimum node count zero is a risk for production batch.
Autoscaling from zero is cost-efficient under normal conditions. Under capacity constraints it means indefinite waits. For jobs with SLAs, maintain a warm baseline.

3. Multi-zone helps — regional shortages need a different response.
Zone diversity protects against single-zone hardware failures. When the shortage is regional, the only effective responses are machine type diversity or capacity reservations made in advance.

4. Know your machine type fallback before you need it.
Google explicitly recommends switching machine series as a workaround. Have a tested alternative machine type ready in your runbook — the wrong time to figure it out is during a production incident at 3 AM.

The frustrating reality of ZONE_RESOURCE_POOL_EXHAUSTED is that it sits in a grey zone: widespread enough to affect multiple teams across months, specific enough that Google doesn't surface it as a regional incident. The community has been documenting it since April. The status page has stayed green throughout.

If you're running SLA-critical workloads in us-central1 on n2 — especially with a minimum node count of zero — the fixes above are worth reviewing this week, not after the next unexplained multi-hour delay.

Sources

If you’re dealing with something like this at work — or you have a war story of your own — I’m on LinkedIn and I actually respond.
I publish one production incident breakdown every week. Subscribe free if you want it directly in your inbox — no tutorials, no fluff:
Let’s Stay in Touch 🪂 Connect with me on: LinkedIn, Instagram, Medium

Claude Went Down Today. And It Says More About AI Than You Might Think !!!!

Prateek Srivastava — Mon, 24 Aug 2026 08:58:45 +0000

Claude Went Down Today — And It Says More About AI Than You Might Think

If you tried using Claude today and suddenly started seeing errors, failed requests, or responses that wouldn't complete, you weren't alone.

On August 24, 2026, Anthropic experienced a service disruption affecting multiple Claude models and services. Anthropic reported elevated errors and said it had identified the cause and was working on a fix.

At the time of writing, the situation has improved and Anthropic's status page shows its services as operational.

But the interesting part isn't simply that Claude went down.

The interesting part is how much this kind of outage matters now.

A few years ago, an AI chatbot going offline would have been an inconvenience.

Today, it can interrupt software development, automation, research, content creation, and entire business workflows.

And developers should pay attention to that.

Claude Isn't Just a Chatbot Anymore

This is probably the biggest change.

People don't use Claude only to ask questions anymore.

Developers use Claude to:

Write and review code
Debug applications
Refactor large codebases
Generate tests
Analyze repositories
Work with Claude Code
Process documents
Automate repetitive tasks
Build applications through the Claude API

For many developers, Claude has become part of the development environment.

So when Claude becomes unavailable, the problem isn't necessarily:

"My AI chatbot isn't working."

It can be:

"Part of my development workflow just stopped."

That's a very different situation.

AI Is Quietly Becoming Infrastructure

We often compare AI models based on intelligence.

Which model is better at coding?

Which one reasons better?

Which one produces better results?

Which one wins the latest benchmark?

Those comparisons are useful, but there's another question that becomes increasingly important as we use AI in production:

Can I depend on this service being available when I need it?

That's an infrastructure question.

And AI providers are increasingly becoming infrastructure providers.

If your application depends on Claude's API, then Claude's availability becomes part of your application's availability.

That's something developers shouldn't ignore.

The Single-Provider Problem

Imagine building an application where every important AI operation goes through one provider.

Everything works perfectly.

Until it doesn't.

Your code may be perfectly written.

Your infrastructure may be healthy.

Your database may be online.

Your users may have a working internet connection.

But if your AI provider is unavailable, a critical feature can still fail.

That's the downside of tightly coupling an application to a single AI provider.

It doesn't mean you should avoid Claude.

It means you should design your system with failure in mind.

What Happens When Claude Goes Down?

This is the question every developer building an AI-powered application should ask.

Not:

"What if the model gives a bad answer?"

But:

"What if I can't reach the model at all?"

These are very different failure modes.

A bad response can potentially be validated, rejected, or corrected.

An unavailable API gives you nothing.

Your application needs to know how to handle that.

For example, a basic architecture might look like this:

User Request
     ↓
AI Service
     ↓
Claude API
     ↓
Success

That's simple.

But a more resilient architecture might look like:

User Request
     ↓
AI Gateway
     ↓
Claude
     ↓
If unavailable
     ↓
Fallback Model
     ↓
If unavailable
     ↓
Cached / Degraded Response

You don't necessarily need a complicated architecture for every project.

But for production systems, the principle is important:

Don't make one AI provider a single point of failure.

Fallbacks Are Becoming Important

One of the easiest improvements is having a fallback model.

For example, your application might use Claude as the primary model.

If requests start failing, your system can route certain requests to another model.

The fallback doesn't have to be identical.

Maybe the primary model handles complex reasoning while the fallback handles simpler requests.

Maybe one provider handles coding while another handles classification or summarization.

The exact architecture depends on the application.

The important thing is having a plan.

Retries Aren't Enough

A common response to API failures is simply:

Request failed
→ retry
→ retry
→ retry

That's not always a good strategy.

During a real outage, thousands of applications may already be retrying requests.

If every application aggressively retries, the additional traffic can make the situation worse.

Instead, use:

Exponential backoff
Reasonable retry limits
Timeouts
Circuit breakers
Rate-limit handling
Idempotency where appropriate

Your application should be able to recognize:

"The provider is having trouble. Stop hammering it."

Graceful Degradation Matters Too

Not every AI feature needs to bring down the entire application.

Suppose your application uses AI to generate summaries.

If the AI provider is unavailable, perhaps users can still:

Upload documents
Browse existing summaries
Save their work
Queue the request
Try again later

The application doesn't need to become completely useless.

This is what graceful degradation looks like.

The AI feature can fail while the rest of the product continues working.

That's much better than:

AI API unavailable
        ↓
500 Internal Server Error
        ↓
Everything is broken

Monitoring Shouldn't Start After an Outage

Another lesson from incidents like this is monitoring.

If you're building with an AI API, don't only monitor whether your own server is healthy.

Monitor the AI dependency too.

At minimum, keep an eye on:

Request success rate
Error rate
Latency
Timeouts
Rate limits
Token usage
Provider availability

And ideally, your system should distinguish between:

My application is broken

and:

My AI provider is having an incident

Those are completely different problems.

Claude Code Makes This Even More Interesting

The impact of Claude outages is particularly interesting for developers because of Claude Code.

AI coding tools have changed the developer workflow.

Instead of asking an AI to complete one line of code, developers can increasingly give an agent a larger task:

Understand this repository.
Find the problem.
Modify the relevant files.
Run the tests.
Fix any failures.

That's a much deeper integration with the development process.

When that tool becomes unavailable, the impact isn't simply losing a chatbot.

It's losing part of your development workflow.

And that's exactly why reliability is becoming so important.

The "Best Model" Isn't Always the Best Choice

Here's another thought.

Suppose Model A is slightly better than Model B.

But Model A is your only provider.

Model B is slightly less capable, but your system can automatically fail over to it.

For a production application, Model B might actually provide a better overall user experience.

Because users don't care about benchmark scores when your application is returning errors.

They care whether the product works.

That's why the future of AI engineering probably isn't going to be about picking one model and sticking with it forever.

It's going to be about model orchestration and resilience.

AI Reliability Is Becoming a Feature

We've traditionally thought about AI companies competing on:

Model intelligence
Context windows
Reasoning
Coding ability
Multimodal capabilities
Cost

Increasingly, we'll also care about:

Uptime
Latency
API stability
Rate limits
Incident response
Observability
Developer experience

That's actually a sign of progress.

When people start caring about uptime, it means the technology has become important enough to depend on.

What Developers Can Do Today

If you're building an application around Claude or any other AI provider, today's incident is a good excuse to review your architecture.

Ask yourself:

1. What happens if the API returns 500 errors?

2. What happens if requests start timing out?

3. What happens if the provider is unavailable for an hour?

4. Do I have a fallback model?

5. Can users continue using the non-AI parts of my application?

6. Do I have monitoring and alerts?

7. Am I retrying intelligently?

8. Can important AI results be cached or queued?

If you don't have good answers yet, that's okay.

But it's worth thinking about before the next outage rather than during it.

The Bigger Lesson

I'm not particularly interested in using today's incident to argue that Claude is bad.

Every major technology platform will eventually have outages.

The more interesting lesson is what the outage reveals about the industry.

AI has moved beyond being a novelty.

It's becoming infrastructure.

Developers are building applications on top of these models.

Companies are integrating them into internal workflows.

People are using them to write software.

And once something becomes infrastructure, reliability matters just as much as capability.

That's the real story behind today's Claude outage.

Not that an AI service stopped working for a while.

But that we've reached a point where an AI service going offline can stop real work from happening.

And if you're building with AI today, that's something worth designing for.

Final Thoughts

The next time someone asks:

"Which AI model should I use?"

Maybe the answer shouldn't only be about benchmarks, coding performance, or reasoning ability.

Ask another question:

"What happens when that model goes down?"

Because eventually, it will.

And the applications that survive won't necessarily be the ones using the smartest model.

They'll be the ones designed to keep working when the smartest model isn't available.

Sources

Anthropic Claude Status — Official service status and incident history:
https://status.claude.com/
Notebookcheck — Coverage of the Claude outage and elevated errors affecting multiple models:
https://www.notebookcheck.net/Claude-faces-another-outage-as-errors-hit-multiple-models.1376666.0.html
Android Authority — Coverage of the August 24 Claude disruption:
https://www.androidauthority.com/claude-outage-august-24-3702012/
Economic Times — Coverage of user reports and the Claude service disruption:
https://m.economictimes.com/news/new-updates/claude-down-today-users-report-issues-as-downdetector-shows-big-spike-heres-latest-update/articleshow/133456017.cms

What do you think?

Has your development workflow become too dependent on AI providers?

I'd be interested to hear how you're handling AI outages — fallback models, queues, caching, or simply switching back to writing code manually.

A Windows Service is Down. Now What?

Prateek Srivastava — Sun, 23 Aug 2026 19:29:55 +0000

It doesn't matter if it's SQL Server, IIS, a background agent, or a custom app — when a Windows service goes down, the investigation is always the same five moves. Learn the pattern once, apply it to anything.

🚫 Don't restart first. Restarting a service without knowing why it stopped can hide a real problem — a failing update, an exhausted host, or an automation script that will stop it again in the next cycle. Spend 5 minutes on the evidence first.

Step 01 — Confirm Current State

Your first question isn't "why did it stop?" — it's "is it still stopped?" Monitoring systems have polling intervals; by the time an alert reaches you, the service may have already recovered. Establish ground truth before taking any action.

Get-Service "ServiceName1", "ServiceName2" |
    Select-Object Name, Status, StartType

Two things matter: Status (is it actually down right now?) and StartType (is it still set to Automatic? OS updates occasionally reset this).

🔵 SQL Server example:
Get-Service MSSQLSERVER, SQLSERVERAGENT, MsDtsServer130 |
    Select-Object Name, Status, StartType
All three came back Running / Automatic — already self-recovered before anyone looked. This told us the alert was valid but not actionable, and shifted focus to understanding root cause.

Step 02 — Map Service Transitions with Event ID 7036

Windows logs every service state change — stopped, running, paused — as Event ID 7036 in the System log. Pull the last 20 transitions and sort them chronologically. This gives you the precise stop-and-start sequence and exact timestamps.

Get-WinEvent -FilterHashtable @{
    LogName='System'; Id=7036
} | Where-Object {
    $_.Message -match 'Your Service Name'
} | Select-Object TimeCreated, Message |
    Sort-Object TimeCreated -Descending |
    Select-Object -First 20

Read the output bottom-up (oldest first). Look for the last time it went from running → stopped, and whether it came back on its own or stayed down. Multiple stop/start cycles in quick succession often mean a crash-and-restart loop.

🔵 SQL Server example:
Get-WinEvent -FilterHashtable @{LogName='System';Id=7036} |
    Where-Object {$_.Message -match
      'SQL Server Agent $MSSQLSERVER$|SQL Server $MSSQLSERVER$|SQL Server Integration Services 13.0'} |
    Select-Object TimeCreated, Message |
    Sort-Object TimeCreated -Descending |
    Select-Object -First 20
The output showed all three services stopped at 10:20 AM IST and returned to running at 10:27 AM IST — a 6-minute outage that overlapped exactly with the alert timestamp.

Step 03 — Look for System-Level Triggers

A service rarely stops in isolation. Before blaming the application, check whether the host itself caused the stop. Four event IDs tell this story:

Event ID	Meaning
1074	Intentional shutdown/restart — includes initiating process (TrustedInstaller, shutdown.exe, Ansible), user, and reason code
6005	Event log service started = OS finished booting. Your "server is up" timestamp
6006	Event log service stopped = OS going down. Pair with 6005 to measure reboot windows
6008	Unexpected/dirty shutdown — previous shutdown was unclean. A crash or power loss leaves this

Get-WinEvent -FilterHashtable @{
    LogName='System'; Id=1074,6005,6006,6008
} | Select-Object TimeCreated, Id, Message |
    Sort-Object TimeCreated -Descending |
    Select-Object -First 20

# For recent 1074s with full detail:
Get-WinEvent -FilterHashtable @{LogName='System';Id=1074} |
    Where-Object {$_.TimeCreated -gt (Get-Date).AddHours(-24)} |
    Format-List TimeCreated, Message

🔵 SQL Server example:

Event 1074 revealed two reboots in the same maintenance window. The first was triggered by o9ansibleuser via Ansible. The second was chained by TrustedInstaller.exe with reason "Operating System: Upgrade (Planned)" — Windows Update had quietly queued a second reboot on top of the Ansible reboot. Without checking Event 1074, this would have looked like an unexplained outage.

💡 If you find a 6008 (dirty shutdown) near your service stop, the host itself crashed — investigate the host before the service.

Step 04 — Read the Application's Own Logs

Windows Event logs tell you when the service stopped. The application's own logs tell you why. Every serious Windows service writes its own log — find it and read the entries immediately before the service stopped.

# Common log locations:
# SQL Server → C:\Program Files\Microsoft SQL Server\MSSQL{ver}.{instance}\MSSQL\Log\ERRORLOG
# IIS        → C:\inetpub\logs\LogFiles\W3SVC1\

# For services that write to Windows Application log:
Get-WinEvent -FilterHashtable @{LogName='Application';Level=1,2} |
    Where-Object {$_.ProviderName -match 'YourService'} |
    Select-Object TimeCreated, Message |
    Sort-Object TimeCreated -Descending |
    Select-Object -First 20

You're looking for one of three signatures:

Error/exception before stop → crash
Clean shutdown message → planned stop
Nothing unusual → external trigger (reboot killed the process)

🔵 SQL Server example — ERRORLOG:

SQL Server rotates its log on every restart. Find the file that covers your outage window by LastWriteTime:
# Find the right log file
Get-ChildItem "C:\Program Files\Microsoft SQL Server\MSSQL13.MSSQLSERVER\MSSQL\Log\ERRORLOG*" |
    Select-Object Name, LastWriteTime, Length |
    Sort-Object LastWriteTime

# Read the last 50 lines of the relevant file
Get-Content "...\MSSQL\Log\ERRORLOG.4" | Select-Object -Last 50
The critical line at the tail of our log:
SQL Server is terminating in response to a 'stop' request from Service Control Manager.
That's a clean stop — no crash, no error. Case closed on fault; focus shifts to who stopped it and why.

⚠️ SQL ERRORLOG uses local server time, not UTC. Convert your alert timestamp before searching.

Step 05 — Classify the Stop, Then Decide

By now you have enough to classify into one of three categories:

🔴 Crash

Exceptions/errors in app log immediately before stop
Event 6008 (dirty shutdown) present
Dump files in the app's log directory
No clean stop message

Response: Don't just restart — find the root cause first. Check memory (Get-Process), disk (Get-PSDrive), dump files. Restarting a crashing service without fixing the cause fails again.

🟢 Clean Stop

"Stop request from SCM" in app log
Config/agent disabled just before stop
No preceding errors — manual or scripted

Response: Safe to restart, but find out who stopped it. Was it automation? Is it expected to auto-restart?

🟡 External Trigger

Event 1074 near the stop time
6005/6006 reboot pair visible
Ansible, Windows Update, manual reboot

Response: The service is a passenger — the host was the cause. Verify the reboot was planned, confirm all services are back, close the ticket.

Going Further: From Reactive to Proactive

🔕 Maintenance Windows — Before any planned reboot, open a monitoring suppression. Automate it as a pre-task in your Ansible playbook, not a manual step someone might forget.

📋 Parse App Logs Continuously — Ship application logs to a central store. Alert on terminating, severity 20+, exception before a service hits the floor.

🧠 Watch Memory Before It Matters — Many crashes are preceded by hours of memory pressure. A service paging to disk for 4 hours before crashing gave you warning you didn't act on.

🤖 Tag Automation in Monitoring — Have Ansible write a comment to your alert system before rebooting. A ticket that says "Ansible reboot — expected" closes in seconds. One without context takes 15 minutes.

📅 Track TrustedInstaller Reboots — Windows Update's chained reboots are invisible unless you watch Event 1074. Add a post-patch check that logs TrustedInstaller-initiated reboots to your CMDB.

📈 Trend Restart Frequency — A service that restarts once a quarter is healthy. One that restarts three times in a week has a problem. Track restart counts over time — the trend tells you before a crash does.

Quick Reference: The Diagnostic Playbook

Save this to your runbook.

# 1 — Current state
Get-Service "ServiceA","ServiceB" | Select-Object Name,Status,StartType

# 2 — Service transitions
Get-WinEvent -FilterHashtable @{LogName='System';Id=7036} |
    Where-Object {$_.Message -match 'YourServiceName'} |
    Select-Object TimeCreated,Message | Sort-Object TimeCreated -Descending |
    Select-Object -First 20

# 3 — Shutdown events
Get-WinEvent -FilterHashtable @{LogName='System';Id=1074,6005,6006,6008} |
    Select-Object TimeCreated,Id,Message | Sort-Object TimeCreated -Descending |
    Select-Object -First 20

# 4 — App event log (errors/criticals)
Get-WinEvent -FilterHashtable @{LogName='Application';Level=1,2} |
    Where-Object {$_.ProviderName -match 'YourService'} |
    Select-Object TimeCreated,Message | Select-Object -First 20

# 5 — Server uptime
(Get-Date) - (Get-CimInstance Win32_OperatingSystem).LastBootUpTime

# SQL Server bonus — find the right ERRORLOG
Get-ChildItem "C:\Program Files\Microsoft SQL Server\MSSQL13.MSSQLSERVER\MSSQL\Log\ERRORLOG*" |
    Select-Object Name,LastWriteTime | Sort-Object LastWriteTime

💡 The three root causes — crash, clean stop, external trigger — each have a different recovery path. Classify before you act, and you'll never restart a service into the same problem twice.

If you're dealing with something like this at work — or you have a war story of your own — I'm on LinkedIn and I actually respond.

I publish one production incident breakdown every week — real commands, real dead ends, real fix. Follow on Medium if you want the next one in your feed: prateeksrivastav598.medium.com

I Managed Kubernetes Across AWS, Azure, and GCP Simultaneously — Here's What Nobody Tells You

Prateek Srivastava — Fri, 21 Aug 2026 19:11:46 +0000

The day the same Helm chart behaved differently on three clouds in the same hour was the day I stopped trusting "cloud-agnostic" as a real thing.

Nobody plans to manage Kubernetes on three clouds at the same time. It happens gradually — an acquisition brings an Azure footprint, a new business unit commits to GCP, and you already run EKS. Then one morning a page fires and you're staring at three terminal windows, three different cluster behaviours, one very angry incident bridge.

I've spent the better part of two years in exactly that situation. What follows isn't a comparison chart you can find on any vendor blog. It's the stuff I learned the hard way — the silent differences that don't show up until something breaks in production at 2 AM.

📟 The Incident That Started It All

We deployed the same Helm chart — identical values, same image tag — to EKS, AKS, and GKE within the same 20-minute release window. EKS came up healthy. AKS came up healthy. GKE's pods started, ran for about 90 seconds, then began OOMKilling in a loop. Same YAML. Same limits. Three different outcomes. Four hours to find why. That story is lesson #3.

1. Networking: The Biggest Lie — "Pods Are Just Pods"

This is where most multi-cloud pain originates. Every Kubernetes tutorial tells you pods get their own IPs and can talk to each other. What it doesn't tell you is that where those IPs come from completely changes your blast radius, subnet planning, security posture, and the failure modes you'll face at 3 AM.

EKS: Your pods eat your VPC subnet IPs

EKS uses the AWS VPC CNI by default. Every pod gets a real, routable IP from your VPC subnet. This sounds fine until you have a /24 subnet and someone deploys a service with 200 replicas. The error isn't "out of IPs" — it's an ENI attachment timeout.

The harder gotcha: each EC2 instance type has a max ENI count × max IPs per ENI cap. A t3.medium can hold 3 ENIs × 6 IPs = 18 pod IPs max, minus 1 for the node. Pods sit Pending on nodes with plenty of CPU and RAM — the constraint is invisible to the scheduler.

# Check ENI and IP capacity on a node
kubectl describe node <node-name> | grep -A5 "Allocatable"

# Check aws-node daemonset for ENI attachment errors
kubectl logs -n kube-system -l k8s-app=aws-node --tail=50 | grep -i "err\|fail\|exhaust"

Fix: Enable VPC CNI prefix delegation — one ENI prefix = 16 IPs instead of 1.

AKS: Azure CNI pre-allocates IPs whether you like it or not

AKS with Azure CNI reserves IPs per node based on --max-pods (default 30) — before any pod runs. A 100-node cluster pre-consumes 3,000 VNet IPs. Use Azure CNI Overlay mode if IP-constrained.

GKE: Clean until you need to peer networks

GKE uses alias IP ranges, isolated from your main VPC. Clean story — until you peer your GKE VPC with on-prem and discover the pod CIDR overlaps your on-prem range, because you let GKE auto-assign ranges six months ago. Always explicitly define secondary ranges.

	EKS	AKS	GKE
Pod IP source	VPC subnet (real IPs)	VNet or overlay	Alias IP ranges
IP exhaustion risk	High — ENI limits	High — pre-allocated	Low — isolated
Common 3 AM failure	ENI attachment timeout	VNet IP exhaustion	CIDR overlap during peering

2. IAM + RBAC: Three Identity Models, One Misconfiguration Away from Disaster

EKS: The IRSA silent fallback trap

If your IRSA annotation has a typo, the assume-role call silently fails and the pod falls through to the node's instance profile — often over-permissive. You won't see an error. Always verify:

kubectl exec -it <pod-name> -- aws sts get-caller-identity

I once spent three hours debugging why a pod had too much S3 access. A typo in the annotation. Silent fallback to the node's IAM role. The pod happily worked — with the wrong permissions.

AKS: Workload Identity federation — fragile on setup

Three pieces must align exactly: Managed Identity, federated credential (exact OIDC issuer URL, including trailing slashes), and the Kubernetes service account annotation. One mismatch = silent 401.

GKE: Node rotation breaks implicit identity

Enabling Workload Identity on a node pool removes the node's default Google service account. Any workload relying on implicit node identity breaks on the next node rotation. Audit before enabling.

3. Memory + cgroups: Why the Same Container OOMKilled on GKE but Not on EKS

Here's the opening incident. Same image, same limits, EKS and AKS healthy, GKE OOMKilling every 90 seconds.

GKE had moved to containerd with cgroup v2 on newer node images. EKS and AKS were still on cgroup v1.

Our Java service used JVM ergonomics to auto-detect heap size:

cgroup v1: JVM reads from /sys/fs/cgroup/memory/memory.limit_in_bytes
cgroup v2: JVM reads from /sys/fs/cgroup/memory.max

Our old JVM didn't handle cgroup v2. It read host memory (64GB) instead of the container limit (2GB), allocated an 8GB heap into a 2GB container, and OOMKilled within 90 seconds of starting.

# Check which cgroup version the container sees
kubectl exec -it <pod-name> -- cat /proc/1/cgroup
# cgroup v1: "12:memory:/kubepods/..."
# cgroup v2: single line "0::/"

# Check what heap the JVM actually allocated
kubectl exec -it <pod-name> -- java -XshowSettings:all -version 2>&1 | grep -i heap

# See OOMKill events across all namespaces
kubectl get events --field-selector reason=OOMKilling -A --sort-by='.lastTimestamp'

Fix: JDK 15+ (native cgroup v2 support) or JDK 11 with -XX:+UseContainerSupport. Always explicitly set -Xmx and -Xms. This isn't just a Java problem — Go's GOMAXPROCS and Python's multiprocessing.cpu_count() have the same pattern.

	EKS	AKS	GKE
Default node OS	Amazon Linux 2023	Ubuntu 22.04 / Azure Linux	Container-Optimized OS
cgroup version	v2 (AL2023), v1 (AL2)	v2 (Ubuntu 22.04+)	v2 (COS since mid-2022)

4. Storage: The AZ-Pinning Trap

Block storage (EBS / Azure Disk / Persistent Disk) is AZ-specific on every cloud. Your PVC and the pod using it must be in the same AZ.

The scenario I've watched happen three times:

Stateful pod uses a PVC in AZ-A. Traffic grows. Cluster Autoscaler spins up nodes in AZ-B and AZ-C. New pods stay Pending — PVC is in AZ-A, the new nodes aren't. On-call engineer stares at CPU and memory graphs seeing nothing wrong.

# The event buried in pod describe that tips you off
kubectl describe pod <pending-pod> | grep "had volume node affinity conflict"
# "0/12 nodes are available: 8 node(s) had volume node affinity conflict"

Fix: Use volumeBindingMode: WaitForFirstConsumer on all storage classes. Delays PV creation until pod scheduling — volume always lands in the same AZ as the pod.

5. Load Balancers: Same Service Type, Three Different Outcomes

Each cloud's controller uses completely different annotation namespaces for the same intent:

annotations:
  # EKS — NLB instead of legacy Classic ELB
  service.beta.kubernetes.io/aws-load-balancer-type: "external"
  service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"

  # AKS — internal (private) load balancer
  service.beta.kubernetes.io/azure-load-balancer-internal: "true"

  # GKE — completely different namespace
  networking.gke.io/load-balancer-type: "Internal"

⚠️ We once deployed a chart to GKE with only AWS annotations. GKE ignored them and created a public Load Balancer. An internal API was internet-reachable for 40 minutes. SolarWinds monitoring caught it. It's the kind of miss that ends careers if the service is sensitive.

Rule: Validate LB visibility (internal vs public) on all three clouds in CI.

6. Autoscaling: Cluster Autoscaler Is Not the Same Everywhere

EKS scale-up path: CA detects unschedulable pods → calls ASG → ASG launches EC2 → joins cluster → kubelet registers → pods schedule. That's 4–8 minutes minimum.

GKE Node Auto Provisioning (NAP): Managed by Google, reacts faster, can create new node pools automatically. Gotcha: NAP creates pools with labels and taints you didn't ask for. Pods without matching tolerations won't schedule — even though the cluster "scaled up" in your dashboards.

What works across all three: Layer KEDA (application-level scaling, seconds) under Cluster Autoscaler (node-level, minutes). Scale pods first; nodes are the last resort.

What I'd Tell Myself Before Starting All Over

Multi-cloud Kubernetes isn't twice the work — it's about six times the work, because every subtle difference compounds.

Standardise observability first. Before you standardise deployments, standardise how you look at your clusters. Same dashboards, same alert expressions, same log structure. When the incident fires at 2 AM, you want muscle memory, not translation overhead.

Document differences, not just similarities. Your runbooks should say "on EKS, check X; on AKS, check Y; on GKE, check Z" — not pretend the clouds are the same.

Test Helm charts on all three before every major release. 20 minutes of CI is cheaper than one 4-hour incident.

Never trust "cloud-agnostic" on the label. Especially networking, storage, and identity. Those break silently rather than loudly.

The day you stop being surprised by cloud differences is the day you've actually become a multi-cloud SRE. Everything before that is just surviving the learning curve.

If you're just starting this journey: keep a war journal. Every weird behaviour, every gotcha that cost you an hour — write it down. In six months it's worth more than any certification.

Follow me on Medium for more incident stories from production.

Our EKS Node Crashed and Four Auto-Recovery Mechanisms All Failed. Here's Why.

Prateek Srivastava — Thu, 20 Aug 2026 11:56:58 +0000

⚡ TL;DR: An EKS worker node running 30+ pods crashed at 08:43 IST on Aug 16. EC2 Auto Scaling didn't act. EKS Node Auto Repair was never enabled. Cluster Autoscaler had no headroom (Min=Max=6). Our memory alert fired 82 seconds after the node was already dead. We fixed it manually at 12:30 IST — 4 hours later. Every fix is one or two CLI commands. Here's the full breakdown.

When an EKS worker node goes down, Kubernetes is supposed to handle it. That's the whole pitch — self-healing infrastructure. Pods evict, workloads reschedule, the cluster heals itself.

In a recent incident, a node running 30+ pods crashed at 08:43 IST. None of the four auto-recovery mechanisms we had in place did anything. The node sat NotReady for almost 4 hours. We discovered it 82 seconds after it died — from a memory alert that fired when the node was already gone — and fixed it manually at 12:30 IST.

The interesting part wasn't that the node crashed. It was that four different layers of protection failed for four completely different reasons.

Here's exactly what failed and why.

The Node That Looked Healthy Until It Didn't

ip-10---*.ec2.internal had been running since Jun 28. From Kubernetes' perspective, it was Ready. CPU requests at ~97%. Memory requests at ~69%. Nothing alarming.

The actual picture was very different:

Allocated resources:
  Resource    Requests       Limits
  cpu         ~97%           ~545%
  memory      ~69%           ~358%

358% memory limit overcommit. Kubernetes scheduled 37 pods there because requests showed 69% — but if those pods pushed toward their limits, the node had no chance. The requests were lies; the limits were the truth. And the truth said this node was 3.5x overloaded.

Underneath the Ready status, the containerd runtime had been in a degraded state since Jul 20, when the OOM killer had hard-killed a MongoDB process. The kernel had logged two containerd-shim deadlocks (Jul 1 and Jul 29) that the node had somehow absorbed. Neither showed up anywhere in monitoring. The control plane saw nothing wrong.

On Aug 16 at 08:22 IST, a service responsible for processing audit events started leaking memory. ELK showed node memory at 81.6% at 08:22:57 IST. By 08:43 IST the service had grown from 2.0GB to 6.2GB — a 4.2GB spike in 21 minutes — pushing the node from ~81% to ~96%.

The memory pressure triggered a third containerd-shim deadlock on the already-corrupted runtime. This time it was fatal. At 08:43:22 IST, SSM lost contact. Kubelet posted its last heartbeat. At 08:45:10, the node went NotReady.

Now watch how every auto-recovery mechanism missed it.

Failure #1 — EC2 Auto Scaling Health Check

The ASG protecting this nodegroup was configured with EC2-level health checks:

HealthCheckType: EC2
Min: 6 | Max: 6 | Desired: 6

EC2 health checks have exactly one question: is the VM powered on?

aws ec2 describe-instance-status \
  --instance-ids <id> --region <region> \
  --query 'InstanceStatuses[].[InstanceState.Name,SystemStatus.Status,InstanceStatus.Status]' \
  --output table

+---------+------+------+
|  running|  ok  |  ok  |
+---------+------+------+

The instance was running. System status: ok. Instance status: ok. The AWS hypervisor was satisfied. The deadlock inside containerd was completely invisible at the hypervisor layer — the kernel was alive, the VM was up, the NIC was responding to health probes.

EC2 health check never triggered. No replacement instance was launched.

The lesson: EC2 health checks only detect hardware failure or VM termination. They cannot detect OS-level hangs, kubelet death, containerd deadlocks, or any software failure that doesn't take down the underlying VM. For Kubernetes node health, EC2 checks are nearly useless.

Failure #2 — EKS Node Auto Repair

EKS Node Auto Repair is designed exactly for this scenario. When a node stays NotReady for a defined period, it automatically cordons, drains, and replaces it. No manual intervention needed.

aws eks describe-nodegroup \
  --cluster-name <name> \
  --nodegroup-name <nodegroup_name> \
  --query 'nodegroup.nodeRepairConfig'

null

null. Never configured. Never enabled.

This feature was available. We had just never set it up. The node sat NotReady for nearly 4 hours while the feature that would have replaced it in 10 minutes was turned off by default.

The lesson: EKS Node Auto Repair is not enabled by default. Check every nodegroup right now with the command above. If you get null, you have this gap.

Failure #3 — Cluster Autoscaler

The cluster autoscaler was running. It's supposed to handle exactly this kind of situation. But our configuration made it powerless:

Min: 6 | Max: 6 | Desired: 6

Min equals Max. The autoscaler has no headroom to operate. To replace a broken node, it needs to launch a new one first (going to 7, violating Max), drain the broken node, then terminate it (back to 6). With Max=6, it couldn't even start that sequence.

The autoscaler logs confirmed it saw the problem — and couldn't act:

I0817 12:17:57 pre_filtering_processor.go:67]
Skipping ip-*-*-*-*.ec2.internal — node group min size reached
(current: 6, min: 6)

It knew the node was bad. It had no authority to fix it.

There was a second problem: the cluster-autoscaler pod itself was running on the broken node, stuck in Terminating state alongside 25+ other pods.

kube-system  cluster-autoscaler-5********8  1/1  Terminating  0  6d5h

The thing responsible for replacing the broken node was stuck on the broken node.

The lesson: Min=Max is a zero-headroom configuration that paralyzes automated recovery. Set Max to at least Min+2 on every nodegroup.

Failure #4 — Memory Alert (Fired Too Late to Matter)

We had a memory alert configured — running on an 11-minute cron interval, watching for node memory usage above 95%.

ELK showed the node at 81.6% for hours before the incident:

IST timestamp    Node memory %
08:22:57         81.6%
08:33:57         81.6%
08:44:57         80.6%  ← node already down, stale value

The 4.2GB memory spike happened entirely between two check cycles. At 08:22 the node was fine. At 08:44 the check returned stale data because the node was already gone.

The alert fired at 08:44 IST. The node went unreachable at 08:43:22 IST. The alert was 82 seconds behind.

This is a structural problem: the metric source is the node itself. When the node dies, metrics stop. If the crash happens between alert evaluation cycles, you miss it entirely. The faster a node crashes, the more likely it dies between evaluations.

The lesson: Metric-based node alerts have a fundamental blind spot. If the node is the source of the metric, the metric disappears when the node dies.

What We're Fixing

Enable EKS Node Auto Repair (do this first)

aws eks update-nodegroup-config \
  --cluster-name <name> \
  --nodegroup-name <nodegroup_name> \
  --node-repair-config enabled=true \
  --region us-east-1

When a node stays NotReady for the configured threshold, EKS automatically cordons, drains, and replaces it. The control plane makes this decision — it doesn't depend on the broken node reporting its own failure.

Set Max > Min on Every ASG

aws autoscaling update-auto-scaling-group \
  --auto-scaling-group-name <name> \
  --min-size 6 \
  --max-size 8 \
  --region us-east-1

Max needs to be at least Min+2 to give the autoscaler room to launch a replacement before removing the broken instance.

Switch to ELB Health Checks

aws autoscaling update-auto-scaling-group \
  --auto-scaling-group-name <name> \
  --health-check-type ELB \
  --health-check-grace-period 300 \
  --region us-east-1

ELB health checks evaluate actual application-layer response, not just hypervisor state.

Replace Metric-Based Node Alert with Condition-Based Alert

# This fires even when the node is completely dead.
# kube-state-metrics reads from the API server, not from the node itself.
- alert: NodeNotReady
  expr: kube_node_status_condition{condition="Ready",status="true"} == 0
  for: 90s
  labels:
    severity: critical
  annotations:
    summary: "Node {{ $labels.node }} NotReady for >90s — EKS Auto Repair should engage"

kube-state-metrics watches the Kubernetes API server — not the node. It fires even when the node is completely unreachable.

Enforce Memory Overcommit Limits

yaml
apiVersion: v1
kind: LimitRange
metadata:
  name: memory-ratio-limit
  namespace: your-namespace
spec:
  limits:
  - type: Container
    maxLimitRequestRatio:
      memory: "4"   # limit cannot exceed 4x request

This forces the actual memory ceiling to stay within a predictable range of what the scheduler sees. A pod with a 1Gi request cannot have a 16Gi limit.

The Broader Problem

This wasn't an unusual incident. A node accumulated damage silently for 46 days. Nothing alerted. Nothing flagged the degraded containerd state. The node reported Ready to the control plane while internally the runtime was already compromised.

When the fatal event came — a memory spike from an unrelated service — four recovery mechanisms failed in different ways:

EC2 health checks were asking the wrong question (is the VM up?)
EKS Auto Repair was never turned on
The autoscaler had no room to maneuver (Min=Max)
The metric-based alert fired 82 seconds after the node was already dead

Any one of these gaps would have slowed recovery. All four together meant 4 hours of a NotReady node and a manual reboot at noon.

The fixes are each one or two commands. None of them are complicated. They just require knowing the gaps exist.

Check Your Cluster Right Now

Run this on every nodegroup:

aws eks describe-nodegroup \
  --cluster-name <your-cluster> \
  --nodegroup-name <your-nodegroup> \
  --query 'nodegroup.nodeRepairConfig'

If you get null — you have this gap. Fix it before the next 2 AM alert.

The names, cluster/node identifiers, timestamps, instance IDs, pod counts, resource values, and other incident-specific details in this article have been intentionally changed or fictionalized for privacy and security. The failure pattern and technical lessons remain representative of the original incident.

Follow me on Medium for more incident stories from production.