On Monday evening, September 28, 2026, iPhone apps started crashing the moment people opened them. Not one app. Not a flaky update. App after app, open, crash, gone. If you were an iOS developer that night, your phone was a crime scene and your users were the witnesses.
The first suspect was obvious. Apple had shipped iOS 27.0.1 that same day. A new OS release plus apps dying on launch is a story every developer has seen before, and the blame wrote itself.
It was the wrong suspect. The apps were not crashing because of the OS. They were crashing because of a two-line change made on a server a thousand miles away, a change nobody on the app teams had ever heard of, in a system they never think about.
This is a detective story about that change, and about the three small checks that would have stopped it.
Clue one: it was not every app
The first thing that did not fit the Apple theory was the pattern. Plenty of apps opened fine. The ones crashing all had something in common: they used Google Analytics for Firebase, the analytics SDK that sits inside a huge share of iOS apps. One developer counted 56 crashes in the first 18 minutes, all in apps carrying that SDK.
Within minutes, developers were piling into a GitHub issue on Firebase's own repository. The crash reports were eerily identical. Every one of them ended the same way, on the Analytics experiment worker queue, with the same error: NSInvalidArgumentException: key cannot be nil.
That is a strange error to see in production. It means the code tried to put something into a dictionary under a key that did not exist. Not a bad value. A missing name. And the reports all traced back to the same moment: right after the SDK received a reply from Google's server at app-analytics-services.com/sdk-exp.
So the OS was innocent. The server had sent something the SDK could not swallow.
Clue two: the two-line change
Google published its postmortem four days later, on October 2. The timeline in it is short enough to read in one breath, and that is what makes it sting.
At 17:38 Pacific time, an engineer cleaned up a stale, legacy configuration flag. A two-line change. At 17:41, three minutes later, the malformed configuration payload began rolling out globally. At 17:59, crash alerts spiked and the first external bug reports landed on GitHub. At 19:16, responders traced the problem to the cleanup change and started a rollback. At 19:52, the rollback was fully deployed. The bad configuration was served for 2 hours and 11 minutes.
To understand what went wrong, you need to know what a remote configuration flag is, and why Google uses them.
The kill switch that ships to everyone
When you ship an SDK inside millions of apps, you cannot push a fix the way you push a server deploy. If a new feature in the SDK has a bug, you cannot reach into every phone and patch it. So you build a kill switch: a remote flag the SDK checks when it starts up. If the flag says the feature is off, the SDK turns the feature off, no app update needed.
That is a good design. But it has a consequence that matters here. Kill switches have to work even when everything else is broken, so they ship globally, to every app, all at once. There is no staged rollout for the thing that exists to stop a bad rollout.
Over time, an SDK like this accumulates flags. Features come and go, and the flags for dead features pile up. Teams mark the dead ones for cleanup, and now and then someone does the tidying.
On September 28, someone did the tidying. They removed a flag that was no longer needed. Two lines. But they did not remove a pointer to that flag that still lived in the SDK. When the SDK fetched the new configuration, it went looking for a flag that was no longer there. The flag's name came back as nothing, nil, and the SDK tried to use that nothing as a dictionary key. In Objective-C, that is a fatal error. The app crashed on launch.
The crash was a one-time event per app, and after it the SDK fell back to its previous configuration, so phones did not get stuck in a permanent crash loop. But a one-time crash on launch, for every user, across a huge population of apps, at the same time, is still an outage. And because of caching, some apps kept crashing for up to four hours after the fix went out.
Clue three: the dashboards were green
Here is the part that should worry you most, because it is the part that has nothing to do with Firebase.
While apps were crashing on millions of phones, the Firebase status dashboard stayed green. So did the Google Ads dashboard. The postmortem says why, plainly: those dashboards rely mostly on server-side health metrics, and this was a client-side crash. The servers were fine. They were serving the bad payload perfectly well. Nothing on the server side looked broken, because from the server's point of view, nothing was.
There was another gap. Google Analytics for Firebase does not have its own entry on the Firebase status dashboard. It links out to the Ads dashboard instead, and updating both dashboards needed manual work that took several hours. So while developers were watching their apps die in real time, the official status pages said everything was fine, and the Firebase team ended up keeping people informed on GitHub instead.
Your monitoring has the same blind spot unless you have checked. A dashboard that only watches your servers cannot see your users. If your failure mode lives on the client, your green dashboard is not evidence that things are fine. It is evidence that you are looking in the wrong place.
The tests passed
The postmortem has one more sentence worth sitting with: the existing tests passed on the change that caused the outage.
Of course they did. The tests checked the cleanup itself, and the cleanup was correct. The flag was stale, and removing it was the right call. What no test checked was the thing the cleanup left behind: a reference to a flag that no longer existed. The failure was not in the line that was deleted. It was in the line that was not deleted with it.
That is a pattern worth naming, because it shows up everywhere. A rename that misses one call site. A deleted column that one old query still selects. A removed environment variable that one script still reads. The change is correct and the tests pass, and the system breaks on the reference nobody remembered.
The three checks that would have stopped it
Google committed to four fixes: stricter SDK validation, better dashboard coverage, automated pipeline checks for orphaned identifiers, and slower, phased configuration rollouts. The first two of those are things you can build for yourself this week, in any language, for any configuration your service consumes.
Check one: never trust a payload from the server. The SDK's real bug was not the missing flag. It was that it trusted the payload. A name field that can be nil must be treated as a field that will be nil, one day, in production, at the worst moment. The fix Google promised is exactly this: validate every incoming payload strictly, ignore a corrupt flag, and fall back to cached defaults. The bad flag gets skipped. The app keeps running.
Here is that guard in Go, for a service that loads feature flags from a remote configuration endpoint:
package flags
import (
"encoding/json"
"log"
)
type Flag struct {
Name string `json:"name"`
Enabled bool `json:"enabled"`
}
// Load validates every flag in a remote payload. A corrupt entry is
// skipped and the last known good value is kept, so one bad flag can
// never take down the whole service.
func Load(payload []byte, cached map[string]bool) map[string]bool {
var incoming []Flag
if err := json.Unmarshal(payload, &incoming); err != nil {
// The whole payload is unreadable. Keep the cached flags
// and serve from those until the next good payload arrives.
return cached
}
next := make(map[string]bool, len(cached))
for k, v := range cached {
next[k] = v
}
for _, f := range incoming {
if f.Name == "" {
// A flag with no name is corrupt, not a flag.
// Log it, skip it, keep serving the cached value.
log.Printf("flags: skipping entry with empty name")
continue
}
next[f.Name] = f.Enabled
}
return next
}
The rule is simple: a missing name is a corrupt entry, and a corrupt entry is skipped, never trusted. The cached defaults are the safety net. If the payload is garbage, you serve the last good configuration and log loudly, instead of crashing or serving nothing.
The same guard in Python is ten lines:
def load_flags(payload, cached):
"""Merge a remote flag payload over cached flags.
Corrupt entries are skipped, never trusted. If the payload
itself is unreadable, the cached flags are served unchanged.
"""
try:
incoming = json.loads(payload)
except (json.JSONDecodeError, TypeError):
return cached
next_flags = dict(cached)
for flag in incoming:
name = flag.get("name")
if not name:
logging.warning("flags: skipping entry with missing name")
continue
next_flags[name] = bool(flag.get("enabled", False))
return next_flags
And a test that would have caught the Firebase bug is even shorter. Feed your loader a payload with a missing name and assert that it does not crash and does not lose the cached flags:
import "testing"
func TestLoadSkipsNamelessFlag(t *testing.T) {
cached := map[string]bool{"new_checkout": true}
payload := []byte(`[{"name": "new_checkout", "enabled": false}, {"enabled": true}]`)
got := Load(payload, cached)
if got["new_checkout"] != false {
t.Errorf("expected flag to update, got %v", got["new_checkout"])
}
}
That test is the whole lesson in one function. The second entry has no name at all. A loader without the guard would crash or store a broken entry. A loader with the guard updates the good flag and skips the bad one.
Check two: block cleanups that leave orphans behind. Deleting a flag, a column, a variable, or an endpoint is half a change. The other half is every reference to it. Google's fix is an automated pipeline check: if a cleanup leaves an orphaned identifier, the release is blocked before it can merge. You can build the cheap version today with a search in your deploy pipeline. Before a config change ships, grep the codebase for the identifier being removed. If anything still references it, the deploy stops. It is not elegant. It would have stopped this outage.
Check three: roll out configuration the way you roll out code. The kill switch shipped globally in three minutes because it had to be able to. But most configuration is not a kill switch, and it does not need to reach everyone at once. Phase it: a small slice of traffic first, watch the client-side error rate, then widen. And watch the client side, not just the server. If your alerting only fires when your servers are unhealthy, add one alert that fires when your clients start crashing. That is the alert that would have caught this one in minutes instead of leaving developers to find it on GitHub.
The question underneath it
Every system you run has a version of that stale flag. A configuration value that was set years ago and never cleaned up. A reference to a service that no longer exists, sitting quietly in a file nobody opens. A kill switch whose own mechanism has never been tested, because the day you need it is the day you find out whether it works.
The Firebase outage is a story about a two-line change. But the two lines were not the problem. The problem was a system that trusted its own configuration, tested the change but not the references, and watched its servers while its users crashed.
So here is my question for you. What is the stale flag in your system, the cleanup you have been putting off, the reference you assume is gone because the thing it points to is gone? And what happens the first time your code goes looking for it?
All facts in this article come from Google's own postmortem, "Google Analytics for Firebase iOS SDK outage on September 28, 2026," published October 2, 2026: https://firebase.blog/posts/2026/10/firebase-analytics-outage/
Top comments (0)