ChatGPT serves 800 million users from one Postgres primary. Here's how 🐘

Hey everyone 👋

800 million people use ChatGPT, and every write still lands on one PostgreSQL primary. No sharding.

When I read OpenAI's post Scaling PostgreSQL to power 800 million ChatGPT users by Bohan Zhang, I couldn't stop thinking about it. So I turned it into an interactive diagram: robots carry queries between the machines, and you can break things to see what happens.

ChatGPT Postgres - Blocks | Muhammad Ahmad

How OpenAI scaled PostgreSQL for 800 million ChatGPT users, animated: the first design and today’s, in normal traffic, a cache-miss spike and a primary failure.

mahmad.me

Here's the short version of how it works.

The problem 🔥

ChatGPT's database load grew more than 10x in a year. One primary handled every write and many reads, and incidents kept following the same pattern: something upstream spiked the load (a cache failure, a burst of expensive joins, a write storm). Queries slowed down, requests timed out, and the retries added even more load.

OpenAI fixed it in five moves.

1. Move reads off the primary

Reads go to nearly 50 read replicas around the world, and the primary keeps the writes. Requests are also split into high- and low-priority replicas, so a heavy low-priority job can't slow down the important ones.

2. Send write-heavy work somewhere else

Write-heavy work that can be sharded moved to Azure Cosmos DB. And new tables aren't allowed on Postgres anymore: new workloads start on the sharded systems.

3. Pool the connections

Azure Postgres allows 5,000 connections, and opening one was slow. PgBouncer keeps connections open and reuses them, so connecting dropped from 50 ms to 5 ms on average.

4. Stop cache stampedes

When a popular cache key expires, thousands of requests miss at once and all hit the database. With cache locking, one request refills the key and the others wait for it.

5. Protect the primary 🛡️

• A hot standby is ready to take over if the primary fails, and reads keep working from the replicas meanwhile.
• Rate limits at several layers, all the way down to blocking a single expensive query pattern.
• No giant joins. One query joined 12 tables, and its spikes caused high-severity incidents.
• Only lightweight schema changes, with a strict timeout.

The result 📈

• Millions of queries per second
• Low double-digit millisecond p99 latency
• Five nines of availability
• One SEV-0 in 12 months, during the viral launch of ChatGPT image generation, when writes jumped more than 10x and over 100 million users signed up in a week

The lesson I took away: you don't always need to shard. A single primary goes very far if you keep everything you can off it.

Play with it 🎮

In the diagram you can:

• switch between the initial design and today's
• trigger a cache-miss spike or take the primary down
• click any machine to take it apart and see what each part does

👉 mahmad.me/blocks/interactive/chatgpt-postgres

All the facts come from OpenAI's post. The live numbers in the simulation are illustrative.