DEV Community: BuildOpsy

The three meters behind a $65 million observability bill

BuildOpsy — Thu, 29 Oct 2026 03:59:18 +0000

$65 million is a strange number to hear on an earnings call. It gets stranger when you work out what it buys at list price.

On 4 May 2023, Datadog reported first-quarter billings of $511 million, up 15% year over year, and flagged one exception: a large upfront bill for a single customer in Q1 2022 that did not recur at the same level or timing. A JPMorgan analyst, Mark Murphy, did the subtraction on the call and arrived at roughly $65 million. Datadog's CFO, David Obstler, confirmed the shape without the name: a crypto company, an early optimizer, now billed in smaller chunks.

A week later, Gergely Orosz reported that current and former Coinbase engineers had confirmed to him that the $65 million was Coinbase's Datadog bill for 2021, settled in the first quarter of 2022. Coinbase has never confirmed it publicly. Datadog never named the customer. Everything below rests on that sourcing.

The number is worth taking seriously rather than gawking at, because it is not really a story about one company overspending. It is a story about three meters that compound independently, and what happens when nobody owns any of them.

Three meters, three units, three growth curves

Observability vendors do not sell one meter. At Datadog, the ones that stack are these, at the annual-billing list rates published today:

Meter	List price	What grows it
Infrastructure, Enterprise	$23 per host per month	Fleet size
Log ingest	$0.10 per GB ingested	Verbosity × volume
Standard indexing, 15-day retention	$1.70 per million events	Event volume × retention
APM	$31 per host per month, with infrastructure attached	Services × hosts

On-demand rates are higher across the board: $27 per host, $2.55 per million indexed events, $36 per APM host.

The compounding is the point. Hosts grow with the fleet. Ingested gigabytes grow with log verbosity, and retention pressure multiplies whatever you ship. Indexed events grow with cardinality, and custom metrics beyond the per-host allotment — 100 on Pro, 200 on Enterprise — bill by usage, so every new tag value on a custom metric mints billable series. A team that adds hosts, turns up log verbosity, and tags everything has tripled its bill three independent ways before writing a single alert.

What $65 million means in each unit

At $23 per host per month, $65 million is about 235,000 host-years of Enterprise infrastructure monitoring. That is arithmetic on list price, not a claim about what Coinbase bought, but it is the right order of magnitude for a company running a large fleet at the peak of a growth phase.

The small-scale version is more useful. Twenty hosts at $23 comes to $460 a month. A thousand gigabytes of ingest at $0.10 adds $100. That is $560 a month, or $6,720 a year — the default output of a cost calculator I built for exactly this, and a modeled figure rather than anyone's real invoice.

Divide one into the other: $65 million is roughly 9,700 years of that model.

The gap between $6,720 and $65,000,000 is not a pricing scandal. It is what four orders of magnitude of fleet growth, log verbosity, and cardinality look like when no individual decision was ever expensive enough to argue about.

How 2021 produced that number

Coinbase went public in 2021 and had a year most companies never see. Infrastructure cost was not a constraint anyone enforced, because revenue covered everything. AWS, Snowflake, and Datadog all grew. A bill that size is what three meters running at startup speed produce when nobody is assigned to watch them.

Then the market turned. Revenue fell hard through 2022, and the bills that growth had hidden became the agenda. Datadog's CEO described the pattern on the same call: a customer whose business got cut to a third or a quarter gets its contract restructured, because the vendor would rather be part of the solution than part of the problem. The customer stayed, on chunked billing matched to a smaller business.

One thing to be careful about: an anonymous commenter claimed the $65 million covered a multi-year prepayment. Orosz could not verify that and neither can anyone else, so it stays out. What is documented is the single-year bill, the settlement quarter, and the renegotiation.

The build-vs-buy line, stated properly

Coinbase's response was to price the alternative: a dedicated observability team with a mandate to move onto Grafana, Prometheus, and ClickHouse, built for control and ownership first, with cost as the second motive once the market turned. They double-wrote the new stack for months before cutting anything over.

The arithmetic that mattered is a single inequality:

infrastructure cost + platform team cost < vendor bill

Orosz's version: ten senior and staff engineers in the Bay Area cost under $5 million a year, with infrastructure on top in the low double-digit millions. Against $65 million, building is not a close call. Against a well-managed $2 million bill, it is a way to hire a team whose job is maintaining dashboards.

His rule of thumb for where the question becomes real is $2–5 million a year in vendor spend — roughly where a team of four or five engineers plus a manager, at $150–400K fully loaded each, starts to pencil against the invoice.

The breakeven moves with the invoice. Which is why the invoice has to be measured before the decision is made, not after someone sees a headline and asks for a migration plan.

They didn't leave

The part the invoice-watchers usually miss: with the replacement stack validated and the plug ready to pull, Coinbase got terms it could accept and stayed. The customer is still on Datadog.

That is not a contradiction. Deep integrations — custom tracing, per-service instrumentation, dashboards wired into every team's workflow — make migration expensive in a way the license math never captures. That lock-in cut both ways: it gave Coinbase leverage for a steep discount, and it gave Datadog a retained logo.

The lesson is uncomfortable for both sides of the table. Price your exit before you threaten it, or the threat is theater.

Four things to do on Monday

Budget tag cardinality like headcount. Every new tag value on a custom metric mints series, and series beyond the per-host allotment bill by usage. Audit which teams emit the most series per host before approving more hosts.

Tier retention by access pattern. Logs you investigate regularly earn standard indexing with alerting. Everything else belongs in a cheaper tier or an archive with rehydration on demand. Retention is the quietest multiplier on the invoice.

Decide sampling as a cost input. Full-fidelity tracing on every request is a second invoice hiding inside the first. Pick the sampling rate when you design the tracing, not when you are debugging.

Alert on ingest slope, not the total. Gigabytes per day times price per gigabyte, tracked against the monthly commit, turns the bill into a metric the on-call actually owns. A meter nobody watches is a meter without a limit.

And the cheapest one of all: assign the bill to the teams that generate it. Chargeback turns "observability is expensive" into named owners with named levers. Coinbase assigned a whole team to the problem. Most companies only need to assign the invoice.

What this doesn't establish

The identification rests on Orosz's reporting, not on a Coinbase or Datadog statement. Neither company has confirmed the customer. The $65 million covers one year of spend, not a multi-year contract that anyone has verified.

Coinbase's current spend is undisclosed, so there is no way to check how much the renegotiation actually saved, or whether the in-house stack would have been cheaper over five years.

The list prices are Datadog's published rates as of October 2026 and will not match any negotiated contract, including the one in this story. And no vendor published utilisation figures, so the three-meter model here describes the structure of the bill, not its actual composition.

Three clocks ran in this story: the meters compounded through 2021, the invoice came due in Q1 2022, and the renegotiation reset the economics after that. Treating any one of them as "the Datadog incident" misses how metered billing actually fails — slowly, across independent units, and only discovered when growth stops covering it.

Sources

A cancelled query can destroy your database connection

BuildOpsy — Thu, 29 Oct 2026 03:37:59 +0000

Two short deadlines took down Inngest's database path twice in one afternoon, and neither was misconfigured. They expired exactly on time. That is what made the failure so effective.

On 10 October 2026, both of Inngest's connection poolers lost every connection to their primary database at 14:00 UTC. They lost them again at 15:30. Run scheduling and queue processing stalled for most accounts in both windows, 14:00–14:48 and 15:30–16:25, and the mitigation that followed caused intermittent delays until 04:17 the next morning.

The report is worth reading because the mechanism generalises to any stack that puts a connection pooler with a wait queue between the application and Postgres.

The sequence, in four steps

A frequently used lookup carried a short deadline.
The deadline expired while its query was still queued inside PgBouncer, waiting for a database connection. The pgx driver cancelled the query and closed its client connection.
PgBouncer had nothing running to cancel, because the query had never been handed a server connection. It later ran the abandoned query on a database connection.
It then closed that connection entirely instead of returning it to the pool.

Step 4 is the whole failure. The client's timeout cost the system a database connection, not a request, and the cost landed after the client had stopped watching. No single request looked expensive. Individually, every one of these is a wasted query. Collectively they are a pool that empties itself.

Two amplifiers arrived at the same moment. Both collapses began at the top and bottom of the hour, when cron-scheduled work roughly triples the query rate, and a newly rolled out set of lookups was failing in large numbers, adding cancellations to a pool that was already busy. Inngest reproduced the behaviour in a test environment.

Why restarting and resizing both failed

Restarting the poolers did not reliably recover them. Raising the pool size did not help either. The report is explicit on why: opening connections was the bottleneck, not the pool limit.

Each pooler opens new database connections one at a time. So the refill path is serial while the destruction path is parallel. As a steady-state sketch rather than a model of this incident, if usable connections are destroyed at d per second and a pooler needs t seconds on average to establish a replacement, the refill ceiling is 1/t and the pool drains whenever:

d > 1/t

The sketch assumes constant destruction and handshake times, and it ignores the point at which the pooler stops trying. Inngest published neither number, so treat this as the shape of the constraint rather than a measurement of it. To see the shape: at 100 ms per connection a single pooler refills at ten per second; at 50 ms, twenty.

The second collapse ended by changing paths, not by growing the pool. The team moved all services onto poolers run alongside the database by the database provider, and processing recovered between 16:10 and 16:25.

The fix that caused a second incident

The new route reached the database through NAT gateways, and some connections through those gateways were silently dropped. Queries on a dropped connection hung for up to 16 minutes before failing, which stalled processing intermittently and most visibly hit cron functions. Customers reported delays at 03:01 on 11 October; the affected service moved back to the original poolers at 04:17, and gateway routing was improved on 12 October.

Sixteen minutes of silent hang is worse than a fast failure for a scheduler. A connection that errors invites a retry. One that simply stops answering consumes a worker slot and a deadline, and you learn about it from a customer. A failover drill that measures successful requests will not catch this. One that measures how long a hung connection stays hung will.

The contract that broke underneath it

The part of the report with the longest reach is not about pools. Internal failures counted against functions' configured retries, and a step that failed permanently on an internal error left its run in Running instead of marking it failed. onFailure handlers did not fire. Functions waiting on that run through step.invoke were not notified.

That is a broken guarantee, not an availability number. A durable execution engine sells the promise that every run reaches a terminal state and that failure handlers run. Under this bug the engine could hold a run in Running indefinitely, after spending the retry budget it was given — the customer's allowance consumed by the platform's own faults, and the transition that would have reported it never sent.

The recovery instructions confirm the state of those runs: they will not continue on their own, and customers can cancel and replay them, with the caveat that a replay starts from the beginning, so completed steps run again. Anyone with non-idempotent side effects in a completed step had a decision the platform could not make for them.

Four things to check in your own stack

Alert on net connection drain, not pool utilisation. Graph opens and closes per pooler per second, and alert when the close rate exceeds the open rate for a sustained interval. A pool at 30% with a negative slope is in more danger than one at 70% that is stable, and utilisation alone shows neither.

Bound the wait, not just the query. PgBouncer's query_wait_timeout disconnects a client whose query has not been assigned a server within the limit.

; pgbouncer.ini — the ordering matters more than the value
query_wait_timeout = 5    ; must sit below your client's own deadline

The point of setting it below the client deadline is to remove the queued entry before the client gives up. That is the reasoning, not proof that it preserves backend connections. Inngest describes the pooler running the abandoned query and then closing that connection; an independent load test from July 2025 puts the waste a step earlier, with the pooler finishing a login for a client that has already gone and discarding the fresh backend connection. The same engineer reported that lowering the timeout helped without being enough on its own. Test it against your driver's real cancellation path before adopting it.

Know what your driver does on cancel. In pgx v5, the thread reports replacing the default cancellation behaviour with a context watcher handler that sets a connection deadline instead of closing immediately, and describes the effect in their load test as large. If your code path cancels queued queries, this is the boundary to inspect.

Fence the lease before you force a terminal state. If you mark a run failed because its step lease expired without a heartbeat, late writes from a worker that is still running can land afterwards, and a replay produces a second copy of the side effects. Mark it failed only when the engine can reject writes from the old lease, or when the step is confirmed not to be in flight.

What the report does not establish

Inngest disclosed no connection counts, handshake timings, pool sizes or customer totals, so the arithmetic here is the report's timestamps and nothing more. The 44-minute, 48-minute and 55-minute windows are my subtraction on its UTC boundaries, not figures it published.

The report also links pgx issue 1780 as the known interaction behind the collapse. I read that issue: it was opened on 26 October 2023 and was still open at its latest comment in February 2026. The upstream record offers a compatible description of the boundary and no confirmed root cause, so Inngest's account is a well-documented instance of an unresolved interaction rather than evidence that it is fixed.

The structural change is the one to watch. Inngest is consolidating onto a single data centre acting as one cell, which removes the cross-data-centre failure class behind the 9 October incident. That trades one failure shape for another: fewer paths that can break, and more consequence when one does.

Sources

Thanks for reading :)

OpenRouter's February 2026 Outages: How a Cold Cache Overwhelmed the Database

BuildOpsy — Sun, 25 Oct 2026 11:38:07 +0000

The database was struggling. The API said the user didn't exist.

That is the part of OpenRouter's February outages worth looking at closely.

The reported failure involved cache connectivity and the database lookups behind API-key authentication. When the cache stopped serving those lookups, the database had to absorb work it normally wouldn't see. Requests started failing with 401 User not found.

There is an important distinction here: an authentication failure and an authentication infrastructure failure are not the same thing. If a database timeout becomes a 401, a client may treat a temporary service failure as an invalid credential. Retries, alerts and incident diagnosis can all head in the wrong direction.

The capacity problem is just as interesting. A database can appear comfortably provisioned because a cache absorbs most of the read traffic. That doesn't mean the database can survive the workload when the cache disappears. Recovery can create its own traffic spike, precisely when the dependency underneath is least able to handle it.

This is why cache-failure testing shouldn't stop at checking whether the application stays online. It should also examine database concurrency, fallback request rates, cache repopulation, retry behavior and the errors returned to clients. A fallback path that works for a few misses may behave very differently when every request takes it at once.

There is also a subtle observability problem here. The first outage coincided with a denial-of-service attack, which initially complicated the investigation. When the caching failure recurred in February without the same accompanying attack, the underlying cause became clearer. Multiple problems can overlap; temporal correlation alone doesn't establish root cause.

The remediation needs to address both load and semantics. Circuit breakers and controlled cache repopulation can limit the burst reaching the database. Returning 503 Service Unavailable rather than 401 Unauthorized when an infrastructure failure prevents an authentication lookup helps clients distinguish an unavailable service from invalid credentials.

A useful test for any cache-backed authentication system: invalidate the cache under realistic concurrency and measure the database's behavior during recovery. How quickly does load rise? Are concurrent misses coalesced? Do retries amplify the spike? Can the service shed work before the database saturates?

Those answers reveal more about the system's failure tolerance than its cache-hit ratio during normal operation.

The full postmortem reconstructs both outages, separates published facts from illustrative capacity calculations, and identifies what the available evidence does—and doesn't—establish about the remediation.

Read teh full teardown here: https://buildopsy.com/system-design/openrouter-cache-thundering-herd-postmortem.html

🚀Rocket thrusters on a car

BuildOpsy — Sun, 18 Oct 2026 10:05:09 +0000

Tesla promised rocket thrusters on a car. Naturally, the next step is to ask the engineers what the physics has to say about that.

Marketing has a wonderful relationship with the future. Engineering has a slightly less romantic relationship with mass, thrust, pressure, and the laws of physics.

So we audited the Tesla Roadster’s thruster claims as an engineering problem, not a headline-writing competition.

What’s theoretically possible? What has actually been demonstrated? What does the available evidence support, and where does the gap between a claim and a verified capability begin?

Because there’s a rather important difference between a technology that could work in principle and one that has been demonstrated to work as advertised.

We went through the claims, examined the evidence, and separated the physics from the promises.

Turns out, “the laws of physics don’t forbid it” and “the product can actually do it” are two very different engineering milestones.

A useful reminder that even when your company builds rockets, the burden of proof doesn’t magically disappear when you put the word Tesla on the slide.

Full engineering audit: https://buildopsy.com/system-design/tesla-roadster-thruster-claims-audit.html

When Go GC Burns 60% CPU: The Container Memory Limit Problem

BuildOpsy — Thu, 15 Oct 2026 18:27:31 +0000

Banner: Go garbage collection, memory pacing, and the container limits that can make a difference.

A Go service can burn CPU collecting garbage without having a memory leak.

That is the counterintuitive part of Zomato's Go garbage collection story.

The reported symptoms were striking: garbage collection consuming 25–60% of CPU in some services, alongside containers getting OOM-killed without an obvious memory leak.

These look like separate problems. But they can share a common cause: the garbage collector's pacing decisions don't automatically account for the container's actual memory budget.

The problem with doubling the live heap

Go's garbage collector uses GOGC to control when the next collection should happen.

With the default GOGC=100, the GC target is roughly based on the live heap growing by 100% relative to the previous cycle, with additional accounting for GC roots and other implementation details.

In simplified terms:

Next GC target ≈ live heap × (1 + GOGC/100)

At GOGC=100:

Live heap:       500 MiB
Approx. target: 1,000 MiB

That target is reasonable as a starting point when the process has enough memory available.

But containers introduce another constraint: the process has a memory budget imposed from outside the Go runtime.

The GC's heap-growth target and the container's memory ceiling are two different things.

When they aren't coordinated, two failure modes become possible.

Failure mode 1: OOM kills without a leak

Suppose a container has a memory limit of 1 GiB.

The Go process has a live heap of 600 MiB. A simplified GOGC=100 calculation puts the next target around 1,200 MiB.

But the container cannot safely accommodate that much memory. The process also needs room for goroutine stacks, runtime metadata, native allocations, and other memory outside the live heap.

The container can hit its limit before the collector's normal heap target becomes appropriate.

The result can be an OOM kill even when reachable memory isn't growing uncontrollably.

That doesn't mean every OOM without an obvious leak is a GC pacing issue. Transient allocation spikes, native memory, and other causes still need investigation.

The important point is that a heap-growth target alone isn't a container memory budget.

Failure mode 2: Excessive GC CPU

Now consider the opposite situation.

A service has a relatively small live heap and plenty of memory available in its container.

The GC keeps using a target derived from that live heap. Depending on the allocation rate and workload, it may collect frequently even though the container has significant unused headroom.

Every collection consumes CPU.

At fleet scale, unnecessary GC work can become a meaningful compute cost.

This is why the following combination deserves attention:

High GC CPU.
A relatively flat live-heap graph.
No convincing evidence of sustained heap growth.

It is a diagnostic clue, not proof. A service can allocate and discard large volumes of temporary objects while maintaining a flat live heap. Allocation rate, GC frequency, and allocation profiles matter too.

But before treating every high-GC-CPU incident as a memory leak, inspect the collector's pacing and the container's actual memory constraints.

The fix: Give Go a realistic memory limit

Go provides GOMEMLIMIT to let applications communicate a soft memory limit to the runtime.

Unlike GOGC, which controls GC aggressiveness in relation to heap growth, GOMEMLIMIT helps the runtime account for its overall memory footprint when deciding how aggressively to collect.

Conceptually, the runtime must balance two competing costs:

Collect too late, and the process risks exhausting its container memory budget.
Collect too aggressively, and the application wastes CPU reclaiming memory it could have used safely.

A memory limit gives the runtime another constraint to consider.

In the Zomato case, the reported approach used container metadata to derive a per-container limit, refreshed it periodically, and combined memory tuning with CPU scheduling adjustments.

A limit around 90% of the container ceiling can be a starting point for investigation, not a universal safe value. The correct margin depends on the application's non-heap memory, traffic spikes, allocation behavior, and operational requirements.

The goal is to give the runtime a realistic budget while preserving enough headroom for the rest of the process.

Why `GOMEMLIMIT` isn't a hard wall

This distinction matters in production.

GOMEMLIMIT is a soft limit, not a strict process-memory cap. The runtime can exceed it when necessary, and it does not guarantee that a container will avoid OOM kills.

The runtime also has to balance memory pressure against GC CPU overhead.

If the limit is too high, it may offer little protection against container memory pressure. If it is too low, the runtime can spend excessive CPU collecting garbage in an attempt to stay within the target.

That is why blindly setting GOMEMLIMIT equal to the container limit is not a complete solution.

A production rollout should account for:

The container's actual memory limit.
Heap and non-heap memory usage.
Peak allocation rates and traffic bursts.
GC CPU and pause behavior.
OOM events and memory headroom after deployment.

Start with representative workloads, observe the results, and tune based on measurements rather than assuming a single multiplier works everywhere.

Don't overlook `

CircleCI's Runners Were Ready. PostgreSQL Couldn't Dispatch Jobs.

BuildOpsy — Thu, 15 Oct 2026 16:20:37 +0000

Architecture diagram by BuildOpsy: How PostgreSQL WAL saturation blocked CircleCI's orchestration database, leaving healthy runners idle and thousands of job starts affected.

CircleCI's Runners Were Fine. PostgreSQL Wasn't.

On October 6, 2026, CircleCI hit a failure that had nothing to do with runner capacity.

The culprit was routine PostgreSQL maintenance.

An anti-wraparound autovacuum ran against large tables, generating write-ahead log (WAL) faster than the database could archive it. The WAL volume filled, and the orchestration database went offline for 50 minutes.

The numbers tell the story:

~3,500 job starts failed
50 minutes of database unavailability
Hours of recovery work after the database returned

Here's the underlying systems problem.

Let g be the WAL generation rate and a the archival rate.

When g > a, unarchived WAL accumulates. Free disk space only determines how long the system can tolerate that imbalance.

A rough estimate of the remaining time is:

Time to saturation ≈ Free WAL space / (g − a)

This assumes relatively stable rates, but the principle is important: a disk-full alert often arrives too late. By the time it fires, the system may already be out of runway.

The architectural impact was bigger than a database outage.

CircleCI's orchestration database sat on the critical path for workflow state, job dispatch, UI updates, and notifications. The runners could be healthy and ready to execute, but without the control plane coordinating work, they couldn't help.

Then came the second problem: recovery.

Restoring database availability didn't instantly clear the backlog. Queued jobs still needed to be processed and dispatched, putting additional pressure on the system as it recovered.

This is the part of incident response that's easy to underestimate: restoring service and restoring normal operation are two different problems.

The safeguards I'd take from this incident:

Monitor WAL generation versus archival throughput. Alert on the rate gap, archive lag, and estimated time to saturation, not just disk utilization.
Test autovacuum under realistic workloads. Large tables and production-level write activity can expose bottlenecks that smaller test environments miss.
Protect the recovery path. Make sure WAL archival and database recovery have enough capacity and clear operational guardrails.
Control backlog redrain. Use bounded concurrency, retry limits, and staggered dispatch to avoid overwhelming downstream services.
Design for control-plane failures. A healthy execution fleet is useless if the system responsible for scheduling work is unavailable.

The deeper lesson is that a healthy execution fleet doesn't guarantee a healthy CI/CD system. The control plane that decides what runs can be the real bottleneck.

I broke down the failure mechanism, incident timeline, recovery bottleneck, and architectural trade-offs in the full postmortem:

👉 CircleCI Outage Postmortem: Full WAL Saturation Blocked 3,500 Job Starts

Question for PostgreSQL engineers: Do you monitor WAL generation rate against archival throughput and estimate time-to-full, or are your alerts still mostly based on disk utilization?

I'd be interested to hear what works in production. Thanks

CircleCI’s runners were ready. The database couldn’t tell them what to run.

BuildOpsy — Mon, 12 Oct 2026 18:08:05 +0000

Around 3,500 job starts failed because PostgreSQL’s WAL generation outpaced archival during routine anti-wraparound autovacuum.

The dedicated log volume filled, and CircleCI’s orchestration database went offline for 50 minutes. A maintenance operation had turned into a developer-feedback outage.

How Routine Vacuum Work Filled the WAL Disk

PostgreSQL autovacuum is routine maintenance, not an optional background nicety. CircleCI's report says an anti-wraparound autovacuum ran on several of its largest tables. On a very large table the process reads every page and produces a large volume of transaction logs. Around 12:40 UTC, this maintenance output plus normal application writes exceeded the rate at which the database could archive and reclaim the dedicated log volume.

The failure was a rate mismatch, not simply a database that had “too little disk.” If WAL is produced at rate g and archived at rate a, free space falls whenever g > a. Disk headroom buys time, but does not correct the mismatch. CircleCI did not publish those rates or the volume size, so the inequality explains the mechanism without pretending to reconstruct its capacity numbers.

Alerts began showing brief orchestration errors at 10:02. They cleared within about 15 minutes and looked similar to alerts responders had seen before. As the WAL volume filled, slowdowns became intermittent around 13:30, then continuous by 15:30. The incident was declared at 14:54. CircleCI's retrospective says the noisy, short-lived alerts and the absence of direct alerts on WAL volume and archive lag delayed recognition of the combined failure.

The more interesting part? Bringing the database back wasn’t enough. The storage change cut processing speed to roughly half its normal rate, and the backlog kept hurting customers for hours. It’s a good reminder that database recovery and service recovery are two very different things.

We broke down the failure mechanism, recovery timeline, architectural bottleneck, and the safeguards worth adding.

Full postmortem: https://buildopsy.com/system-design/circleci-wal-autovacuum-postmortem.html

How fast Starlink is when it is up. as India debates entry, latency is the number that decides whether satellite broadband can carry gaming, calls, and real-time trading at all https://buildopsy.com/system-design/starlink-latency-whitepaper-teardown.html

BuildOpsy — Mon, 12 Oct 2026 04:01:26 +0000

Starlink Latency, Priced: How SpaceX Cut Peak Median From 48.5ms Toward 20ms | Buildopsy

Physics allows under 10ms. SpaceX measured 48.5ms. The 15.5ms gap was buffers, queues, WiFi, and ground layout — and they priced every millisecond.

buildopsy.com

A small search win: Google's AI Overview cited my Figma multiplayer teardown

BuildOpsy — Tue, 06 Oct 2026 18:49:32 +0000

I write about production system design on Buildopsy. Recently, I published a teardown of Figma’s multiplayer architecture: how edits move between browsers and servers, how conflicts get resolved, and where the system’s scaling costs come from.

I later searched for “how Figma’s multiplayer technology works.” Figma’s own engineering post appeared in the results. In the AI Overview below it, I noticed two Buildopsy citations beside explanations of optimistic rendering and the server’s role in ordering edits.

What appeared in the result

I saved a screenshot because I didn’t expect to see my article cited there. It’s one result for one query—not proof of a lasting ranking or a traffic increase.

The interesting part is how edits get reconciled

A browser applies an edit immediately, then sends the operation over a WebSocket to the process that owns that document. The owner orders the change and broadcasts the accepted version to the other clients.

The conflict rule is more specific than “the server wins.” In the architecture described in my teardown, the server orders operations, same-property conflicts use last-writer-wins, and tree changes get an additional validation step to prevent cycles. That’s a product-specific trade-off: a design canvas can tolerate some conflicts that would be painful in a collaborative text editor.

The Rust rewrite addressed another boundary. A slow document could stall unrelated documents sharing a TypeScript worker. Figma moved document operations into a Rust child process while Node.js kept handling the network layer. Figma reported that serialization became more than 10× faster in its engineering write-up.

The numbers make the next question more interesting. In one scenario, 1.5 million concurrent users sending two coalesced cursor frames per second produce 3 million ephemeral frames per second. That’s modeled math, not a Figma disclosure. The design question is: which events need durable ordering, and which can be dropped when they’re stale?

The full teardown works through that boundary, a 65k-RPS scenario, and a modeled monthly cost range. The cost figures are my assumptions, not Figma’s reported spend: Read the Figma multiplayer teardown.

I don’t know whether the citation will persist or bring readers. It was still encouraging to see a technical explanation appear alongside the primary source it discusses.

DEV Community: BuildOpsy

The three meters behind a $65 million observability bill

Three meters, three units, three growth curves

What $65 million means in each unit

How 2021 produced that number

The build-vs-buy line, stated properly

They didn't leave

Four things to do on Monday

What this doesn't establish

Sources

A cancelled query can destroy your database connection

The sequence, in four steps

Why restarting and resizing both failed

The fix that caused a second incident

The contract that broke underneath it

Four things to check in your own stack

What the report does not establish

Sources

OpenRouter's February 2026 Outages: How a Cold Cache Overwhelmed the Database

🚀Rocket thrusters on a car

When Go GC Burns 60% CPU: The Container Memory Limit Problem

The problem with doubling the live heap

Failure mode 1: OOM kills without a leak

Failure mode 2: Excessive GC CPU

The fix: Give Go a realistic memory limit

Why GOMEMLIMIT isn't a hard wall

Don't overlook `

CircleCI's Runners Were Ready. PostgreSQL Couldn't Dispatch Jobs.

CircleCI's Runners Were Fine. PostgreSQL Wasn't.

CircleCI’s runners were ready. The database couldn’t tell them what to run.

How fast Starlink is when it is up. as India debates entry, latency is the number that decides whether satellite broadband can carry gaming, calls, and real-time trading at all https://buildopsy.com/system-design/starlink-latency-whitepaper-teardown.html

Starlink Latency, Priced: How SpaceX Cut Peak Median From 48.5ms Toward 20ms | Buildopsy

A small search win: Google's AI Overview cited my Figma multiplayer teardown

What appeared in the result

The interesting part is how edits get reconciled

Why `GOMEMLIMIT` isn't a hard wall