<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: BuildOpsy</title>
    <description>The latest articles on DEV Community by BuildOpsy (@buildopsy).</description>
    <link>https://dev.to/buildopsy</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4167148%2F450cc842-1067-483a-b2ca-2425ddb63fe1.png</url>
      <title>DEV Community: BuildOpsy</title>
      <link>https://dev.to/buildopsy</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/buildopsy"/>
    <language>en</language>
    <item>
      <title>What the eMudhra CA Incident Teaches Us About Cryptographic Key Generation</title>
      <dc:creator>BuildOpsy</dc:creator>
      <pubDate>Thu, 12 Nov 2026 03:10:09 +0000</pubDate>
      <link>https://dev.to/buildopsy/what-the-emudhra-ca-incident-teaches-us-about-cryptographic-key-generation-2ngk</link>
      <guid>https://dev.to/buildopsy/what-the-emudhra-ca-incident-teaches-us-about-cryptographic-key-generation-2ngk</guid>
      <description>&lt;p&gt;Certificate authorities are one of those pieces of infrastructure most developers rarely think about until something goes wrong.&lt;/p&gt;

&lt;p&gt;We trust browsers and operating systems to maintain a collection of trusted root certificates. Those roots establish trust for intermediate certificate authorities, which in turn issue certificates used by websites and services.&lt;/p&gt;

&lt;p&gt;It is a hierarchy that works quietly in the background.&lt;/p&gt;

&lt;p&gt;The October 2026 eMudhra incident is a useful reminder of what happens when something goes wrong inside that hierarchy — and why fixing the original vulnerability isn't always the end of the problem.&lt;/p&gt;

&lt;h2&gt;
  
  
  What happened?
&lt;/h2&gt;

&lt;p&gt;On October 17, 2026, a security researcher reported that eight eMudhra intermediate CA certificates had been generated using insufficient cryptographic randomness.&lt;/p&gt;

&lt;p&gt;The important detail is that these weren't conventional stolen private keys.&lt;/p&gt;

&lt;p&gt;According to CERT-In, the affected P-384 private key values were smaller than 256. This meant their corresponding public keys could be used to recover the private values almost instantly.&lt;/p&gt;

&lt;p&gt;In other words, the weakness was in key generation, not in the underlying elliptic-curve cryptographic algorithm.&lt;/p&gt;

&lt;p&gt;eMudhra subsequently explained that a key wrap/unwrap operation during an April 2025 key-generation ceremony had produced a second, weak set of keys alongside the intended strong keys. The weak set was the one that eventually reached browser root stores.&lt;/p&gt;

&lt;p&gt;The exact underlying process failure remains subject to further investigation.&lt;/p&gt;

&lt;h2&gt;
  
  
  The response was relatively fast
&lt;/h2&gt;

&lt;p&gt;The timeline is worth looking at because certificate revocation is not a single operation.&lt;/p&gt;

&lt;p&gt;Chrome received the report at 18:05 UTC on October 17 and corroborated the finding at 19:45 UTC.&lt;/p&gt;

&lt;p&gt;That is 100 minutes from report to Chrome's block.&lt;/p&gt;

&lt;p&gt;Mozilla received the report at 20:06 UTC. Its OneCRL update reached users with standard configurations by 22:59 UTC.&lt;/p&gt;

&lt;p&gt;eMudhra revoked all eight affected intermediates on October 18.&lt;/p&gt;

&lt;p&gt;There is another important detail: according to eMudhra, seven of the eight intermediates had never been used in production, and the one that had been used had not issued any certificates.&lt;/p&gt;

&lt;p&gt;This meant the immediate revocation of those eight intermediates did not invalidate customer certificates issued by them.&lt;/p&gt;

&lt;p&gt;But that wasn't the end of the incident.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why removing the root certificate was a bigger problem
&lt;/h2&gt;

&lt;p&gt;This is where the incident becomes particularly interesting from an infrastructure perspective.&lt;/p&gt;

&lt;p&gt;There are two different ways to respond to a compromised or defective certificate hierarchy:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Block the specific affected intermediate certificates or public keys.&lt;/li&gt;
&lt;li&gt;Remove the root certificate that establishes trust for the entire hierarchy.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The first approach is targeted.&lt;/p&gt;

&lt;p&gt;The second has a much wider impact.&lt;/p&gt;

&lt;p&gt;Google announced the removal of eMudhra's emSign ECC Root CA - G3 from the Chrome Root Store.&lt;/p&gt;

&lt;p&gt;That decision wasn't limited to the eight affected intermediates.&lt;/p&gt;

&lt;p&gt;Any certificate chain ending at that root could potentially be affected, including legitimate certificates issued through other intermediates.&lt;/p&gt;

&lt;p&gt;eMudhra subsequently reported that approximately 30,800 certificates in a US client ecosystem were issued under the affected G3 hierarchies.&lt;/p&gt;

&lt;p&gt;The company said this ecosystem already had an alternative issuance path through its G1 RSA root and was using that path for migration.&lt;/p&gt;

&lt;p&gt;One distinction matters here: those approximately 30,800 certificates should not be described as certificates issued by the eight weak intermediates. eMudhra said those eight had issued no end-entity certificates. The wider certificate population is associated with the root hierarchy, and the precise intermediate-to-certificate mapping has not been published.&lt;/p&gt;

&lt;p&gt;This is the difference between fixing a compromised signing path and withdrawing trust from an entire trust anchor.&lt;/p&gt;

&lt;h2&gt;
  
  
  The part I find most interesting: the audit gap
&lt;/h2&gt;

&lt;p&gt;The affected intermediates were created on April 30, 2025.&lt;/p&gt;

&lt;p&gt;The vulnerability was reported in October 2026.&lt;/p&gt;

&lt;p&gt;During that period, the hierarchy had been included in browser trust lists and had passed two independent audit cycles.&lt;/p&gt;

&lt;p&gt;Yet an external researcher was able to identify the weakness using the published certificates.&lt;/p&gt;

&lt;p&gt;That raises a question worth asking in any security-sensitive engineering environment:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Are we validating the process, or are we validating the actual artifact produced by that process?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;A documented key-generation ceremony can look correct. Hardware security modules can be present. Witnesses can be involved. Audit records can be complete.&lt;/p&gt;

&lt;p&gt;None of those things independently proves that the resulting public key has the expected cryptographic properties.&lt;/p&gt;

&lt;p&gt;The published key itself is what ultimately matters.&lt;/p&gt;

&lt;p&gt;This is not specific to certificate authorities. The same principle applies to signing keys, encryption keys, generated tokens, and other security-critical material.&lt;/p&gt;

&lt;h2&gt;
  
  
  A few engineering lessons
&lt;/h2&gt;

&lt;p&gt;There are several practical lessons here.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. Validate generated cryptographic material.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Don't rely exclusively on the procedure used to generate a key. Validate the resulting artifact against the expected cryptographic properties.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Treat backup and recovery paths as production security boundaries.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;eMudhra's account points to a key wrap/unwrap step as part of the failure. A recovery or transfer mechanism can influence production key material. It deserves the same scrutiny as the primary signing path.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Understand the difference between revocation and distrust.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Blocking a particular intermediate is not equivalent to removing its root. The latter can affect legitimate certificates that have nothing to do with the original defective keys.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. Maintain alternative issuance paths.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;eMudhra's existing G1 RSA issuance path became important during the migration. Redundant certificate issuance is not just an availability feature; it can become an essential part of incident recovery.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;5. Verify that security updates actually reach clients.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Chrome, Firefox, Android, NSS-based applications, operating-system trust stores and pinned clients do not necessarily receive or apply trust updates through the same mechanism.&lt;/p&gt;

&lt;p&gt;A published security update is not proof that every dependent client has applied it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Final thoughts
&lt;/h2&gt;

&lt;p&gt;I don't think the most useful takeaway from this incident is simply that a certificate authority generated weak keys.&lt;/p&gt;

&lt;p&gt;The more interesting engineering problem is the distance between a security control existing on paper and that control producing the expected result.&lt;/p&gt;

&lt;p&gt;The affected intermediates were revoked quickly. Browser vendors responded within hours. But the broader root-trust decision created a migration problem for a much larger certificate population.&lt;/p&gt;

&lt;p&gt;That is an important distinction when evaluating security incidents: the scope of the original defect and the scope of the remediation are not necessarily the same.&lt;/p&gt;

&lt;p&gt;The full incident analysis, including the source timeline, certificate hierarchy, trust-store behaviour, count discrepancies and unresolved questions, is available here:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://buildopsy.com/system-design/emudhra-weak-ca-keys-postmortem.html" rel="noopener noreferrer"&gt;Why Browsers Stopped Trusting eMudhra's Root — BuildOpsy&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;I think this is a useful case study for anyone working on PKI, cryptographic infrastructure, security engineering or systems that depend on externally maintained trust stores.&lt;/p&gt;

</description>
      <category>cybersecurity</category>
      <category>devops</category>
      <category>security</category>
      <category>programming</category>
    </item>
    <item>
      <title>Starlink's 2.5-Hour Global Outage: The Core Network Failed, Not the Satellites</title>
      <dc:creator>BuildOpsy</dc:creator>
      <pubDate>Tue, 10 Nov 2026 14:55:48 +0000</pubDate>
      <link>https://dev.to/buildopsy/starlinks-25-hour-global-outage-the-core-network-failed-not-the-satellites-3f4l</link>
      <guid>https://dev.to/buildopsy/starlinks-25-hour-global-outage-the-core-network-failed-not-the-satellites-3f4l</guid>
      <description>&lt;p&gt;On July 24, 2025, around 19:13 UTC, Starlink suffered a total, synchronized global outage. For roughly 2.5 hours, hundreds of thousands of user terminals (UTs) across North America, Europe, Australia, and active conflict zones dropped off the grid simultaneously. User terminals entered an endless loop of searching for satellites, unable to associate or pass traffic.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fhih7w10ltpo6cv6ph2ol.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fhih7w10ltpo6cv6ph2ol.png" alt=" The failure in one diagram: the space segment was not the reported failure, the core-services layer was, every gateway sat behind it, and SpaceX named no internal component. Buildopsy illustration from public reports, not an incident artifact." width="800" height="426"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Unlike traditional terrestrial ISP failures caused by a severed fiber backhaul or a localized BGP leak, this was a planetary-scale control plane and core routing failure.&lt;/p&gt;

&lt;p&gt;Let's break down the distributed systems architecture, what actually happened under the hood of a Low Earth Orbit (LEO) megaconstellation, and the engineering takeaways every infrastructure team can learn from it.&lt;/p&gt;




&lt;h2&gt;
  
  
  1. The Architecture of a LEO Mesh Network
&lt;/h2&gt;

&lt;p&gt;To understand why the failure was global and instantaneous, we have to look at how Starlink routes packets compared to legacy geostationary (GEO) satellites or terrestrial fiber.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;High-Velocity Topology:&lt;/strong&gt; Starlink's constellation consists of thousands of satellites moving at roughly 7.5 km/s in Low Earth Orbit. A single satellite is only in view of a given user terminal for a few minutes.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Dynamic Handovers:&lt;/strong&gt; User terminals constantly switch tracking beams from satellite to satellite. This requires a hyper-agile control plane that updates routing tables, beam schedules, and ground-station gateway links in real time.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The Core Network (Ground Stations &amp;amp; POPs):&lt;/strong&gt; Satellites act primarily as space-borne repeaters (though newer generations feature optical inter-satellite links). Traffic is beamed down to gateway earth stations, which route packets into Starlink’s software-defined core network and out to Internet Points of Presence (POPs).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;In a network where the topology changes second-by-second, &lt;strong&gt;software is the glue holding the physics together&lt;/strong&gt;.&lt;/p&gt;




&lt;h2&gt;
  
  
  2. What Happened? (The Forensic Timeline)
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;19:13 UTC — Simultaneous Global Drop:&lt;/strong&gt; ThousandEyes and global telemetry sensors registered an immediate, precipitous drop in traffic and availability. There was no rolling regional wave; users in London, Seattle, and Melbourne went dark at the exact same minute.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The Symptom (Control Plane vs. Data Plane):&lt;/strong&gt; User terminals could not associate with satellites, or when they briefly did, packets dropped instantly. This ruled out physical hardware failures (like a rocket or satellite anomaly), which would have been localized to specific orbital planes. Instead, it pointed squarely to a centralized or globally distributed &lt;strong&gt;control plane software service&lt;/strong&gt; failure.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;21:31 to 21:44 UTC — Staggered Recovery:&lt;/strong&gt; As core services were rolled back, patched, or restarted, telemetry showed a classic &lt;em&gt;staggered recovery pattern&lt;/em&gt;. As routing tables re-converged and control planes re-established neighbor states across thousands of moving nodes, connectivity trickled back online.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;SpaceX later confirmed that the outage stemmed from a failure in &lt;strong&gt;key internal software services operating the core network&lt;/strong&gt;, compounded during a software upgrade/rollout procedure.&lt;/p&gt;




&lt;h2&gt;
  
  
  3. Engineering Lessons for Distributed Systems
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Lesson 1: Global Blast Radius of Centralized Control Planes
&lt;/h3&gt;

&lt;p&gt;Even in a decentralized physical topology (thousands of independent satellites), a centralized or globally synchronized software dependency can create a single point of failure. If your configuration management, authorization, or dynamic routing orchestration layer assumes a global scope, a bad deploy can turn into a global outage instantly.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Mitigation:&lt;/strong&gt; Implement strict cell-based architectures or regional blast-radius isolation. If a control plane update fails, it should fail within a localized domain (e.g., a single orbital shell or regional gateway cluster), rather than taking down the entire planetary fleet.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Lesson 2: The Thundering Herd on Re-Convergence
&lt;/h3&gt;

&lt;p&gt;When a LEO network drops its routing state, millions of user terminals simultaneously attempt to re-authenticate, fetch ephemeris data, and re-establish handshakes. This creates a massive, synchronized distributed denial-of-service (DDoS) against your own authentication and gateway servers just as they are trying to recover.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Mitigation:&lt;/strong&gt; Build aggressive randomized exponential backoff and jitter into edge clients. When a terminal loses connection to the core, its retry logic must be mathematically randomized to prevent overwhelming recovering control-plane nodes.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Lesson 3: Transport Diversity is Non-Negotiable
&lt;/h3&gt;

&lt;p&gt;For enterprise, maritime, aviation, and mission-critical users, relying on a single LEO layer—no matter how redundant the constellation—creates systemic risk.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Mitigation:&lt;/strong&gt; True high availability requires heterogeneous transport diversity. Systems should dynamically blend LEO satellite links with 5G, traditional GEO, or terrestrial fiber via software-defined WAN (SD-WAN) layers that can fail over seamlessly when a control plane locks up.&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  Final Thoughts
&lt;/h2&gt;

&lt;p&gt;The Starlink July 2025 outage was a masterclass in how modern infrastructure has shifted from hardware reliability problems to &lt;strong&gt;software orchestration and control-plane complexity problems&lt;/strong&gt;. Managing a network moving at 7,500 meters per second leaves zero margin for error when pushing core updates.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Have you ever dealt with a global control-plane failure that bypassed your regional safeguards? Let’s talk about it in the comments.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>distributedsystems</category>
      <category>sre</category>
      <category>architecture</category>
      <category>devops</category>
    </item>
    <item>
      <title>AWS EC2 Pricing and Billing: The Meters Around the Instance-Hour</title>
      <dc:creator>BuildOpsy</dc:creator>
      <pubDate>Mon, 09 Nov 2026 05:40:04 +0000</pubDate>
      <link>https://dev.to/buildopsy/aws-ec2-pricing-and-billing-the-meters-around-the-instance-hour-150o</link>
      <guid>https://dev.to/buildopsy/aws-ec2-pricing-and-billing-the-meters-around-the-instance-hour-150o</guid>
      <description>&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fcuxv48pncjffzt5xeotr.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fcuxv48pncjffzt5xeotr.png" alt="EC2 billing schematic: a single instance-hour meter carries $700.80 a month for ten m5.large instances, beside separate EBS, Elastic IP, capacity-reservation and data-transfer meters that add to the same bill" width="800" height="420"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Figure 1. Illustrative schematic. The instance-hour total is one meter, not the bill: storage, addresses, reservations and transfer are billed separately. Run your own numbers in the &lt;a href="https://buildopsy.com/tools/ec2-compute-cost-calculator.html" rel="noopener noreferrer"&gt;EC2 Compute Cost Calculator&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;The instance-hour gets all the attention—and often accounts for only a fraction of the actual AWS EC2 invoice.&lt;/p&gt;

&lt;p&gt;Below are the US East (&lt;code&gt;us-east-1&lt;/code&gt;) Linux On-Demand rates for 13 common instance types (from the AWS price list published October 29, 2026) along with their 730-hour monthly cost, followed by the five billing meters the hourly price leaves out: &lt;strong&gt;the 60-second floor on short launches, attached EBS volumes on stopped instances, idle Elastic IPs, unused Capacity Reservations, and T-family Unlimited surplus credits&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Quick numbers from the worked model:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;10 × &lt;code&gt;m5.large&lt;/code&gt; running all month (730 hours):&lt;/strong&gt; &lt;strong&gt;$700.80&lt;/strong&gt; at &lt;code&gt;$0.0960&lt;/code&gt; per instance-hour.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;10,000 short jobs running 30 seconds each on &lt;code&gt;m5.large&lt;/code&gt;:&lt;/strong&gt; &lt;strong&gt;$16.00&lt;/strong&gt; for 83.33 hours of actual wall-clock work—because the &lt;strong&gt;60-second minimum floor&lt;/strong&gt; doubles the billed hours to 166.67.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;An On-Demand Capacity Reservation for 20 &lt;code&gt;m5.large&lt;/code&gt; instances with 12 in use:&lt;/strong&gt; bills &lt;strong&gt;$560.64/month&lt;/strong&gt; for the 8 idle slots nobody touched.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;You can test your own fleet numbers locally in the &lt;a href="https://buildopsy.com/tools/ec2-compute-cost-calculator.html" rel="noopener noreferrer"&gt;EC2 Compute Cost Calculator&lt;/a&gt; (or read the full teardown on &lt;a href="https://buildopsy.com/system-design/aws-ec2-pricing-model.html" rel="noopener noreferrer"&gt;Buildopsy&lt;/a&gt;).&lt;/p&gt;




&lt;h2&gt;
  
  
  1. What the Hourly Rate Actually Buys
&lt;/h2&gt;

&lt;p&gt;An On-Demand rate is a price for runtime, not for work done. AWS describes it as capacity by the hour or second with no long-term commitment, and it does not fall when the CPU sits idle. A ten-instance fleet that serves nothing for a month costs the same as one under load. Utilization changes how many instance-hours you need; it does not change the rate.&lt;/p&gt;

&lt;p&gt;The same pricing page separates what the instance charge covers from what it does not: storage, addressing, load balancing, monitoring, and transfer are billed on their own meters.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Scope note:&lt;/strong&gt; Rates move by Region, operating system, tenancy, purchase model, and instance family. The numbers below use one Region (&lt;code&gt;us-east-1&lt;/code&gt;), one OS (Linux), and one purchase model (On-Demand) so every calculation can be checked.&lt;/p&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  2. Thirteen US East Rates to Anchor Against
&lt;/h2&gt;

&lt;p&gt;These are Linux On-Demand rates for US East (N. Virginia), read from AWS's published EC2 pricing data (published October 29, 2026), covering burstable, general-purpose, compute-optimized, and memory-optimized shapes:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Instance type&lt;/th&gt;
&lt;th&gt;vCPU&lt;/th&gt;
&lt;th&gt;Memory&lt;/th&gt;
&lt;th&gt;USD per instance-hour&lt;/th&gt;
&lt;th&gt;Monthly at 730 h&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;t3.micro&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;1 GiB&lt;/td&gt;
&lt;td&gt;&lt;code&gt;$0.0104&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;$7.59&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;t4g.small&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;2 GiB&lt;/td&gt;
&lt;td&gt;&lt;code&gt;$0.0168&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;$12.26&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;t3.small&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;2 GiB&lt;/td&gt;
&lt;td&gt;&lt;code&gt;$0.0208&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;$15.18&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;t4g.medium&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;4 GiB&lt;/td&gt;
&lt;td&gt;&lt;code&gt;$0.0336&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;$24.53&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;t3.medium&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;4 GiB&lt;/td&gt;
&lt;td&gt;&lt;code&gt;$0.0416&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;$30.37&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;c7g.large&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;4 GiB&lt;/td&gt;
&lt;td&gt;&lt;code&gt;$0.0725&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;$52.93&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;c5.large&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;4 GiB&lt;/td&gt;
&lt;td&gt;&lt;code&gt;$0.0850&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;$62.05&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;m5.large&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;8 GiB&lt;/td&gt;
&lt;td&gt;&lt;code&gt;$0.0960&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;$70.08&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;m6i.large&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;8 GiB&lt;/td&gt;
&lt;td&gt;&lt;code&gt;$0.0960&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;$70.08&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;r5.large&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;16 GiB&lt;/td&gt;
&lt;td&gt;&lt;code&gt;$0.1260&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;$91.98&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;c5.xlarge&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;8 GiB&lt;/td&gt;
&lt;td&gt;&lt;code&gt;$0.1700&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;$124.10&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;m5.xlarge&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;16 GiB&lt;/td&gt;
&lt;td&gt;&lt;code&gt;$0.1920&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;$140.16&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;code&gt;r5.xlarge&lt;/code&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;32 GiB&lt;/td&gt;
&lt;td&gt;&lt;code&gt;$0.2520&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;$183.96&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;em&gt;(The monthly column is rate × 730 hours, the mean month at 8,760 hours/year.)&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Two rows are worth comparing side by side: &lt;code&gt;c7g.large&lt;/code&gt; and &lt;code&gt;c5.large&lt;/code&gt; both give 2 vCPU and 4 GiB, and the Graviton (&lt;code&gt;c7g.large&lt;/code&gt;) option lists at &lt;code&gt;$0.0725&lt;/code&gt; against &lt;code&gt;$0.0850&lt;/code&gt;—a difference of &lt;strong&gt;$9.13/month per instance&lt;/strong&gt;, or &lt;strong&gt;$912.50/month across 100 instances&lt;/strong&gt; for the same core and memory count.&lt;/p&gt;




&lt;h2&gt;
  
  
  3. The 60-Second Floor on Short Launches
&lt;/h2&gt;

&lt;p&gt;Partial instance-hours bill by the second, which usually gets read as &lt;em&gt;"a 30-second job costs 30 seconds."&lt;/em&gt; It does not.&lt;/p&gt;

&lt;p&gt;The &lt;a href="https://aws.amazon.com/ec2/pricing/" rel="noopener noreferrer"&gt;EC2 pricing page&lt;/a&gt; states a &lt;strong&gt;60-second minimum&lt;/strong&gt;, so any launch shorter than a minute bills a full 60 seconds. &lt;em&gt;(SUSE Linux Enterprise Server is the single exception listed: those instances bill a full hour regardless.)&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Take &lt;strong&gt;10,000 short CI or batch jobs&lt;/strong&gt; running &lt;strong&gt;30 seconds each&lt;/strong&gt; on &lt;code&gt;m5.large&lt;/code&gt;:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Wall-clock runtime:&lt;/strong&gt; &lt;code&gt;10,000 × 30 s = 300,000 s&lt;/code&gt; = &lt;strong&gt;83.33 hours&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Billed runtime at the 60-second floor:&lt;/strong&gt; &lt;code&gt;10,000 × 60 s = 600,000 s&lt;/code&gt; = &lt;strong&gt;166.67 hours&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Billed cost:&lt;/strong&gt; &lt;code&gt;166.67 × $0.0960&lt;/code&gt; = &lt;strong&gt;$16.00&lt;/strong&gt; (instead of &lt;code&gt;$8.00&lt;/code&gt;)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The floor is the economic argument for batching: launching once for 600 seconds bills 10 minutes, while launching twelve times for 30 seconds bills 12 minutes for 6 minutes of work.&lt;/p&gt;




&lt;h2&gt;
  
  
  4. What Keeps Billing After &lt;code&gt;StopInstances&lt;/code&gt;
&lt;/h2&gt;

&lt;p&gt;Stopping an EC2 instance ends the instance-hour charge, but several attached resources keep their own meters running:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Attached EBS volumes&lt;/strong&gt; bill by provisioned GB-month whether the instance is running or stopped (in one-second increments with the same 60-second minimum).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Every in-use and idle Elastic IP address&lt;/strong&gt; is charged.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Load balancers and CloudWatch&lt;/strong&gt; bill on their own product pages.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Auto Scaling&lt;/strong&gt; itself carries no separate fee—only the instances and monitoring it launches bill normally.&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  5. Capacity Reservations Bill the Slot, Not Just the Instance
&lt;/h2&gt;

&lt;p&gt;An On-Demand Capacity Reservation is &lt;a href="https://aws.amazon.com/ec2/pricing/on-demand/" rel="noopener noreferrer"&gt;priced exactly like equivalent On-Demand usage&lt;/a&gt;. If the reservation is 100% utilized, you pay instance usage and nothing extra. If it is partially used, &lt;strong&gt;you pay for the unused portion too.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;For &lt;strong&gt;20 reserved &lt;code&gt;m5.large&lt;/code&gt; instances&lt;/strong&gt; with &lt;strong&gt;12 running&lt;/strong&gt; for a full 730-hour month:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Total reservation commitment:&lt;/strong&gt; &lt;code&gt;20 × 730 × $0.0960&lt;/code&gt; = &lt;strong&gt;$1,401.60&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Used by running instances:&lt;/strong&gt; &lt;code&gt;12 × 730 × $0.0960&lt;/code&gt; = &lt;strong&gt;$840.96&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Billed for idle capacity:&lt;/strong&gt; &lt;code&gt;8 × 730 × $0.0960&lt;/code&gt; = &lt;strong&gt;$560.64&lt;/strong&gt;
&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  6. Purchase Options and What "Up To" Means
&lt;/h2&gt;

&lt;p&gt;AWS publishes headline discount ceilings rather than guaranteed rates:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Savings Plans:&lt;/strong&gt; up to &lt;strong&gt;72%&lt;/strong&gt; off On-Demand in exchange for an hourly spend commitment.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Spot Instances:&lt;/strong&gt; up to &lt;strong&gt;90%&lt;/strong&gt; off On-Demand in exchange for interruption risk.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;At the published 72% ceiling, a &lt;code&gt;$0.0960&lt;/code&gt; &lt;code&gt;m5.large&lt;/code&gt; would land at &lt;code&gt;$0.0269/hour&lt;/code&gt; (&lt;code&gt;$19.62&lt;/code&gt; across 730 hours)—but that is a maximum ceiling illustration, and actual plan rates depend on term, payment option, and family scope.&lt;/p&gt;




&lt;h2&gt;
  
  
  7. Burstable Instances (&lt;code&gt;T2&lt;/code&gt; / &lt;code&gt;T3&lt;/code&gt; / &lt;code&gt;T4g&lt;/code&gt;) and the Unlimited Credit Bill
&lt;/h2&gt;

&lt;p&gt;T-family instances accumulate CPU credits while idle and spend them when bursting. In &lt;strong&gt;Unlimited mode&lt;/strong&gt;, they can burst past their credit balance, and surplus credits bill per vCPU-hour regardless of instance size or purchase option:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;T2 and T3 (Linux, RHEL, SLES):&lt;/strong&gt; &lt;code&gt;$0.05&lt;/code&gt; per vCPU-hour&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;T4g (Linux, RHEL, SLES):&lt;/strong&gt; &lt;code&gt;$0.04&lt;/code&gt; per vCPU-hour&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;T8i (Linux, RHEL, SLES):&lt;/strong&gt; &lt;code&gt;$0.06&lt;/code&gt; per vCPU-hour&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;T2 and T3 (Windows / SQL Web):&lt;/strong&gt; &lt;code&gt;$0.096&lt;/code&gt; per vCPU-hour&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;T8i (Windows / SQL Web):&lt;/strong&gt; &lt;code&gt;$0.1152&lt;/code&gt; per vCPU-hour&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  8. Windows &amp;amp; SQL Licences: The &lt;code&gt;Optimize CPUs&lt;/code&gt; Lever
&lt;/h2&gt;

&lt;p&gt;Windows and SQL Server images bundle the licence into the hourly instance price. AWS exposes a lever here with no Linux equivalent: &lt;strong&gt;&lt;a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-optimize-cpu.html" rel="noopener noreferrer"&gt;Optimize CPUs&lt;/a&gt;&lt;/strong&gt;, which lowers the active vCPU count or disables hyper-threading while keeping the same instance RAM and network tier.&lt;/p&gt;

&lt;p&gt;Published savings per vCPU-hour removed:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Windows Server:&lt;/strong&gt; &lt;code&gt;$0.046&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Windows with SQL Enterprise:&lt;/strong&gt; &lt;code&gt;$0.421&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;SQL Standard:&lt;/strong&gt; &lt;code&gt;$0.166&lt;/code&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;SQL Web:&lt;/strong&gt; &lt;code&gt;$0.063&lt;/code&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;On an 8-vCPU &lt;code&gt;m5.2xlarge&lt;/code&gt;, dropping active vCPUs from 8 to 4 saves &lt;code&gt;4 × 730 × $0.046&lt;/code&gt; = &lt;strong&gt;$134.32/month&lt;/strong&gt; on the Windows Server licence component alone.&lt;/p&gt;




&lt;h2&gt;
  
  
  9. Data Transfer Is Aggregated Across Services
&lt;/h2&gt;

&lt;p&gt;Outbound internet transfer is aggregated across EC2, S3, Glacier, RDS, Redshift, SageMaker, SES, SimpleDB, SQS, SNS, DynamoDB, Storage Gateway, CloudShell, and CloudWatch Logs, with a &lt;strong&gt;100 GB/month free outbound allowance&lt;/strong&gt; across AWS services and Regions (except China and GovCloud). A terabyte leaving an EC2 load balancer and a terabyte leaving an S3 bucket climb the exact same tier ladder (see our &lt;a href="https://buildopsy.com/system-design/aws-s3-pricing-model.html" rel="noopener noreferrer"&gt;S3 pricing teardown&lt;/a&gt; and &lt;a href="https://buildopsy.com/system-design/cdn-egress-pricing-model.html" rel="noopener noreferrer"&gt;CDN/egress model&lt;/a&gt;).&lt;/p&gt;




&lt;h2&gt;
  
  
  10. Run Your Own Numbers
&lt;/h2&gt;

&lt;p&gt;An EC2 bill is a runtime meter surrounded by meters that do not care whether anything ran: a 60-second floor that doubles short launches, EBS volumes and Elastic IPs that outlive stopped instances, and Capacity Reservations that bill empty slots.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Interactive Calculator:&lt;/strong&gt; &lt;a href="https://buildopsy.com/tools/ec2-compute-cost-calculator.html" rel="noopener noreferrer"&gt;EC2 Compute Cost Calculator&lt;/a&gt; (models instance-hours, the 60-second short-launch floor, and attached EBS in your browser)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Full Rate-Card Audit &amp;amp; FAQ:&lt;/strong&gt; &lt;a href="https://buildopsy.com/system-design/aws-ec2-pricing-model.html" rel="noopener noreferrer"&gt;AWS EC2 Pricing and Billing: The Meters Around the Instance-Hour&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;&lt;strong&gt;Question for everyone running workloads on EC2:&lt;/strong&gt; Which "non-instance" meter has surprised you most on a real AWS bill—forgotten EBS volumes on stopped instances, T-family Unlimited surplus credits, NAT Gateway / egress traffic, or idle Elastic IPs?&lt;/p&gt;

</description>
      <category>aws</category>
      <category>cloud</category>
      <category>devops</category>
      <category>architecture</category>
    </item>
    <item>
      <title>AWS S3 Pricing and Billing: Storage, Request Fees and Egress Costs Explained</title>
      <dc:creator>BuildOpsy</dc:creator>
      <pubDate>Sat, 07 Nov 2026 22:27:48 +0000</pubDate>
      <link>https://dev.to/buildopsy/aws-s3-pricing-and-billing-storage-request-fees-and-egress-costs-explained-39d1</link>
      <guid>https://dev.to/buildopsy/aws-s3-pricing-and-billing-storage-request-fees-and-egress-costs-explained-39d1</guid>
      <description>&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F48lt5g382pmrmcw2q0z9.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F48lt5g382pmrmcw2q0z9.png" alt="Worked S3 Standard estimate: $23.55 storage, $5 PUT and $4 GET requests total $32.55 before transfer and management charges" width="800" height="420"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Figure 1. The same inputs used by the &lt;a href="https://buildopsy.com/tools/aws-s3-storage-cost-calculator.html" rel="noopener noreferrer"&gt;Buildopsy S3 calculator&lt;/a&gt;. The $32.55 subtotal excludes transfer and other enabled features.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;I start with the storage rate because it is the easiest line to quote. Then I price requests, retrieval, transfer, object size and retention as separate meters. This worked model uses 1 TiB in S3 Standard to show what one million PUTs and ten million GETs add.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;TL;DR:&lt;/strong&gt; 1,024 billable GB and one million PUTs and ten million GETs produce a &lt;strong&gt;$32.55&lt;/strong&gt; modeled monthly subtotal at the example US East rates. It excludes transfer, management features, replication, tax, account credits and negotiated pricing. Rerun the inputs in the &lt;a href="https://buildopsy.com/tools/aws-s3-storage-cost-calculator.html" rel="noopener noreferrer"&gt;AWS S3 Storage Cost Calculator&lt;/a&gt; (or read the original audit on &lt;a href="https://buildopsy.com/system-design/aws-s3-pricing-model.html" rel="noopener noreferrer"&gt;Buildopsy&lt;/a&gt;).&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Scope:&lt;/strong&gt; The worked model is S3 Standard in US East (N. Virginia) over a full billing month; the rate tables cover the eight classes AWS prices on its S3 page and four Regions. Rates are public list prices in USD. This is a modeled estimate, not an AWS quote. AWS measures storage in binary GB, so 1 TiB is represented as 1,024 billable GB. Check the &lt;a href="https://aws.amazon.com/s3/pricing/" rel="noopener noreferrer"&gt;live S3 pricing page&lt;/a&gt; before budgeting.&lt;/p&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  S3 Rates at a Glance
&lt;/h2&gt;

&lt;p&gt;Every rate in the first two tables is a US East (N. Virginia) list price from the AWS price list snapshot published October 15, 2026; the Deep Archive row comes from the separate list noted at the end of this section, and the four-Region table uses each Region's own index at the same version. The worked model in this article uses S3 Standard. The other classes matter because the storage rate alone misleads: Glacier Instant Retrieval stores at 17% of the Standard rate ($0.004 ÷ $0.023) and then charges $0.03 for every gigabyte read back.&lt;/p&gt;

&lt;h3&gt;
  
  
  Storage: the eight classes AWS prices on the S3 page
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Storage class&lt;/th&gt;
&lt;th&gt;Storage per GB-month&lt;/th&gt;
&lt;th&gt;Retrieval per GB&lt;/th&gt;
&lt;th&gt;Object-size rule&lt;/th&gt;
&lt;th&gt;Minimum duration&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;S3 Standard&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;$0.023 first 50 TB; $0.022 next 450 TB; $0.021 above 500 TB&lt;/td&gt;
&lt;td&gt;None&lt;/td&gt;
&lt;td&gt;None&lt;/td&gt;
&lt;td&gt;None&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;S3 Intelligent-Tiering&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;$0.023 / $0.022 / $0.021 Frequent Access; $0.0125 Infrequent Access; $0.004 Archive Instant Access; $0.0036 Archive Access; $0.00099 Deep Archive Access&lt;/td&gt;
&lt;td&gt;None for standard and bulk; expedited from Archive Access bills at the expedited rate&lt;/td&gt;
&lt;td&gt;None; objects under 128 KiB stay in Frequent Access and are not monitored&lt;/td&gt;
&lt;td&gt;None&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;S3 Standard-IA&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;$0.0125&lt;/td&gt;
&lt;td&gt;$0.01&lt;/td&gt;
&lt;td&gt;128 KiB minimum billable&lt;/td&gt;
&lt;td&gt;30 days&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;S3 One Zone-IA&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;$0.01&lt;/td&gt;
&lt;td&gt;$0.01&lt;/td&gt;
&lt;td&gt;128 KiB minimum billable&lt;/td&gt;
&lt;td&gt;30 days&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;S3 Glacier Instant Retrieval&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;$0.004&lt;/td&gt;
&lt;td&gt;$0.03&lt;/td&gt;
&lt;td&gt;128 KiB minimum billable&lt;/td&gt;
&lt;td&gt;90 days&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;S3 Glacier Flexible Retrieval&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;$0.0036&lt;/td&gt;
&lt;td&gt;$0.01 standard; $0.03 expedited; bulk free&lt;/td&gt;
&lt;td&gt;40 KiB overhead per object&lt;/td&gt;
&lt;td&gt;90 days&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;S3 Glacier Deep Archive&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;$0.00099&lt;/td&gt;
&lt;td&gt;$0.02 standard; $0.0025 bulk&lt;/td&gt;
&lt;td&gt;40 KiB overhead per object&lt;/td&gt;
&lt;td&gt;180 days&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;S3 Express One Zone&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;$0.11&lt;/td&gt;
&lt;td&gt;$0.0006&lt;/td&gt;
&lt;td&gt;None&lt;/td&gt;
&lt;td&gt;None&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Intelligent-Tiering bills a separate monitoring and automation charge of $0.0025 per 1,000 objects per month. Among the classes in this table, Express One Zone is the only one priced above Standard per stored GB, at 4.8 times the first-tier rate ($0.11 ÷ $0.023). Three per-GB meters sit outside the table. Reduced Redundancy is still listed in the same price list at $0.024 for the first TB and above Standard at every tier, and AWS's storage-class documentation marks it not recommended, so it is excluded here rather than ranked. S3 Vectors and S3 Files appear on the same pricing page as separate products with their own meters, not as storage classes.&lt;/p&gt;

&lt;h3&gt;
  
  
  Requests: the price moves with the class
&lt;/h3&gt;

&lt;p&gt;A PUT into Standard and a PUT into Glacier Instant Retrieval are not the same meter. The per-request rate changes by class, and for the archive classes the write path changes too.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Storage class&lt;/th&gt;
&lt;th&gt;PUT, COPY, POST per 1,000&lt;/th&gt;
&lt;th&gt;GET and all other requests per 1,000&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;S3 Standard&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;$0.005&lt;/td&gt;
&lt;td&gt;$0.0004&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;S3 Intelligent-Tiering&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;$0.005&lt;/td&gt;
&lt;td&gt;$0.0004&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;S3 Standard-IA&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;$0.01&lt;/td&gt;
&lt;td&gt;$0.001&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;S3 One Zone-IA&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;$0.01&lt;/td&gt;
&lt;td&gt;$0.001&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;S3 Glacier Instant Retrieval&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;$0.02&lt;/td&gt;
&lt;td&gt;$0.01&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;S3 Glacier Flexible Retrieval&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;$0.03&lt;/td&gt;
&lt;td&gt;$0.0004&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;S3 Glacier Deep Archive&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;$0.05, the same rate as a lifecycle transition in&lt;/td&gt;
&lt;td&gt;$0.0004&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;S3 Express One Zone&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;$0.00113&lt;/td&gt;
&lt;td&gt;$0.00003&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;LIST requests are the exception: AWS charges them at the S3 Standard PUT rate on every class, so a LIST costs $0.005 per 1,000 even where a PUT costs $0.02 or $0.03. On the write meters the spread is wide enough to change a design. A PUT costs 4 times more in Glacier Instant Retrieval than in Standard ($0.02 ÷ $0.005) and a GET costs 25 times more ($0.01 ÷ $0.0004). Express One Zone inverts it: its requests are cheaper than Standard's by 4.4 times for writes and 13.3 times for reads, which is why the $0.11 storage rate is not the whole comparison. Multipart upload parts to Glacier Flexible Retrieval bill at $0.005 per 1,000, separate from the $0.03 per 1,000 object-level writes, and Deep Archive parts bill at $0.005 per 1,000 against its $0.05 object-level write.&lt;/p&gt;

&lt;h3&gt;
  
  
  How the rate moves across four Regions
&lt;/h3&gt;

&lt;p&gt;All four Regions below are published under the same price list version, so this is a like-for-like comparison rather than a snapshot mismatch.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Region&lt;/th&gt;
&lt;th&gt;Standard first 50 TB&lt;/th&gt;
&lt;th&gt;Standard 50–500 TB&lt;/th&gt;
&lt;th&gt;Standard above 500 TB&lt;/th&gt;
&lt;th&gt;Standard-IA&lt;/th&gt;
&lt;th&gt;Glacier Instant Retrieval&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;strong&gt;US East (N. Virginia)&lt;/strong&gt;, &lt;code&gt;us-east-1&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;$0.023&lt;/td&gt;
&lt;td&gt;$0.022&lt;/td&gt;
&lt;td&gt;$0.021&lt;/td&gt;
&lt;td&gt;$0.0125&lt;/td&gt;
&lt;td&gt;$0.004&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;strong&gt;US West (Oregon)&lt;/strong&gt;, &lt;code&gt;us-west-2&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;$0.023&lt;/td&gt;
&lt;td&gt;$0.022&lt;/td&gt;
&lt;td&gt;$0.021&lt;/td&gt;
&lt;td&gt;$0.0125&lt;/td&gt;
&lt;td&gt;$0.004&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;strong&gt;Europe (Ireland)&lt;/strong&gt;, &lt;code&gt;eu-west-1&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;$0.023&lt;/td&gt;
&lt;td&gt;$0.022&lt;/td&gt;
&lt;td&gt;$0.021&lt;/td&gt;
&lt;td&gt;$0.0125&lt;/td&gt;
&lt;td&gt;$0.004&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;strong&gt;Asia Pacific (Mumbai)&lt;/strong&gt;, &lt;code&gt;ap-south-1&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;$0.025&lt;/td&gt;
&lt;td&gt;$0.024&lt;/td&gt;
&lt;td&gt;$0.023&lt;/td&gt;
&lt;td&gt;$0.0138&lt;/td&gt;
&lt;td&gt;$0.005&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Moving the same bucket from US East to Mumbai raises the first-tier Standard rate by 8.7% ($0.025 ÷ $0.023 = 1.087), Standard-IA by 10.4% ($0.0138 ÷ $0.0125 = 1.104) and Glacier Instant Retrieval by 25% ($0.005 ÷ $0.004 = 1.25). Request prices for these three classes are identical in all four Regions: $0.005 and $0.0004 per 1,000 for Standard, $0.01 and $0.001 for Standard-IA, $0.02 and $0.01 for Glacier Instant Retrieval. The colder request meters are not identical. A Glacier Flexible Retrieval PUT is $0.03 per 1,000 in US East and US West, $0.033 in Ireland and $0.036 in Mumbai, and a Deep Archive lifecycle transition is $0.05, $0.055 and $0.07 per 1,000 in the same order. So the storage rate is the only Regional difference among the three classes in this table, not across every S3 meter. Four Regions are a sample, not a rule; check the price list for the Region you deploy in.&lt;/p&gt;

&lt;p&gt;At these rates, 1,024 billable GB for a full month and 1 million PUTs and 10 million GETs is $23.55 + $5.00 + $4.00 = &lt;strong&gt;$32.55&lt;/strong&gt; before retrieval and transfer. &lt;a href="https://buildopsy.com/tools/aws-s3-storage-cost-calculator.html" rel="noopener noreferrer"&gt;Run your own inputs in the S3 calculator&lt;/a&gt;. Compute meters behave differently: the &lt;a href="https://buildopsy.com/system-design/aws-ec2-pricing-model.html" rel="noopener noreferrer"&gt;EC2 pricing and billing teardown&lt;/a&gt; prices the 60-second floor, idle Elastic IPs and capacity reservations that sit outside an instance-hour.&lt;/p&gt;

&lt;h3&gt;
  
  
  What the free tier actually covers
&lt;/h3&gt;

&lt;p&gt;The shape of this changed in 2025. AWS's S3 pricing page states that, as of July 15, 2025, new AWS customers receive up to $200 in AWS Free Tier credits applicable to eligible services including Amazon S3, and that all Free Tier credits must be used within 12 months of account creation. The Free plan itself runs for six months: AWS's Free Tier page says the account closes six months after it is opened, or when the credits run out, whichever comes first. Either way it is a credit balance rather than a permanent monthly storage and request allowance, so a bucket does not reset to free each month once the credits are spent.&lt;/p&gt;

&lt;p&gt;The one standing allowance is on transfer: 100 GB per month of data transfer out to the internet, which AWS aggregates across all AWS services and Regions except China and GovCloud. Storage, requests and retrieval are billed at the rates above from the first byte; the transfer allowance does not offset them.&lt;/p&gt;

&lt;h3&gt;
  
  
  1 TiB storage-only comparison
&lt;/h3&gt;

&lt;p&gt;Using the same US East (N. Virginia) list rates and &lt;strong&gt;1 TiB = 1,024 GiB = 1,024 billable GB&lt;/strong&gt;, the storage-only monthly charge is:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Storage class&lt;/th&gt;
&lt;th&gt;US East list rate&lt;/th&gt;
&lt;th&gt;1 TiB storage-only&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;S3 Standard&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;$0.023 per GB-month (first 50 TB tier)&lt;/td&gt;
&lt;td&gt;$23.55&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;S3 Standard-IA&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;$0.0125 per GB-month&lt;/td&gt;
&lt;td&gt;$12.80&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;S3 One Zone-IA&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;$0.01 per GB-month&lt;/td&gt;
&lt;td&gt;$10.24&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;S3 Glacier Instant Retrieval&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;$0.004 per GB-month&lt;/td&gt;
&lt;td&gt;$4.10&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;S3 Glacier Flexible Retrieval&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;$0.0036 per GB-month&lt;/td&gt;
&lt;td&gt;$3.69&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;S3 Glacier Deep Archive&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;$0.00099 per GB-month&lt;/td&gt;
&lt;td&gt;$1.01&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;S3 Express One Zone&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;$0.11 per GB-month&lt;/td&gt;
&lt;td&gt;$112.64&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Each result is 1,024 × the listed rate, rounded to cents. Unrounded: $23.552, $12.80, $10.24, $4.096, $3.6864, $1.01376 and $112.64. The Standard row uses the first 50 TB tier, and the Intelligent-Tiering Frequent Access tier is identical at $0.023 and is omitted rather than repeated. The remaining rows use their flat US East storage rates, with Deep Archive from the separate Glacier Deep Archive price list cited below. This is storage only: it excludes requests, retrieval, transfer, minimum-storage-duration charges and additional billable metadata overhead.&lt;/p&gt;




&lt;h2&gt;
  
  
  1. The Storage Rate Is Not the Bill
&lt;/h2&gt;

&lt;p&gt;S3 pricing is a sum of usage meters. AWS lists storage, requests, retrieval, transfer, management, replication, and transform and query features as separate cost components. The right question is not "What does a gigabyte cost?" It is "How many billable bytes, operations and transfers does this workload create in each class?"&lt;/p&gt;

&lt;p&gt;The model below uses a general-purpose bucket in S3 Standard in US East (N. Virginia). Its storage input is 1 TiB, or 1,024 billable GB. The example rate is $0.023 per billable GB-month in the first 50 TB tier. Public AWS prices vary by Region, class and volume. The rate is an input, not a permanent constant.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Boundary:&lt;/strong&gt; S3 Standard has no per-GB retrieval fee. That does not make reads free. GET requests are billed by request count, then data leaving S3 can add a separate transfer charge.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  2. Rebuild the $32.55 Worked Model
&lt;/h2&gt;

&lt;p&gt;The &lt;a href="https://buildopsy.com/tools/aws-s3-storage-cost-calculator.html" rel="noopener noreferrer"&gt;Buildopsy S3 pricing calculator&lt;/a&gt; starts with stored volume, PUT count, GET count and retrieval. At the example rates, the arithmetic is:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Line item&lt;/th&gt;
&lt;th&gt;Calculation&lt;/th&gt;
&lt;th&gt;Modeled charge&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;S3 Standard storage, first 50 TB tier&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;1,024 GB-month × $0.023&lt;/td&gt;
&lt;td&gt;$23.55&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;PUT requests&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;1,000,000 ÷ 1,000 × $0.005&lt;/td&gt;
&lt;td&gt;$5.00&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;GET requests&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;10,000,000 ÷ 1,000 × $0.0004&lt;/td&gt;
&lt;td&gt;$4.00&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Class retrieval&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;0 GB × entered rate&lt;/td&gt;
&lt;td&gt;$0.00&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Modeled subtotal&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Storage + requests + retrieval&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;$32.55&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;At these inputs, requests contribute $9.00. That is about 28 percent of the subtotal before any data leaves the bucket. Raising GET volume from 10 million to 100 million adds $36.00 at the same example rate. The 1 TiB storage line remains $23.55. Requests now contribute $45.00, almost twice the storage line. This is how a request-heavy workload can outgrow the number everyone quotes per gigabyte.&lt;/p&gt;

&lt;p&gt;The model prices request counts, not payload size. One GET of a large object and one GET of a small object are each one request for this line item. Their transfer and storage consequences differ. So does a workload that repeatedly lists millions of keys. Browsing S3 in the console can also create billable GET, LIST and other requests.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. Object Size Can Reverse the Storage-Class Choice
&lt;/h2&gt;

&lt;p&gt;S3 Standard has no minimum billable object size. Standard-IA and One Zone-IA bill at least 128 KiB per object and impose a 30-day minimum duration. Glacier Instant Retrieval has a 128 KiB minimum and a 90-day minimum. Glacier Flexible Retrieval has a 90-day minimum. Deep Archive has a 180-day minimum. Both archive classes add 40 KiB of metadata per object: 8 KiB billed at S3 Standard rates and 32 KiB billed at the applicable Glacier rate. These details matter when a pipeline creates millions of small records.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F9y95vwqij69ihmyzv9xi.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F9y95vwqij69ihmyzv9xi.png" alt="S3 Standard bills a 1 KiB object as 1 KiB. Standard-IA, One Zone-IA and Glacier Instant Retrieval bill 128 KiB, with 30- or 90-day minimums. Glacier Flexible, Deep Archive add 40 KiB metadata with 90-, and 180-day minimums." width="800" height="420"&gt;&lt;/a&gt;&lt;br&gt;
&lt;em&gt;Figure 2. Documented object-size and retention minimums. The diagram is not a complete class bill.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;For a 1 KiB object in Standard-IA, the 128 KiB floor represents 128 times the payload in billable storage units before the 30-day duration rule. A class can still win when data stays long enough and is read rarely. It can lose when a job writes tiny objects, transitions each one, then deletes or overwrites them early.&lt;/p&gt;

&lt;p&gt;Intelligent-Tiering has a different constraint. Objects under 128 KiB are not monitored for automatic tiering and remain in Frequent Access. For Glacier Flexible Retrieval or Deep Archive, each archived object also carries the documented metadata overhead. Count objects and transitions, not only terabytes.&lt;/p&gt;

&lt;h2&gt;
  
  
  4. A GET Request and a Retrieval Fee Are Different Meters
&lt;/h2&gt;

&lt;p&gt;Every class has request pricing. Some less-frequent and archive classes add per-GB retrieval pricing. Standard-IA, One Zone-IA and Glacier Instant Retrieval charge retrieval when data is read. Intelligent-Tiering has no retrieval fee for its standard and bulk retrieval paths, though its monitoring charge and optional accelerated archive retrieval have separate rules.&lt;/p&gt;

&lt;p&gt;Restoring an object from Glacier Flexible Retrieval or Deep Archive is not the same as a normal Standard GET. A restore creates a temporary accessible copy for a chosen period. During that window, storage can be billed for both the archive and the restored Standard copy. The archive still has its minimum-duration rules.&lt;/p&gt;

&lt;p&gt;Lifecycle is a request stream too. A rule that moves ten million objects can generate per-object transition requests. If objects are then expired before a class minimum has elapsed, the remaining minimum-duration charge can still apply.&lt;/p&gt;

&lt;h2&gt;
  
  
  5. Transfer, Versioning and Features Sit Outside the Base
&lt;/h2&gt;

&lt;p&gt;Data transfer is a separate line. AWS lists a first 100 GB per month internet-transfer allowance aggregated across AWS services and Regions, with China and GovCloud exceptions. Same-Region transfers between S3 and AWS services and transfers to CloudFront have stated exceptions. Cross-Region transfer, transfer acceleration and internet egress follow their own pricing. The transfer side gets its own teardown: &lt;a href="https://buildopsy.com/system-design/cdn-egress-pricing-model.html" rel="noopener noreferrer"&gt;the $0.09 toll&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;This is why the S3 tool leaves transfer out. Use the &lt;a href="https://buildopsy.com/tools/cloud-egress-cost-calculator.html" rel="noopener noreferrer"&gt;Cloud Egress Cost Calculator&lt;/a&gt; with the applicable billable volume and rate. Do not subtract a free allowance twice if other services in the account already used it.&lt;/p&gt;

&lt;p&gt;Versioning changes retention. An overwrite creates a new full object version. The older version stays billable until a noncurrent-version lifecycle rule expires it or a version-specific permanent delete removes it. In a versioned bucket, a current-version expiration rule generally adds a delete marker rather than removing older versions. Cross-Region Replication creates destination storage and request and transfer charges. S3 Inventory, Storage Lens, Object Tagging, Intelligent-Tiering monitoring, S3 Select, S3 Tables and S3 Vectors add distinct cost dimensions. They are outside the $32.55 example.&lt;/p&gt;

&lt;h2&gt;
  
  
  6. Reconcile the Model With the Bill
&lt;/h2&gt;

&lt;p&gt;Use AWS Cost and Usage Reports for billed cost. S3 usage reports can break activity out by operation, usage type, Region, resource, request count and transfer volume. AWS lets you aggregate its S3 usage report hourly, daily or monthly. That is the bridge from application telemetry to the invoice.&lt;/p&gt;

&lt;h3&gt;
  
  
  Representative S3 usage types in a bill
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Usage-type pattern&lt;/th&gt;
&lt;th&gt;Charge&lt;/th&gt;
&lt;th&gt;Meter to reconcile&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;TimedStorage-*ByteHrs&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Storage usage by storage class&lt;/td&gt;
&lt;td&gt;Billable GB-month for the class, including applicable minimum-size rules. Minimum-duration remainders appear under &lt;code&gt;EarlyDelete-*&lt;/code&gt;.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;Requests-Tier1&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;PUT, COPY, POST and LIST requests&lt;/td&gt;
&lt;td&gt;Request count per 1,000 at the class rate; LIST requests use the S3 Standard PUT/COPY/POST rate.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;Requests-Tier2&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;GET and other applicable requests&lt;/td&gt;
&lt;td&gt;Request count per 1,000; Standard GET has no per-GB retrieval fee, while colder classes can add retrieval.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;Retrieval-SIA&lt;/code&gt; / &lt;code&gt;Retrieval-GIR&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Standard-IA and Glacier Instant Retrieval retrieval&lt;/td&gt;
&lt;td&gt;Retrieved GB at the class rate; does not apply to S3 Standard.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;DataTransfer-Out-Bytes&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Internet data transfer out&lt;/td&gt;
&lt;td&gt;Billable GB leaving S3, after applicable free allowances and tiers.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;EarlyDelete-*&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Minimum-storage-duration charges&lt;/td&gt;
&lt;td&gt;Prorated GB-hours when an object leaves a class before its 30-, 90- or 180-day minimum.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  7. The Verdict: Price the Access Pattern
&lt;/h2&gt;

&lt;p&gt;The storage-class rate is a useful lower bound, not a workload estimate. A small-object archive can pay for unused minimum bytes. A hot bucket can pay more for operations or transfer than for storage. A versioned bucket can retain every overwritten byte until lifecycle rules remove it.&lt;/p&gt;

&lt;p&gt;Start with the workload shape: object-size distribution, change rate, reads per object, retrieval volume, retention period and destination. Price each meter against the actual Region and class. Run the first four meters through the &lt;a href="https://buildopsy.com/tools/aws-s3-storage-cost-calculator.html" rel="noopener noreferrer"&gt;S3 calculator&lt;/a&gt;, add transfer separately and reconcile the result against CUR line items. The bill hides in the boundary between stored bytes and the work required to use them.&lt;/p&gt;

&lt;p&gt;For a disclosed data-lake architecture with an explicit modeled bill, see &lt;a href="https://buildopsy.com/system-design/notion-data-lake-cost-postmortem.html" rel="noopener noreferrer"&gt;Notion's 200 Billion Blocks and the Warehouse Bill Behind a Blank Page&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>aws</category>
      <category>cloud</category>
      <category>devops</category>
      <category>architecture</category>
    </item>
    <item>
      <title>The three meters behind a $65 million observability bill</title>
      <dc:creator>BuildOpsy</dc:creator>
      <pubDate>Thu, 29 Oct 2026 03:59:18 +0000</pubDate>
      <link>https://dev.to/buildopsy/the-three-meters-behind-a-65-million-observability-bill-42dd</link>
      <guid>https://dev.to/buildopsy/the-three-meters-behind-a-65-million-observability-bill-42dd</guid>
      <description>&lt;p&gt;$65 million is a strange number to hear on an earnings call. It gets stranger when you work out what it buys at list price.&lt;/p&gt;

&lt;p&gt;On 4 May 2023, Datadog reported first-quarter billings of $511 million, up 15% year over year, and flagged one exception: a large upfront bill for a single customer in Q1 2022 that did not recur at the same level or timing. A JPMorgan analyst, Mark Murphy, did the subtraction on the call and arrived at roughly $65 million. Datadog's CFO, David Obstler, confirmed the shape without the name: a crypto company, an early optimizer, now billed in smaller chunks.&lt;/p&gt;

&lt;p&gt;A week later, Gergely Orosz reported that current and former Coinbase engineers had confirmed to him that the $65 million was Coinbase's Datadog bill for 2021, settled in the first quarter of 2022. Coinbase has never confirmed it publicly. Datadog never named the customer. Everything below rests on that sourcing.&lt;/p&gt;

&lt;p&gt;The number is worth taking seriously rather than gawking at, because it is not really a story about one company overspending. It is a story about three meters that compound independently, and what happens when nobody owns any of them.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmm0oxxwkb25x4ig8uz2p.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fmm0oxxwkb25x4ig8uz2p.png" alt="Datadog observability bill waterfall: host fleet, log ingest, and indexed events combine into the monthly meter, with Coinbase's $65M 2021 bill as the scale check" width="800" height="420"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Three meters, three units, three growth curves
&lt;/h2&gt;

&lt;p&gt;Observability vendors do not sell one meter. At Datadog, the ones that stack are these, at the annual-billing list rates published today:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Meter&lt;/th&gt;
&lt;th&gt;List price&lt;/th&gt;
&lt;th&gt;What grows it&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Infrastructure, Enterprise&lt;/td&gt;
&lt;td&gt;$23 per host per month&lt;/td&gt;
&lt;td&gt;Fleet size&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Log ingest&lt;/td&gt;
&lt;td&gt;$0.10 per GB ingested&lt;/td&gt;
&lt;td&gt;Verbosity × volume&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Standard indexing, 15-day retention&lt;/td&gt;
&lt;td&gt;$1.70 per million events&lt;/td&gt;
&lt;td&gt;Event volume × retention&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;APM&lt;/td&gt;
&lt;td&gt;$31 per host per month, with infrastructure attached&lt;/td&gt;
&lt;td&gt;Services × hosts&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;On-demand rates are higher across the board: $27 per host, $2.55 per million indexed events, $36 per APM host.&lt;/p&gt;

&lt;p&gt;The compounding is the point. Hosts grow with the fleet. Ingested gigabytes grow with log verbosity, and retention pressure multiplies whatever you ship. Indexed events grow with cardinality, and custom metrics beyond the per-host allotment — 100 on Pro, 200 on Enterprise — bill by usage, so every new tag value on a custom metric mints billable series. A team that adds hosts, turns up log verbosity, and tags everything has tripled its bill three independent ways before writing a single alert.&lt;/p&gt;

&lt;h2&gt;
  
  
  What $65 million means in each unit
&lt;/h2&gt;

&lt;p&gt;At $23 per host per month, $65 million is about 235,000 host-years of Enterprise infrastructure monitoring. That is arithmetic on list price, not a claim about what Coinbase bought, but it is the right order of magnitude for a company running a large fleet at the peak of a growth phase.&lt;/p&gt;

&lt;p&gt;The small-scale version is more useful. Twenty hosts at $23 comes to $460 a month. A thousand gigabytes of ingest at $0.10 adds $100. That is $560 a month, or $6,720 a year — the default output of a cost calculator I built for exactly this, and a modeled figure rather than anyone's real invoice.&lt;/p&gt;

&lt;p&gt;Divide one into the other: $65 million is roughly 9,700 years of that model.&lt;/p&gt;

&lt;p&gt;The gap between $6,720 and $65,000,000 is not a pricing scandal. It is what four orders of magnitude of fleet growth, log verbosity, and cardinality look like when no individual decision was ever expensive enough to argue about.&lt;/p&gt;

&lt;h2&gt;
  
  
  How 2021 produced that number
&lt;/h2&gt;

&lt;p&gt;Coinbase went public in 2021 and had a year most companies never see. Infrastructure cost was not a constraint anyone enforced, because revenue covered everything. AWS, Snowflake, and Datadog all grew. A bill that size is what three meters running at startup speed produce when nobody is assigned to watch them.&lt;/p&gt;

&lt;p&gt;Then the market turned. Revenue fell hard through 2022, and the bills that growth had hidden became the agenda. Datadog's CEO described the pattern on the same call: a customer whose business got cut to a third or a quarter gets its contract restructured, because the vendor would rather be part of the solution than part of the problem. The customer stayed, on chunked billing matched to a smaller business.&lt;/p&gt;

&lt;p&gt;One thing to be careful about: an anonymous commenter claimed the $65 million covered a multi-year prepayment. Orosz could not verify that and neither can anyone else, so it stays out. What is documented is the single-year bill, the settlement quarter, and the renegotiation.&lt;/p&gt;

&lt;h2&gt;
  
  
  The build-vs-buy line, stated properly
&lt;/h2&gt;

&lt;p&gt;Coinbase's response was to price the alternative: a dedicated observability team with a mandate to move onto Grafana, Prometheus, and ClickHouse, built for control and ownership first, with cost as the second motive once the market turned. They double-wrote the new stack for months before cutting anything over.&lt;/p&gt;

&lt;p&gt;The arithmetic that mattered is a single inequality:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;infrastructure cost + platform team cost &amp;lt; vendor bill
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Orosz's version: ten senior and staff engineers in the Bay Area cost under $5 million a year, with infrastructure on top in the low double-digit millions. Against $65 million, building is not a close call. Against a well-managed $2 million bill, it is a way to hire a team whose job is maintaining dashboards.&lt;/p&gt;

&lt;p&gt;His rule of thumb for where the question becomes real is $2–5 million a year in vendor spend — roughly where a team of four or five engineers plus a manager, at $150–400K fully loaded each, starts to pencil against the invoice.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The breakeven moves with the invoice.&lt;/strong&gt; Which is why the invoice has to be measured before the decision is made, not after someone sees a headline and asks for a migration plan.&lt;/p&gt;

&lt;h2&gt;
  
  
  They didn't leave
&lt;/h2&gt;

&lt;p&gt;The part the invoice-watchers usually miss: with the replacement stack validated and the plug ready to pull, Coinbase got terms it could accept and stayed. The customer is still on Datadog.&lt;/p&gt;

&lt;p&gt;That is not a contradiction. Deep integrations — custom tracing, per-service instrumentation, dashboards wired into every team's workflow — make migration expensive in a way the license math never captures. That lock-in cut both ways: it gave Coinbase leverage for a steep discount, and it gave Datadog a retained logo.&lt;/p&gt;

&lt;p&gt;The lesson is uncomfortable for both sides of the table. Price your exit before you threaten it, or the threat is theater.&lt;/p&gt;

&lt;h2&gt;
  
  
  Four things to do on Monday
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Budget tag cardinality like headcount.&lt;/strong&gt; Every new tag value on a custom metric mints series, and series beyond the per-host allotment bill by usage. Audit which teams emit the most series per host before approving more hosts.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Tier retention by access pattern.&lt;/strong&gt; Logs you investigate regularly earn standard indexing with alerting. Everything else belongs in a cheaper tier or an archive with rehydration on demand. Retention is the quietest multiplier on the invoice.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Decide sampling as a cost input.&lt;/strong&gt; Full-fidelity tracing on every request is a second invoice hiding inside the first. Pick the sampling rate when you design the tracing, not when you are debugging.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Alert on ingest slope, not the total.&lt;/strong&gt; Gigabytes per day times price per gigabyte, tracked against the monthly commit, turns the bill into a metric the on-call actually owns. A meter nobody watches is a meter without a limit.&lt;/p&gt;

&lt;p&gt;And the cheapest one of all: assign the bill to the teams that generate it. Chargeback turns "observability is expensive" into named owners with named levers. Coinbase assigned a whole team to the problem. Most companies only need to assign the invoice.&lt;/p&gt;

&lt;h2&gt;
  
  
  What this doesn't establish
&lt;/h2&gt;

&lt;p&gt;The identification rests on Orosz's reporting, not on a Coinbase or Datadog statement. Neither company has confirmed the customer. The $65 million covers one year of spend, not a multi-year contract that anyone has verified.&lt;/p&gt;

&lt;p&gt;Coinbase's current spend is undisclosed, so there is no way to check how much the renegotiation actually saved, or whether the in-house stack would have been cheaper over five years.&lt;/p&gt;

&lt;p&gt;The list prices are Datadog's published rates as of October 2026 and will not match any negotiated contract, including the one in this story. And no vendor published utilisation figures, so the three-meter model here describes the structure of the bill, not its actual composition.&lt;/p&gt;

&lt;p&gt;Three clocks ran in this story: the meters compounded through 2021, the invoice came due in Q1 2022, and the renegotiation reset the economics after that. Treating any one of them as "the Datadog incident" misses how metered billing actually fails — slowly, across independent units, and only discovered when growth stops covering it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Sources
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;p&gt;&lt;a href="https://blog.pragmaticengineer.com/datadog-65m-year-customer-mystery/" rel="noopener noreferrer"&gt;The Pragmatic Engineer: Datadog's $65M/year customer mystery solved&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;a href="https://thenewstack.io/datadogs-65m-bill-and-why-developers-should-care/" rel="noopener noreferrer"&gt;The New Stack: Datadog's $65M bill and why developers should care&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;a href="https://www.datadoghq.com/pricing/" rel="noopener noreferrer"&gt;Datadog pricing&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>devops</category>
      <category>sre</category>
      <category>cloud</category>
      <category>observability</category>
    </item>
    <item>
      <title>A cancelled query can destroy your database connection</title>
      <dc:creator>BuildOpsy</dc:creator>
      <pubDate>Thu, 29 Oct 2026 03:37:59 +0000</pubDate>
      <link>https://dev.to/buildopsy/a-cancelled-query-can-destroy-your-database-connection-6ea</link>
      <guid>https://dev.to/buildopsy/a-cancelled-query-can-destroy-your-database-connection-6ea</guid>
      <description>&lt;p&gt;Two short deadlines took down Inngest's database path twice in one afternoon, and neither was misconfigured. They expired exactly on time. That is what made the failure so effective.&lt;/p&gt;

&lt;p&gt;On 10 October 2026, both of Inngest's connection poolers lost every connection to their primary database at 14:00 UTC. They lost them again at 15:30. Run scheduling and queue processing stalled for most accounts in both windows, 14:00–14:48 and 15:30–16:25, and the mitigation that followed caused intermittent delays until 04:17 the next morning.&lt;/p&gt;

&lt;p&gt;The report is worth reading because the mechanism generalises to any stack that puts a connection pooler with a wait queue between the application and Postgres.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1n3zyv7ykgezzfm07veu.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1n3zyv7ykgezzfm07veu.png" alt="Component schematic: a client row with an expired deadline, a PgBouncer waiting queue, a gate admitting one new connection at a time, and a rack of connection slots marked closed rather than returned to the pool" width="800" height="420"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The sequence, in four steps
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;A frequently used lookup carried a short deadline.&lt;/li&gt;
&lt;li&gt;The deadline expired while its query was still queued inside PgBouncer, waiting for a database connection. The pgx driver cancelled the query and closed its client connection.&lt;/li&gt;
&lt;li&gt;PgBouncer had nothing running to cancel, because the query had never been handed a server connection. It later ran the abandoned query on a database connection.&lt;/li&gt;
&lt;li&gt;It then closed that connection entirely instead of returning it to the pool.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Step 4 is the whole failure. The client's timeout cost the system a database connection, not a request, and the cost landed after the client had stopped watching. No single request looked expensive. Individually, every one of these is a wasted query. Collectively they are a pool that empties itself.&lt;/p&gt;

&lt;p&gt;Two amplifiers arrived at the same moment. Both collapses began at the top and bottom of the hour, when cron-scheduled work roughly triples the query rate, and a newly rolled out set of lookups was failing in large numbers, adding cancellations to a pool that was already busy. Inngest reproduced the behaviour in a test environment.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why restarting and resizing both failed
&lt;/h2&gt;

&lt;p&gt;Restarting the poolers did not reliably recover them. Raising the pool size did not help either. The report is explicit on why: opening connections was the bottleneck, not the pool limit.&lt;/p&gt;

&lt;p&gt;Each pooler opens new database connections one at a time. So the refill path is serial while the destruction path is parallel. As a steady-state sketch rather than a model of this incident, if usable connections are destroyed at &lt;code&gt;d&lt;/code&gt; per second and a pooler needs &lt;code&gt;t&lt;/code&gt; seconds on average to establish a replacement, the refill ceiling is &lt;code&gt;1/t&lt;/code&gt; and the pool drains whenever:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;d &amp;gt; 1/t
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The sketch assumes constant destruction and handshake times, and it ignores the point at which the pooler stops trying. Inngest published neither number, so treat this as the shape of the constraint rather than a measurement of it. To see the shape: at 100 ms per connection a single pooler refills at ten per second; at 50 ms, twenty.&lt;/p&gt;

&lt;p&gt;The second collapse ended by changing paths, not by growing the pool. The team moved all services onto poolers run alongside the database by the database provider, and processing recovered between 16:10 and 16:25.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fix that caused a second incident
&lt;/h2&gt;

&lt;p&gt;The new route reached the database through NAT gateways, and some connections through those gateways were silently dropped. Queries on a dropped connection hung for up to 16 minutes before failing, which stalled processing intermittently and most visibly hit cron functions. Customers reported delays at 03:01 on 11 October; the affected service moved back to the original poolers at 04:17, and gateway routing was improved on 12 October.&lt;/p&gt;

&lt;p&gt;Sixteen minutes of silent hang is worse than a fast failure for a scheduler. A connection that errors invites a retry. One that simply stops answering consumes a worker slot and a deadline, and you learn about it from a customer. A failover drill that measures successful requests will not catch this. One that measures how long a hung connection stays hung will.&lt;/p&gt;

&lt;h2&gt;
  
  
  The contract that broke underneath it
&lt;/h2&gt;

&lt;p&gt;The part of the report with the longest reach is not about pools. Internal failures counted against functions' configured retries, and a step that failed permanently on an internal error left its run in &lt;code&gt;Running&lt;/code&gt; instead of marking it failed. &lt;code&gt;onFailure&lt;/code&gt; handlers did not fire. Functions waiting on that run through &lt;code&gt;step.invoke&lt;/code&gt; were not notified.&lt;/p&gt;

&lt;p&gt;That is a broken guarantee, not an availability number. A durable execution engine sells the promise that every run reaches a terminal state and that failure handlers run. Under this bug the engine could hold a run in &lt;code&gt;Running&lt;/code&gt; indefinitely, after spending the retry budget it was given — the customer's allowance consumed by the platform's own faults, and the transition that would have reported it never sent.&lt;/p&gt;

&lt;p&gt;The recovery instructions confirm the state of those runs: they will not continue on their own, and customers can cancel and replay them, with the caveat that a replay starts from the beginning, so completed steps run again. Anyone with non-idempotent side effects in a completed step had a decision the platform could not make for them.&lt;/p&gt;

&lt;h2&gt;
  
  
  Four things to check in your own stack
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Alert on net connection drain, not pool utilisation.&lt;/strong&gt; Graph opens and closes per pooler per second, and alert when the close rate exceeds the open rate for a sustained interval. A pool at 30% with a negative slope is in more danger than one at 70% that is stable, and utilisation alone shows neither.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Bound the wait, not just the query.&lt;/strong&gt; PgBouncer's &lt;code&gt;query_wait_timeout&lt;/code&gt; disconnects a client whose query has not been assigned a server within the limit.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="c"&gt;; pgbouncer.ini — the ordering matters more than the value
&lt;/span&gt;&lt;span class="py"&gt;query_wait_timeout&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;5    ; must sit below your client's own deadline&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The point of setting it below the client deadline is to remove the queued entry before the client gives up. That is the reasoning, not proof that it preserves backend connections. Inngest describes the pooler running the abandoned query and then closing that connection; an independent load test from July 2025 puts the waste a step earlier, with the pooler finishing a login for a client that has already gone and discarding the fresh backend connection. The same engineer reported that lowering the timeout helped without being enough on its own. Test it against your driver's real cancellation path before adopting it.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Know what your driver does on cancel.&lt;/strong&gt; In pgx v5, the thread reports replacing the default cancellation behaviour with a context watcher handler that sets a connection deadline instead of closing immediately, and describes the effect in their load test as large. If your code path cancels queued queries, this is the boundary to inspect.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Fence the lease before you force a terminal state.&lt;/strong&gt; If you mark a run failed because its step lease expired without a heartbeat, late writes from a worker that is still running can land afterwards, and a replay produces a second copy of the side effects. Mark it failed only when the engine can reject writes from the old lease, or when the step is confirmed not to be in flight.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the report does not establish
&lt;/h2&gt;

&lt;p&gt;Inngest disclosed no connection counts, handshake timings, pool sizes or customer totals, so the arithmetic here is the report's timestamps and nothing more. The 44-minute, 48-minute and 55-minute windows are my subtraction on its UTC boundaries, not figures it published.&lt;/p&gt;

&lt;p&gt;The report also links pgx issue 1780 as the known interaction behind the collapse. I read that issue: it was opened on 26 October 2023 and was still open at its latest comment in February 2026. The upstream record offers a compatible description of the boundary and no confirmed root cause, so Inngest's account is a well-documented instance of an unresolved interaction rather than evidence that it is fixed.&lt;/p&gt;

&lt;p&gt;The structural change is the one to watch. Inngest is consolidating onto a single data centre acting as one cell, which removes the cross-data-centre failure class behind the 9 October incident. That trades one failure shape for another: fewer paths that can break, and more consequence when one does.&lt;/p&gt;

&lt;h2&gt;
  
  
  Sources
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://www.inngest.com/blog/2026-10-15-incident-report-for-october-9-and-10-2026" rel="noopener noreferrer"&gt;Inngest: Incident report for October 9 and 10, 2026&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/jackc/pgx/issues/1780" rel="noopener noreferrer"&gt;jackc/pgx issue 1780&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Thanks for reading :)&lt;/p&gt;

</description>
      <category>database</category>
      <category>postgres</category>
      <category>sre</category>
      <category>architecture</category>
    </item>
    <item>
      <title>OpenRouter's February 2026 Outages: How a Cold Cache Overwhelmed the Database</title>
      <dc:creator>BuildOpsy</dc:creator>
      <pubDate>Sun, 25 Oct 2026 11:38:07 +0000</pubDate>
      <link>https://dev.to/buildopsy/openrouters-february-2026-outages-how-a-cold-cache-overwhelmed-the-database-3in5</link>
      <guid>https://dev.to/buildopsy/openrouters-february-2026-outages-how-a-cold-cache-overwhelmed-the-database-3in5</guid>
      <description>&lt;p&gt;The database was struggling. The API said the user didn't exist.&lt;/p&gt;

&lt;p&gt;That is the part of OpenRouter's February outages worth looking at closely.&lt;/p&gt;

&lt;p&gt;The reported failure involved cache connectivity and the database lookups behind API-key authentication. When the cache stopped serving those lookups, the database had to absorb work it normally wouldn't see. Requests started failing with 401 User not found.&lt;/p&gt;

&lt;p&gt;There is an important distinction here: an authentication failure and an authentication infrastructure failure are not the same thing. If a database timeout becomes a 401, a client may treat a temporary service failure as an invalid credential. Retries, alerts and incident diagnosis can all head in the wrong direction.&lt;/p&gt;

&lt;p&gt;The capacity problem is just as interesting. A database can appear comfortably provisioned because a cache absorbs most of the read traffic. That doesn't mean the database can survive the workload when the cache disappears. Recovery can create its own traffic spike, precisely when the dependency underneath is least able to handle it.&lt;/p&gt;

&lt;p&gt;This is why cache-failure testing shouldn't stop at checking whether the application stays online. It should also examine database concurrency, fallback request rates, cache repopulation, retry behavior and the errors returned to clients. A fallback path that works for a few misses may behave very differently when every request takes it at once.&lt;/p&gt;

&lt;p&gt;There is also a subtle observability problem here. The first outage coincided with a denial-of-service attack, which initially complicated the investigation. When the caching failure recurred in February without the same accompanying attack, the underlying cause became clearer. Multiple problems can overlap; temporal correlation alone doesn't establish root cause.&lt;/p&gt;

&lt;p&gt;The remediation needs to address both load and semantics. Circuit breakers and controlled cache repopulation can limit the burst reaching the database. Returning 503 Service Unavailable rather than 401 Unauthorized when an infrastructure failure prevents an authentication lookup helps clients distinguish an unavailable service from invalid credentials.&lt;/p&gt;

&lt;p&gt;A useful test for any cache-backed authentication system: invalidate the cache under realistic concurrency and measure the database's behavior during recovery. How quickly does load rise? Are concurrent misses coalesced? Do retries amplify the spike? Can the service shed work before the database saturates?&lt;/p&gt;

&lt;p&gt;Those answers reveal more about the system's failure tolerance than its cache-hit ratio during normal operation.&lt;/p&gt;

&lt;p&gt;The full postmortem reconstructs both outages, separates published facts from illustrative capacity calculations, and identifies what the available evidence does—and doesn't—establish about the remediation.&lt;/p&gt;

&lt;p&gt;Read the full teardown here: &lt;a href="https://buildopsy.com/system-design/openrouter-cache-thundering-herd-postmortem.html" rel="noopener noreferrer"&gt;https://buildopsy.com/system-design/openrouter-cache-thundering-herd-postmortem.html&lt;/a&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>programming</category>
      <category>productivity</category>
      <category>architecture</category>
    </item>
    <item>
      <title>🚀Rocket thrusters on a car</title>
      <dc:creator>BuildOpsy</dc:creator>
      <pubDate>Sun, 18 Oct 2026 10:05:09 +0000</pubDate>
      <link>https://dev.to/buildopsy/rocket-thrusters-on-a-car-57e6</link>
      <guid>https://dev.to/buildopsy/rocket-thrusters-on-a-car-57e6</guid>
      <description>&lt;p&gt;Tesla promised rocket thrusters on a car. Naturally, the next step is to ask the engineers what the physics has to say about that.&lt;/p&gt;

&lt;p&gt;Marketing has a wonderful relationship with the future. Engineering has a slightly less romantic relationship with mass, thrust, pressure, and the laws of physics.&lt;/p&gt;

&lt;p&gt;So we audited the Tesla Roadster’s thruster claims as an engineering problem, not a headline-writing competition.&lt;/p&gt;

&lt;p&gt;What’s theoretically possible? What has actually been demonstrated? What does the available evidence support, and where does the gap between a claim and a verified capability begin?&lt;/p&gt;

&lt;p&gt;Because there’s a rather important difference between a technology that &lt;em&gt;could work in principle&lt;/em&gt; and one that has been demonstrated to work as advertised.&lt;/p&gt;

&lt;p&gt;We went through the claims, examined the evidence, and separated the physics from the promises.&lt;/p&gt;

&lt;p&gt;Turns out, “the laws of physics don’t forbid it” and “the product can actually do it” are two very different engineering milestones.&lt;/p&gt;

&lt;p&gt;A useful reminder that even when your company builds rockets, the burden of proof doesn’t magically disappear when you put the word &lt;em&gt;Tesla&lt;/em&gt; on the slide.&lt;/p&gt;

&lt;p&gt;Full engineering audit: &lt;a href="https://buildopsy.com/system-design/tesla-roadster-thruster-claims-audit.html" rel="noopener noreferrer"&gt;https://buildopsy.com/system-design/tesla-roadster-thruster-claims-audit.html&lt;/a&gt;&lt;/p&gt;

</description>
      <category>distributedsystems</category>
      <category>webdev</category>
      <category>programming</category>
      <category>software</category>
    </item>
    <item>
      <title>When Go GC Burns 60% CPU: The Container Memory Limit Problem</title>
      <dc:creator>BuildOpsy</dc:creator>
      <pubDate>Thu, 15 Oct 2026 18:27:31 +0000</pubDate>
      <link>https://dev.to/buildopsy/when-go-gc-burns-60-cpu-the-container-memory-limit-problem-5g2d</link>
      <guid>https://dev.to/buildopsy/when-go-gc-burns-60-cpu-the-container-memory-limit-problem-5g2d</guid>
      <description>&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fnb9n11d0g7ox8cn256yl.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fnb9n11d0g7ox8cn256yl.png" alt="A technical illustration of Go garbage collection pacing against container memory limits" width="800" height="420"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Banner: Go garbage collection, memory pacing, and the container limits that can make a difference.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;A Go service can burn CPU collecting garbage without having a memory leak.&lt;/p&gt;

&lt;p&gt;That is the counterintuitive part of Zomato's Go garbage collection story.&lt;/p&gt;

&lt;p&gt;The reported symptoms were striking: garbage collection consuming 25–60% of CPU in some services, alongside containers getting OOM-killed without an obvious memory leak.&lt;/p&gt;

&lt;p&gt;These look like separate problems. But they can share a common cause: &lt;strong&gt;the garbage collector's pacing decisions don't automatically account for the container's actual memory budget.&lt;/strong&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The problem with doubling the live heap
&lt;/h2&gt;

&lt;p&gt;Go's garbage collector uses &lt;code&gt;GOGC&lt;/code&gt; to control when the next collection should happen.&lt;/p&gt;

&lt;p&gt;With the default &lt;code&gt;GOGC=100&lt;/code&gt;, the GC target is roughly based on the live heap growing by 100% relative to the previous cycle, with additional accounting for GC roots and other implementation details.&lt;/p&gt;

&lt;p&gt;In simplified terms:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Next GC target ≈ live heap × (1 + GOGC/100)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;At &lt;code&gt;GOGC=100&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Live heap:       500 MiB
Approx. target: 1,000 MiB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That target is reasonable as a starting point when the process has enough memory available.&lt;/p&gt;

&lt;p&gt;But containers introduce another constraint: &lt;strong&gt;the process has a memory budget imposed from outside the Go runtime.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The GC's heap-growth target and the container's memory ceiling are two different things.&lt;/p&gt;

&lt;p&gt;When they aren't coordinated, two failure modes become possible.&lt;/p&gt;

&lt;h2&gt;
  
  
  Failure mode 1: OOM kills without a leak
&lt;/h2&gt;

&lt;p&gt;Suppose a container has a memory limit of 1 GiB.&lt;/p&gt;

&lt;p&gt;The Go process has a live heap of 600 MiB. A simplified &lt;code&gt;GOGC=100&lt;/code&gt; calculation puts the next target around 1,200 MiB.&lt;/p&gt;

&lt;p&gt;But the container cannot safely accommodate that much memory. The process also needs room for goroutine stacks, runtime metadata, native allocations, and other memory outside the live heap.&lt;/p&gt;

&lt;p&gt;The container can hit its limit before the collector's normal heap target becomes appropriate.&lt;/p&gt;

&lt;p&gt;The result can be an OOM kill even when reachable memory isn't growing uncontrollably.&lt;/p&gt;

&lt;p&gt;That doesn't mean every OOM without an obvious leak is a GC pacing issue. Transient allocation spikes, native memory, and other causes still need investigation.&lt;/p&gt;

&lt;p&gt;The important point is that a heap-growth target alone isn't a container memory budget.&lt;/p&gt;

&lt;h2&gt;
  
  
  Failure mode 2: Excessive GC CPU
&lt;/h2&gt;

&lt;p&gt;Now consider the opposite situation.&lt;/p&gt;

&lt;p&gt;A service has a relatively small live heap and plenty of memory available in its container.&lt;/p&gt;

&lt;p&gt;The GC keeps using a target derived from that live heap. Depending on the allocation rate and workload, it may collect frequently even though the container has significant unused headroom.&lt;/p&gt;

&lt;p&gt;Every collection consumes CPU.&lt;/p&gt;

&lt;p&gt;At fleet scale, unnecessary GC work can become a meaningful compute cost.&lt;/p&gt;

&lt;p&gt;This is why the following combination deserves attention:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;High GC CPU.&lt;/li&gt;
&lt;li&gt;A relatively flat live-heap graph.&lt;/li&gt;
&lt;li&gt;No convincing evidence of sustained heap growth.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;It is a diagnostic clue, not proof. A service can allocate and discard large volumes of temporary objects while maintaining a flat live heap. Allocation rate, GC frequency, and allocation profiles matter too.&lt;/p&gt;

&lt;p&gt;But before treating every high-GC-CPU incident as a memory leak, inspect the collector's pacing and the container's actual memory constraints.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fix: Give Go a realistic memory limit
&lt;/h2&gt;

&lt;p&gt;Go provides &lt;code&gt;GOMEMLIMIT&lt;/code&gt; to let applications communicate a soft memory limit to the runtime.&lt;/p&gt;

&lt;p&gt;Unlike &lt;code&gt;GOGC&lt;/code&gt;, which controls GC aggressiveness in relation to heap growth, &lt;code&gt;GOMEMLIMIT&lt;/code&gt; helps the runtime account for its overall memory footprint when deciding how aggressively to collect.&lt;/p&gt;

&lt;p&gt;Conceptually, the runtime must balance two competing costs:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Collect too late, and the process risks exhausting its container memory budget.&lt;/li&gt;
&lt;li&gt;Collect too aggressively, and the application wastes CPU reclaiming memory it could have used safely.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A memory limit gives the runtime another constraint to consider.&lt;/p&gt;

&lt;p&gt;In the Zomato case, the reported approach used container metadata to derive a per-container limit, refreshed it periodically, and combined memory tuning with CPU scheduling adjustments.&lt;/p&gt;

&lt;p&gt;A limit around 90% of the container ceiling can be a starting point for investigation, &lt;strong&gt;not a universal safe value&lt;/strong&gt;. The correct margin depends on the application's non-heap memory, traffic spikes, allocation behavior, and operational requirements.&lt;/p&gt;

&lt;p&gt;The goal is to give the runtime a realistic budget while preserving enough headroom for the rest of the process.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why &lt;code&gt;GOMEMLIMIT&lt;/code&gt; isn't a hard wall
&lt;/h2&gt;

&lt;p&gt;This distinction matters in production.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;GOMEMLIMIT&lt;/code&gt; is a soft limit, not a strict process-memory cap. The runtime can exceed it when necessary, and it does not guarantee that a container will avoid OOM kills.&lt;/p&gt;

&lt;p&gt;The runtime also has to balance memory pressure against GC CPU overhead.&lt;/p&gt;

&lt;p&gt;If the limit is too high, it may offer little protection against container memory pressure. If it is too low, the runtime can spend excessive CPU collecting garbage in an attempt to stay within the target.&lt;/p&gt;

&lt;p&gt;That is why blindly setting &lt;code&gt;GOMEMLIMIT&lt;/code&gt; equal to the container limit is not a complete solution.&lt;/p&gt;

&lt;p&gt;A production rollout should account for:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;The container's actual memory limit.&lt;/li&gt;
&lt;li&gt;Heap and non-heap memory usage.&lt;/li&gt;
&lt;li&gt;Peak allocation rates and traffic bursts.&lt;/li&gt;
&lt;li&gt;GC CPU and pause behavior.&lt;/li&gt;
&lt;li&gt;OOM events and memory headroom after deployment.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Start with representative workloads, observe the results, and tune based on measurements rather than assuming a single multiplier works everywhere.&lt;/p&gt;

&lt;h2&gt;
  
  
  Don't overlook `
&lt;/h2&gt;

</description>
      <category>go</category>
      <category>performance</category>
      <category>systemdesign</category>
      <category>sre</category>
    </item>
    <item>
      <title>CircleCI's Runners Were Ready. PostgreSQL Couldn't Dispatch Jobs.</title>
      <dc:creator>BuildOpsy</dc:creator>
      <pubDate>Thu, 15 Oct 2026 16:20:37 +0000</pubDate>
      <link>https://dev.to/buildopsy/the-database-was-the-healthiest-part-of-circlecis-fleet-it-still-took-down-3500-jobs-2fi5</link>
      <guid>https://dev.to/buildopsy/the-database-was-the-healthiest-part-of-circlecis-fleet-it-still-took-down-3500-jobs-2fi5</guid>
      <description>&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fbs0n0nvqu1xkij8hruc0.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fbs0n0nvqu1xkij8hruc0.png" alt="CircleCI outage architecture: PostgreSQL WAL saturation blocked job dispatch" width="800" height="420"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Architecture diagram by BuildOpsy: How PostgreSQL WAL saturation blocked CircleCI's orchestration database, leaving healthy runners idle and thousands of job starts affected.&lt;/em&gt;&lt;/p&gt;

&lt;h1&gt;
  
  
  CircleCI's Runners Were Fine. PostgreSQL Wasn't.
&lt;/h1&gt;

&lt;p&gt;On October 6, 2026, CircleCI hit a failure that had nothing to do with runner capacity.&lt;/p&gt;

&lt;p&gt;The culprit was routine PostgreSQL maintenance.&lt;/p&gt;

&lt;p&gt;An anti-wraparound autovacuum ran against large tables, generating write-ahead log (WAL) faster than the database could archive it. The WAL volume filled, and the orchestration database went offline for 50 minutes.&lt;/p&gt;

&lt;p&gt;The numbers tell the story:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;~3,500 job starts failed&lt;/strong&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;50 minutes of database unavailability&lt;/strong&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Hours of recovery work after the database returned&lt;/strong&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Here's the underlying systems problem.&lt;/p&gt;

&lt;p&gt;Let &lt;code&gt;g&lt;/code&gt; be the WAL generation rate and &lt;code&gt;a&lt;/code&gt; the archival rate.&lt;/p&gt;

&lt;p&gt;When &lt;code&gt;g &amp;gt; a&lt;/code&gt;, unarchived WAL accumulates. Free disk space only determines how long the system can tolerate that imbalance.&lt;/p&gt;

&lt;p&gt;A rough estimate of the remaining time is:&lt;/p&gt;

&lt;p&gt;&lt;code&gt;Time to saturation ≈ Free WAL space / (g − a)&lt;/code&gt;&lt;/p&gt;

&lt;p&gt;This assumes relatively stable rates, but the principle is important: a disk-full alert often arrives too late. By the time it fires, the system may already be out of runway.&lt;/p&gt;

&lt;p&gt;The architectural impact was bigger than a database outage.&lt;/p&gt;

&lt;p&gt;CircleCI's orchestration database sat on the critical path for workflow state, job dispatch, UI updates, and notifications. The runners could be healthy and ready to execute, but without the control plane coordinating work, they couldn't help.&lt;/p&gt;

&lt;p&gt;Then came the second problem: recovery.&lt;/p&gt;

&lt;p&gt;Restoring database availability didn't instantly clear the backlog. Queued jobs still needed to be processed and dispatched, putting additional pressure on the system as it recovered.&lt;/p&gt;

&lt;p&gt;This is the part of incident response that's easy to underestimate: &lt;strong&gt;restoring service and restoring normal operation are two different problems.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The safeguards I'd take from this incident:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Monitor WAL generation versus archival throughput.&lt;/strong&gt; Alert on the rate gap, archive lag, and estimated time to saturation, not just disk utilization.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Test autovacuum under realistic workloads.&lt;/strong&gt; Large tables and production-level write activity can expose bottlenecks that smaller test environments miss.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Protect the recovery path.&lt;/strong&gt; Make sure WAL archival and database recovery have enough capacity and clear operational guardrails.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Control backlog redrain.&lt;/strong&gt; Use bounded concurrency, retry limits, and staggered dispatch to avoid overwhelming downstream services.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Design for control-plane failures.&lt;/strong&gt; A healthy execution fleet is useless if the system responsible for scheduling work is unavailable.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The deeper lesson is that a healthy execution fleet doesn't guarantee a healthy CI/CD system. The control plane that decides what runs can be the real bottleneck.&lt;/p&gt;

&lt;p&gt;I broke down the failure mechanism, incident timeline, recovery bottleneck, and architectural trade-offs in the full postmortem:&lt;/p&gt;

&lt;p&gt;👉 &lt;strong&gt;&lt;a href="https://buildopsy.com/system-design/circleci-wal-autovacuum-postmortem.html" rel="noopener noreferrer"&gt;CircleCI Outage Postmortem: Full WAL Saturation Blocked 3,500 Job Starts&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Question for PostgreSQL engineers:&lt;/strong&gt; Do you monitor WAL generation rate against archival throughput and estimate time-to-full, or are your alerts still mostly based on disk utilization?&lt;/p&gt;

&lt;p&gt;I'd be interested to hear what works in production. Thanks&lt;/p&gt;

</description>
      <category>postgres</category>
      <category>devops</category>
      <category>sre</category>
      <category>software</category>
    </item>
    <item>
      <title>A small search win: Google's AI Overview cited my Figma multiplayer teardown</title>
      <dc:creator>BuildOpsy</dc:creator>
      <pubDate>Tue, 06 Oct 2026 18:49:32 +0000</pubDate>
      <link>https://dev.to/buildopsy/a-small-search-win-googles-ai-overview-cited-my-figma-multiplayer-teardown-1639</link>
      <guid>https://dev.to/buildopsy/a-small-search-win-googles-ai-overview-cited-my-figma-multiplayer-teardown-1639</guid>
      <description>&lt;p&gt;I write about production system design on Buildopsy. Recently, I published a teardown of Figma’s multiplayer architecture: how edits move between browsers and servers, how conflicts get resolved, and where the system’s scaling costs come from.&lt;/p&gt;

&lt;p&gt;I later searched for “how Figma’s multiplayer technology works.” Figma’s own engineering post appeared in the results. In the AI Overview below it, I noticed two Buildopsy citations beside explanations of optimistic rendering and the server’s role in ordering edits.&lt;/p&gt;

&lt;h2&gt;
  
  
  What appeared in the result
&lt;/h2&gt;

&lt;p&gt;I saved a screenshot because I didn’t expect to see my article cited there. It’s one result for one query—not proof of a lasting ranking or a traffic increase.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fsdcmlvput9gqt4t2h8if.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fsdcmlvput9gqt4t2h8if.png" alt="Google AI Overview citing Buildopsy for Figma multiplayer details" width="800" height="1423"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The interesting part is how edits get reconciled
&lt;/h2&gt;

&lt;p&gt;A browser applies an edit immediately, then sends the operation over a WebSocket to the process that owns that document. The owner orders the change and broadcasts the accepted version to the other clients.&lt;/p&gt;

&lt;p&gt;The conflict rule is more specific than “the server wins.” In the architecture described in my teardown, the server orders operations, same-property conflicts use last-writer-wins, and tree changes get an additional validation step to prevent cycles. That’s a product-specific trade-off: a design canvas can tolerate some conflicts that would be painful in a collaborative text editor.&lt;/p&gt;

&lt;p&gt;The Rust rewrite addressed another boundary. A slow document could stall unrelated documents sharing a TypeScript worker. Figma moved document operations into a Rust child process while Node.js kept handling the network layer. Figma reported that serialization became more than 10× faster in its &lt;a href="https://www.figma.com/blog/rust-in-production-at-figma/" rel="noopener noreferrer"&gt;engineering write-up&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;The numbers make the next question more interesting. In one scenario, 1.5 million concurrent users sending two coalesced cursor frames per second produce &lt;strong&gt;3 million ephemeral frames per second&lt;/strong&gt;. That’s modeled math, not a Figma disclosure. The design question is: which events need durable ordering, and which can be dropped when they’re stale?&lt;/p&gt;

&lt;p&gt;The full teardown works through that boundary, a 65k-RPS scenario, and a modeled monthly cost range. The cost figures are my assumptions, not Figma’s reported spend: &lt;a href="https://buildopsy.com/system-design/figma-system-design.html" rel="noopener noreferrer"&gt;Read the Figma multiplayer teardown&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;I don’t know whether the citation will persist or bring readers. It was still encouraging to see a technical explanation appear alongside the primary source it discusses.&lt;/p&gt;

</description>
      <category>figma</category>
      <category>systemdesign</category>
      <category>architecture</category>
      <category>webdev</category>
    </item>
  </channel>
</rss>
