<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Jatin Jain Saraf</title>
    <description>The latest articles on DEV Community by Jatin Jain Saraf (@jatinjainsaraf).</description>
    <link>https://dev.to/jatinjainsaraf</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3979918%2F540012a9-fdb8-46fc-b44f-9e33bf09c240.jpg</url>
      <title>DEV Community: Jatin Jain Saraf</title>
      <link>https://dev.to/jatinjainsaraf</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/jatinjainsaraf"/>
    <language>en</language>
    <item>
      <title>Your Node.js App Never Imports zlib. It Might Still Be Vulnerable to a Brotli RCE</title>
      <dc:creator>Jatin Jain Saraf</dc:creator>
      <pubDate>Thu, 22 Oct 2026 17:39:28 +0000</pubDate>
      <link>https://dev.to/jatinjainsaraf/your-nodejs-app-never-imports-zlib-it-might-still-be-vulnerable-to-a-brotli-rce-2da9</link>
      <guid>https://dev.to/jatinjainsaraf/your-nodejs-app-never-imports-zlib-it-might-still-be-vulnerable-to-a-brotli-rce-2da9</guid>
      <description>&lt;h2&gt;
  
  
  A Node.js Patch You Shouldn't Skip
&lt;/h2&gt;

&lt;p&gt;On October 21, 2026, Node.js shipped security releases for all three supported lines. The headline fix is &lt;a href="https://nodejs.org/en/blog/vulnerability/october-2026-security-releases" rel="noopener noreferrer"&gt;CVE-2026-111497&lt;/a&gt;, a High-severity out-of-bounds write in Node's Brotli decoder that, according to the advisory, permits remote code execution.&lt;/p&gt;

&lt;p&gt;The natural first reaction is "we don't decompress anything, so this isn't ours." I had that reaction too. A search of my own site's application code found no &lt;code&gt;zlib&lt;/code&gt; usage at all. Then I traced what Node's built-in &lt;code&gt;fetch()&lt;/code&gt; does with a compressed response, and found the Brotli decoder sitting on a path my code uses every day.&lt;/p&gt;

&lt;p&gt;This post walks through that investigation: the source code, a small reproduction you can run yourself, why some features deserve more urgency than others, and the checklist I'd run on any Node service right now.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The facts, up front:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Component:&lt;/strong&gt; the Brotli decoder in Node.js (&lt;code&gt;zlib&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;CVE:&lt;/strong&gt; &lt;a href="https://nodejs.org/en/blog/vulnerability/october-2026-security-releases" rel="noopener noreferrer"&gt;CVE-2026-111497&lt;/a&gt;, rated High in the Node.js advisory&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Fixed in:&lt;/strong&gt; v22.23.4, v24.22.1, v26.12.1&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;What to do:&lt;/strong&gt; upgrade to the patched release on your line, rebuild, redeploy, and verify the version production actually runs&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;It landed in a loud news cycle that also included two PHP remote code execution vulnerabilities, a Railway security disclosure about customer environment variables, and the compromise of eight eMudhra intermediate certificate keys. The Node.js release received less attention, but JavaScript teams should treat it as urgent runtime maintenance.&lt;/p&gt;

&lt;h2&gt;
  
  
  What Shipped
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Line&lt;/th&gt;
&lt;th&gt;Fixed version&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;26.x&lt;/td&gt;
&lt;td&gt;v26.12.1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;24.x&lt;/td&gt;
&lt;td&gt;v24.22.1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;22.x&lt;/td&gt;
&lt;td&gt;v22.23.4&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The release covers 10 CVEs: 4 rated High, 3 Medium and 3 Low. All the High and Medium issues affect 24.x and 26.x, and all but one affect 22.x as well. It also updates bundled dependencies, including Undici (the library behind Node's built-in &lt;code&gt;fetch()&lt;/code&gt;), c-ares, OpenSSL on 22.x and 24.x, and the bundled CA certificates.&lt;/p&gt;

&lt;p&gt;Node 20 and Node 25 are end-of-life. The advisory is explicit that end-of-life versions are always affected when a security release ships, and they get no fixes. If anything you run is still on one of them, that's the first thing to change.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Brotli Bug, Without the Jargon
&lt;/h2&gt;

&lt;p&gt;Brotli is a compression format. Servers use it to shrink HTTP responses, and Node ships a native Brotli decoder (inside &lt;code&gt;zlib&lt;/code&gt;) to expand them again.&lt;/p&gt;

&lt;p&gt;A decompressor works like a moving crew unpacking boxes into a room. The compressed data says, in effect, "this box holds this much, put it here." The crew trusts the instructions and keeps unpacking. An out-of-bounds write is what happens when a carefully crafted box convinces the crew to keep unpacking past the wall of the room, straight into the neighbour's flat. In a process, the neighbour's flat is other memory: data structures, pointers, things the program relies on. Overwrite the right bytes and you can steer what the program does next. That's the road from "decompression bug" to "remote code execution."&lt;/p&gt;

&lt;p&gt;The advisory credits the OpenAI Codex Security team (Khai Tran and Sylvie Mayer) with reporting it, and Matteo Collina and Antoine du Hamel with the fix. (The first version of the advisory credited Matteo Collina as the reporter; the Node.js team &lt;a href="https://github.com/nodejs/nodejs.org/commit/d216a65d" rel="noopener noreferrer"&gt;updated the credits&lt;/a&gt; on October 22, so you may still see the older wording in cached copies.) The &lt;a href="https://nodejs.org/en/blog/release/v24.22.1" rel="noopener noreferrer"&gt;v24.22.1 release notes&lt;/a&gt; identify the fix as a cherry-pick from the upstream Brotli library, attributed to Evgenii Kliuchnikov. That identifies the patch as coming from the Brotli dependency Node bundles, rather than as a change to Node's own JavaScript. The advisory doesn't describe the exact input that triggers the bug, so I won't guess at exploit details. What matters for us is a simpler question: does untrusted Brotli data ever reach that decoder in my process?&lt;/p&gt;

&lt;h2&gt;
  
  
  Why "We Don't Use zlib" Isn't the Answer
&lt;/h2&gt;

&lt;p&gt;Here's the search I ran across my site's source tree (excluding the course content folder, whose lessons mention &lt;code&gt;zlib&lt;/code&gt; in prose):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;grep&lt;/span&gt; &lt;span class="nt"&gt;-rnE&lt;/span&gt; &lt;span class="s2"&gt;"Brotli|zlib"&lt;/span&gt; src &lt;span class="nt"&gt;--exclude-dir&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;content

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;No results. My application code never imports &lt;code&gt;zlib&lt;/code&gt; and never decompresses anything by hand. (That covers my code, not my dependencies, which is exactly the point of what follows.)&lt;/p&gt;

&lt;p&gt;But the site calls &lt;code&gt;fetch()&lt;/code&gt; constantly: the LinkedIn API, the YouTube API, the Instagram Graph API, Google's font CSS. And &lt;code&gt;fetch()&lt;/code&gt; decompresses responses for you. That's the whole point of it: you call &lt;code&gt;res.json()&lt;/code&gt; and get JSON, not compressed bytes.&lt;/p&gt;

&lt;p&gt;So I checked how. Node v24.22.1 bundles Undici 7.30.1. In &lt;a href="https://github.com/nodejs/undici/blob/v7.30.1/lib/web/fetch/index.js#L2256-L2260" rel="noopener noreferrer"&gt;that version's fetch implementation&lt;/a&gt;, the response's &lt;code&gt;Content-Encoding&lt;/code&gt; header picks the decoder, and for &lt;code&gt;br&lt;/code&gt; it creates a native Brotli decoder:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;coding&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;br&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;decoders&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;push&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;zlib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;createBrotliDecompress&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
    &lt;span class="na"&gt;flush&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;zlib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;constants&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;BROTLI_OPERATION_FLUSH&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="na"&gt;finishFlush&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;zlib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;constants&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;BROTLI_OPERATION_FLUSH&lt;/span&gt;
  &lt;span class="p"&gt;}))&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Reading source is one thing, so I also tested it. A tiny local server returns a Brotli-compressed body over plain HTTP, and Node's built-in &lt;code&gt;fetch()&lt;/code&gt; reads it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nx"&gt;http&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;node:http&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nx"&gt;zlib&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;node:zlib&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;body&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;zlib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;brotliCompressSync&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="nx"&gt;Buffer&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="k"&gt;from&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;{"hello":"decoded by Node&lt;/span&gt;&lt;span class="se"&gt;\'&lt;/span&gt;&lt;span class="s1"&gt;s Brotli decoder"}&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="kd"&gt;let&lt;/span&gt; &lt;span class="nx"&gt;seenAE&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;srv&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;http&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;createServer&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="nx"&gt;req&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;res&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="nx"&gt;seenAE&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nx"&gt;req&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;accept-encoding&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;];&lt;/span&gt;
  &lt;span class="nx"&gt;res&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;writeHead&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;200&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;content-type&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;application/json&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;content-encoding&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;br&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="p"&gt;});&lt;/span&gt;
  &lt;span class="nx"&gt;res&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;end&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;body&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="p"&gt;}).&lt;/span&gt;&lt;span class="nf"&gt;listen&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;async &lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;r&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;fetch&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;`http://127.0.0.1:&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;srv&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;address&lt;/span&gt;&lt;span class="p"&gt;().&lt;/span&gt;&lt;span class="nx"&gt;port&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;/`&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="nx"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;client sent accept-encoding:&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;seenAE&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="nx"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;json():&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;());&lt;/span&gt;
  &lt;span class="nx"&gt;srv&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;close&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
&lt;span class="p"&gt;});&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Output on Node v24.22.1 (bundled Undici 7.30.1):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;client sent accept-encoding: gzip, deflate
json(): { hello: "decoded by Node's Brotli decoder" }

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Look at the first line. Over plain HTTP, &lt;code&gt;fetch()&lt;/code&gt; didn't advertise Brotli; it asked for &lt;code&gt;gzip, deflate&lt;/code&gt;. The server sent Brotli anyway, and &lt;code&gt;fetch()&lt;/code&gt; decoded it. In the Node.js &lt;code&gt;fetch()&lt;/code&gt; implementation I tested, the response's &lt;code&gt;Content-Encoding&lt;/code&gt; decides which decoder runs, not the request's &lt;code&gt;Accept-Encoding&lt;/code&gt;. (Over HTTPS, this version does advertise &lt;code&gt;br&lt;/code&gt;, so a normal server will happily use it.)&lt;/p&gt;

&lt;p&gt;Two things this experiment does not show. It isn't an exploit: it proves the decoding path exists, not that the vulnerability can be triggered through it. And it ran on the patched v24.22.1, so it says nothing about the patched runtime being vulnerable. What it does establish is that, in the Node.js v24.22.1 &lt;code&gt;fetch()&lt;/code&gt; implementation I tested, a Brotli-encoded response reaches the native decoder even when the request did not advertise Brotli support.&lt;/p&gt;

&lt;p&gt;That changes the threat model. Don't treat &lt;code&gt;Accept-Encoding&lt;/code&gt; as a security boundary. With the built-in &lt;code&gt;fetch()&lt;/code&gt; I tested, whoever controls the response controls whether your process runs the Brotli decoder. If you use a different HTTP client or runtime, verify its behavior rather than assuming it matches.&lt;/p&gt;

&lt;h2&gt;
  
  
  Who Controls Your Responses?
&lt;/h2&gt;

&lt;p&gt;This is the production question, and the answer decides how urgent this is for you.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Calls to big, well-known APIs.&lt;/strong&gt; LinkedIn, Google, Stripe and the like are unlikely to send you malicious Brotli. The risk here is low, but not zero: a compromised upstream, a hijacked domain or a proxy in the middle all change who's really answering.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Calls to URLs your users give you.&lt;/strong&gt; This is where it gets serious. Think of webhook URL testers, link preview and Open Graph extractors, document importers, URL-to-PDF services, image proxies and RSS readers. In any feature where a user types a URL and your server fetches it, an attacker may control the response that reaches the decompressor. On an affected runtime, that's a potentially dangerous path and the first place to patch. The exact impact depends on the vulnerable code path and your environment, so don't assume every URL-fetching feature is automatically exploitable, but don't assume it's safe either.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Redirects.&lt;/strong&gt; A request to a host you trust can redirect to one you don't. &lt;code&gt;fetch()&lt;/code&gt; follows redirects by default, and the response that finally gets decoded may come from a server you never meant to talk to.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Incoming request bodies.&lt;/strong&gt; If any middleware in your stack decompresses request bodies based on the client's &lt;code&gt;Content-Encoding&lt;/code&gt; header, then every client on the internet can choose to send you Brotli. Check what your body parser does with compressed uploads; don't assume it ignores them.&lt;/p&gt;

&lt;p&gt;A useful analogy for all four: your code is the person who signed for the parcel, but &lt;code&gt;fetch()&lt;/code&gt; is the assistant who opens it. You never touched the box, and it still got opened in your office.&lt;/p&gt;

&lt;p&gt;One more point, because patching tends to feel like the end of the job: upgrading Node fixes this bug, not the risk of fetching user-supplied URLs. Those features still need normal server-side request forgery (SSRF) protection: restrict which destinations are allowed, validate the resolved IP address (not just the hostname), and re-check the destination on every redirect. The next section shows why the "validate the resolved address" part matters.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Other High-Severity Fixes
&lt;/h2&gt;

&lt;p&gt;Three more issues are rated High, and two of them are about exactly the kind of checks SSRF protection relies on:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;CVE-2026-105483: fullwidth-digit IP addresses.&lt;/strong&gt; TLS certificate hostname checks can be bypassed when an IP address is written with fullwidth digits (Unicode characters that look like &lt;code&gt;1&lt;/code&gt;, &lt;code&gt;2&lt;/code&gt;, &lt;code&gt;3&lt;/code&gt; but aren't ASCII).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;CVE-2026-106069: loosely resolved IP addresses.&lt;/strong&gt; A separate bypass, where a host that loosely resolves to an IP address passes the certificate hostname check. Hostname verification is the part of TLS that proves you're talking to the server you meant to. Service-to-service calls are where teams trust TLS the most and look at it the least.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;CVE-2026-105482: &lt;code&gt;dns.resolve&lt;/code&gt; truncates names containing a NUL character.&lt;/strong&gt; When a name is truncated at a NUL byte, the string your code checked and the name that actually gets resolved can differ. Not every allowlist is vulnerable, but any check-then-resolve flow that handles untrusted hostnames is worth a look.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;One Medium-rated fix matters to anyone who builds startup snapshots or single executable applications (SEAs): CVE-2026-106074. The entropy caches behind &lt;code&gt;crypto.randomUUID()&lt;/code&gt; and &lt;code&gt;crypto.randomInt()&lt;/code&gt; were being saved into the snapshot, so every process started from it repeats the same "random" values. The fix is in the new Node binary, but a snapshot you already built with an affected version still contains the old cached values. If you ship snapshots or SEAs, rebuild them with the patched Node. If you don't know what a startup snapshot is, you almost certainly don't use one.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Certificate Scare Reached Node Too
&lt;/h2&gt;

&lt;p&gt;The eMudhra incident from the same news cycle is easy to file under "browser problem." Eight eMudhra intermediate CA keys were so weak they could be recovered from public certificates, which in principle lets someone issue certificates that impersonate websites. Chrome, Firefox, Apple and Microsoft moved to block them within days.&lt;/p&gt;

&lt;p&gt;Browsers have their own trust stores and update them quickly. Non-browser clients are a different story: API clients, build pipelines, containers and server runtimes each trust whatever certificate list they were configured with. By default, Node verifies HTTPS connections against a CA list bundled into the binary (you can opt into the operating system's store with &lt;code&gt;--use-system-ca&lt;/code&gt;), and this release updates that bundled list in response to the eMudhra compromise.&lt;/p&gt;

&lt;p&gt;That doesn't mean every outdated Node app is about to have its traffic intercepted. An attacker would still need a usable certificate and a position on the network. But it does mean the browser fix didn't fix your servers. The runtime's trust configuration is part of its security surface, and an old binary carries an old list.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why This Can't Wait for Next Sprint
&lt;/h2&gt;

&lt;p&gt;The PHP flaws disclosed the same day show how quickly a public security fix attracts attention. &lt;a href="https://x.com/eastdakota/status/2113079352682570051" rel="noopener noreferrer"&gt;Cloudflare CEO Matthew Prince reported&lt;/a&gt; that Cloudflare's network saw thousands of exploit attempts in the 10 hours before the announcement (details had leaked early), tens of thousands in the 45 minutes after it, hundreds of thousands within two hours, and millions by the time he posted. A published patch also tells attackers where the bug was.&lt;/p&gt;

&lt;p&gt;That's the case for treating runtime security releases as urgent maintenance rather than routine dependency housekeeping. To be clear, it isn't evidence that CVE-2026-111497 is being exploited; I haven't found a reliable report that it is. The broader threat environment justifies the urgency on its own.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Checklist
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;1. Check what you run locally.&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;node &lt;span class="nt"&gt;-v&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;You want v22.23.4, v24.22.1, v26.12.1, or later on the same line. A local upgrade is the start, not the fix.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Check what production actually runs.&lt;/strong&gt; This is the step people skip. Your &lt;code&gt;package.json&lt;/code&gt; saying &lt;code&gt;"node": "24.x"&lt;/code&gt; is a range, not a version. On a managed platform, the provider picks the patch release. In a container, the base image does. Log the version at startup and read it from a real deployment:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="nx"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;info&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;nodeVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;process&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;version&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;nodeEnv&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;process&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;env&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;NODE_ENV&lt;/span&gt; &lt;span class="p"&gt;});&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Use logs, deployment metadata or authenticated internal tooling for this. Don't add a public endpoint that announces your runtime version to anyone who asks.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Upgrade, rebuild and redeploy.&lt;/strong&gt; Upgrade to the patched release on your supported line, rebuild the deployment artifact, redeploy it, then repeat step 2 against the new deployment.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. Make sure the image actually changed.&lt;/strong&gt; What matters is that the final image contains a patched Node, not that you ran a build. If you pin an exact version (&lt;code&gt;node:24.21.0-alpine&lt;/code&gt;), bump the pin deliberately. If you use a moving tag like &lt;code&gt;node:24&lt;/code&gt;, pull the updated base image and rebuild. Either way, check the result:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;docker run &lt;span class="nt"&gt;--rm&lt;/span&gt; your-image:tag node &lt;span class="nt"&gt;-v&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;5. Check your serverless platform's runtime policy.&lt;/strong&gt; Providers handle runtime patches differently: some roll them out automatically, some only on the next deploy. Find out which yours does, and redeploy if it's required.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;6. List every place you fetch a URL a user controls.&lt;/strong&gt; Webhook testers, link previews and Open Graph extractors, importers, URL-to-PDF services, image proxies. Patch the services behind them first, and confirm their redirect handling and destination validation while you're there.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;7. Rebuild snapshots and SEAs&lt;/strong&gt; if you use them, after upgrading Node.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;8. Get off Node 20 and Node 25.&lt;/strong&gt; They are end-of-life and will not receive these fixes.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Takeaway
&lt;/h2&gt;

&lt;p&gt;The chain is short: no &lt;code&gt;zlib&lt;/code&gt; import, then the built-in &lt;code&gt;fetch()&lt;/code&gt;, then Undici's decompression, then the native Brotli decoder. Security boundaries can sit below the layer where you normally look for them, and you can't remove this one from your own code.&lt;/p&gt;

&lt;p&gt;So the action is one you can take today: find out which Node version production is actually running, not which version your config file hopes for, and if it's older than the fixed release on its line, upgrade, rebuild and redeploy.&lt;/p&gt;

&lt;h2&gt;
  
  
  FAQ
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Can Node.js &lt;code&gt;fetch()&lt;/code&gt; expose an application to Brotli decoder vulnerabilities?&lt;/strong&gt;It can reach the decoder. In Node v24.22.1 (Undici 7.30.1), &lt;code&gt;fetch()&lt;/code&gt; passed a response marked &lt;code&gt;Content-Encoding: br&lt;/code&gt; to the native Brotli decoder even though the request didn't advertise Brotli, so a server that controls the response controls whether that decoder runs. My test shows that path exists; it doesn't show that CVE-2026-111497 can be triggered through it. The advisory is the source for the vulnerability and its impact, and upgrading is the fix either way.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Does removing &lt;code&gt;zlib&lt;/code&gt; imports eliminate the risk?&lt;/strong&gt; No. The decoder is used by the runtime on your behalf, through &lt;code&gt;fetch()&lt;/code&gt; and anything else that decompresses data. The fix is upgrading the Node runtime itself.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How do I verify the Node.js version running in production?&lt;/strong&gt; Log &lt;code&gt;process.version&lt;/code&gt; at startup and read it from a live deployment's logs, or check the runtime inside your built image with &lt;code&gt;docker run --rm your-image:tag node -v&lt;/code&gt;. Don't trust the &lt;code&gt;engines&lt;/code&gt; field or your local &lt;code&gt;node -v&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  References
&lt;/h2&gt;

&lt;p&gt;&lt;em&gt;Last checked against the Node.js advisory on October 22, 2026. If the advisory changes, this post will be updated and the change noted here.&lt;/em&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;a href="https://nodejs.org/en/blog/vulnerability/october-2026-security-releases" rel="noopener noreferrer"&gt;Node.js: October 21, 2026 Security Releases&lt;/a&gt;: full CVE list, severities, credits and fixed versions&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://nodejs.org/en/blog/release/v24.22.1" rel="noopener noreferrer"&gt;Node.js v24.22.1 release notes&lt;/a&gt;: the release used for the experiment above&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/nodejs/undici/blob/v7.30.1/lib/web/fetch/index.js#L2256-L2260" rel="noopener noreferrer"&gt;Undici v7.30.1, &lt;code&gt;lib/web/fetch/index.js&lt;/code&gt;&lt;/a&gt;: the response decoding code quoted in this post&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://x.com/eastdakota/status/2113079352682570051" rel="noopener noreferrer"&gt;Matthew Prince on X: #php2shell stats&lt;/a&gt;: the original post with Cloudflare's cumulative exploit-attempt figures&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://officechai.com/stories/cloudflare-ceo-says-php-exploit-attempts-went-from-thousands-to-millions-within-hours-of-disclosure/" rel="noopener noreferrer"&gt;OfficeChai: Cloudflare CEO on PHP exploit attempts&lt;/a&gt;: Matthew Prince's figures on attack traffic after the PHP disclosure&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://thehackernews.com/2026/10/eight-emudhra-ca-keys-compromised.html" rel="noopener noreferrer"&gt;The Hacker News: Eight eMudhra CA Keys Compromised&lt;/a&gt;: background on the certificate incident&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://blog.railway.com/p/security-disclosure-wednesday-october-21st" rel="noopener noreferrer"&gt;Railway: Security Disclosure, October 21&lt;/a&gt;: the Railway incident mentioned in the introduction&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Going Deeper
&lt;/h2&gt;

&lt;p&gt;The Node.js In-Depth modules on &lt;a href="https://academy.jatinjainsaraf.com/nodejs-in-depth/native-streams-off-heap" rel="noopener noreferrer"&gt;Native Streams and Off-Heap Buffer Storage&lt;/a&gt; (where &lt;code&gt;zlib&lt;/code&gt; streams live), &lt;a href="https://academy.jatinjainsaraf.com/nodejs-in-depth/config-security-rate-limiting" rel="noopener noreferrer"&gt;Configuration, Security Hardening, and Rate Limiting&lt;/a&gt;, &lt;a href="https://academy.jatinjainsaraf.com/nodejs-in-depth/npm-ecosystem" rel="noopener noreferrer"&gt;npm and the Node.js Ecosystem&lt;/a&gt; and &lt;a href="https://academy.jatinjainsaraf.com/nodejs-in-depth/docker-containerization-production" rel="noopener noreferrer"&gt;Dockerizing Node.js Applications for Production&lt;/a&gt; cover the pieces this post touched in more depth.&lt;/p&gt;

&lt;p&gt;And if this feels familiar, it should. In December 2025, &lt;a href="https://insight.jatinjainsaraf.com/react-js-next-js-security" rel="noopener noreferrer"&gt;the React Server Components remote code execution vulnerability&lt;/a&gt; taught the same lesson from a different layer: application security depends on the platform and framework underneath you, not only the code your team writes.&lt;/p&gt;

</description>
      <category>node</category>
      <category>security</category>
      <category>backend</category>
      <category>devops</category>
    </item>
    <item>
      <title>You Might Not Need a Queue Service: Postgres SKIP LOCKED as a Job Queue</title>
      <dc:creator>Jatin Jain Saraf</dc:creator>
      <pubDate>Thu, 22 Oct 2026 17:00:07 +0000</pubDate>
      <link>https://dev.to/jatinjainsaraf/you-might-not-need-a-queue-service-postgres-skip-locked-as-a-job-queue-1b77</link>
      <guid>https://dev.to/jatinjainsaraf/you-might-not-need-a-queue-service-postgres-skip-locked-as-a-job-queue-1b77</guid>
      <description>&lt;h2&gt;
  
  
  The Problem Every Indexer Hits Eventually
&lt;/h2&gt;

&lt;p&gt;A blockchain indexer starts life as one process reading one block at a time. (The full architecture of the one I work on is in the case study &lt;a href="https://insight.jatinjainsaraf.com/case-study/indexing-a-blockchain-in-real-time-at-scale" rel="noopener noreferrer"&gt;Indexing a Blockchain in Real Time, at Scale&lt;/a&gt;.) Then a backfill comes along: two million historical blocks to catch up on. One worker would take days, so you start ten.&lt;/p&gt;

&lt;p&gt;Now the ten workers need to split the work. Worker 3 takes blocks 0 to 9,999, worker 7 takes 10,000 to 19,999, and no two workers should be indexing the same range at the same time. When a worker crashes halfway through its range, someone else has to pick that range up.&lt;/p&gt;

&lt;p&gt;The reflex is to reach for a queue: Redis with BullMQ, SQS, RabbitMQ. That's one more service to deploy, monitor, secure and pay for, and one more place for state to drift out of sync with your database.&lt;/p&gt;

&lt;p&gt;Before you add any of that, look at the database you already run. PostgreSQL has had a feature built for exactly this since version 9.5: &lt;code&gt;FOR UPDATE SKIP LOCKED&lt;/code&gt;. With one table and one query, ten workers can claim different jobs concurrently without waiting on each other, and leases let another worker retry a job when its owner crashes.&lt;/p&gt;

&lt;p&gt;This post covers how it works, the traps that hurt in production (stale workers, long transactions and table bloat), how to watch the queue, and the point where you really should move to a dedicated queue.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Jobs Table
&lt;/h2&gt;

&lt;p&gt;Start with the table. Each row is one unit of work, here a range of blocks to index.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;jobs&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="n"&gt;bigserial&lt;/span&gt; &lt;span class="k"&gt;PRIMARY&lt;/span&gt; &lt;span class="k"&gt;KEY&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;kind&lt;/span&gt; &lt;span class="nb"&gt;text&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;payload&lt;/span&gt; &lt;span class="n"&gt;jsonb&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="nb"&gt;text&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt; &lt;span class="k"&gt;DEFAULT&lt;/span&gt; &lt;span class="s1"&gt;'pending'&lt;/span&gt;
                &lt;span class="k"&gt;CHECK&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="k"&gt;IN&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'pending'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'running'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'done'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'failed'&lt;/span&gt;&lt;span class="p"&gt;)),&lt;/span&gt;
  &lt;span class="n"&gt;run_at&lt;/span&gt; &lt;span class="n"&gt;timestamptz&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt; &lt;span class="k"&gt;DEFAULT&lt;/span&gt; &lt;span class="n"&gt;now&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt;
  &lt;span class="n"&gt;attempts&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt; &lt;span class="k"&gt;DEFAULT&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;locked_by&lt;/span&gt; &lt;span class="nb"&gt;text&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;claim_token&lt;/span&gt; &lt;span class="n"&gt;uuid&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;locked_until&lt;/span&gt; &lt;span class="n"&gt;timestamptz&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;created_at&lt;/span&gt; &lt;span class="n"&gt;timestamptz&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt; &lt;span class="k"&gt;DEFAULT&lt;/span&gt; &lt;span class="n"&gt;now&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="k"&gt;INSERT&lt;/span&gt; &lt;span class="k"&gt;INTO&lt;/span&gt; &lt;span class="n"&gt;jobs&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;payload&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="s1"&gt;'index_blocks'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
       &lt;span class="n"&gt;jsonb_build_object&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'from'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;g&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'to'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;g&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;9999&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;generate_series&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1999999&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;10000&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="k"&gt;g&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That's 200 jobs. Now ten workers need to take them one at a time, safely. (&lt;code&gt;claim_token&lt;/code&gt; will matter later, when we deal with workers that come back from the dead.)&lt;/p&gt;

&lt;h2&gt;
  
  
  Three Ways to Get It Wrong
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Attempt 1: Read, then update
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;jobs&lt;/span&gt; &lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'pending'&lt;/span&gt; &lt;span class="k"&gt;ORDER&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;run_at&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="k"&gt;LIMIT&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="c1"&gt;-- application code holds the id&lt;/span&gt;
&lt;span class="k"&gt;UPDATE&lt;/span&gt; &lt;span class="n"&gt;jobs&lt;/span&gt; &lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'running'&lt;/span&gt; &lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two workers run the &lt;code&gt;SELECT&lt;/code&gt; at the same moment, both see job 42, and both update it. Nothing in the second statement checks that the job is still pending, so both workers index the same blocks.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;In production:&lt;/strong&gt; duplicate rows, unique constraint violations in the indexed tables, and a backfill that does more work than it needs to. You might add &lt;code&gt;AND status = 'pending'&lt;/code&gt; to the update and check the affected row count, which closes the race, but now the loser has to retry, and under load most workers spend their time losing.&lt;/p&gt;

&lt;h3&gt;
  
  
  Attempt 2: Lock the row with FOR UPDATE
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;BEGIN&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;jobs&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'pending'&lt;/span&gt;
&lt;span class="k"&gt;ORDER&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;run_at&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt;
&lt;span class="k"&gt;LIMIT&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
&lt;span class="k"&gt;FOR&lt;/span&gt; &lt;span class="k"&gt;UPDATE&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="c1"&gt;-- ... mark it running, commit ...&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This one is correct: only one transaction can hold the row lock. (How row locks and advisory locks work underneath is covered in the &lt;a href="https://academy.jatinjainsaraf.com/postgresql-in-depth/locking-internals" rel="noopener noreferrer"&gt;Locking Internals&lt;/a&gt; module of PostgreSQL In-Depth.) The problem is what the other nine workers do. They all run the same query on the same data, so they all reach the same first row and wait for its lock.&lt;/p&gt;

&lt;p&gt;When the lock is released, PostgreSQL rechecks the row against the query's conditions. Its status is now &lt;code&gt;running&lt;/code&gt;, so the waiting worker can't have it and moves on to the next pending row. But the other eight waiters do the same thing at the same moment, so they pile onto the second row together, and the line forms again.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;In production:&lt;/strong&gt; you scale from one worker to ten and throughput barely moves. With this query pattern, adding workers mostly adds lock contention, not throughput. Workers spend their time waiting in a line for rows someone else is about to take, instead of claiming independent jobs.&lt;/p&gt;

&lt;h3&gt;
  
  
  Attempt 3: Advisory locks
&lt;/h3&gt;

&lt;p&gt;&lt;code&gt;pg_try_advisory_lock(id)&lt;/code&gt; can work, but you manage the lock yourself, session-level locks survive a connection being returned to the pool, and a lock call in the &lt;code&gt;WHERE&lt;/code&gt; clause can lock rows the &lt;code&gt;LIMIT&lt;/code&gt; later throws away. It's a sharp tool for a problem that already has a built-in answer.&lt;/p&gt;

&lt;h2&gt;
  
  
  SKIP LOCKED: Walk Past the Plates Someone Is Holding
&lt;/h2&gt;

&lt;p&gt;Picture a buffet. With plain &lt;code&gt;FOR UPDATE&lt;/code&gt;, everyone queues behind the first person and waits for them to finish choosing. With &lt;code&gt;SKIP LOCKED&lt;/code&gt;, if someone already has their hand on a plate, you take the next one. Nobody waits and nobody grabs the same plate.&lt;/p&gt;

&lt;p&gt;That's exactly what the clause does. When the query hits a row that another transaction has locked, it doesn't wait. It skips the row and keeps scanning until it has found enough rows it &lt;em&gt;can&lt;/em&gt; lock.&lt;/p&gt;

&lt;p&gt;Here's the claim query. It picks a job, locks it, marks it running and returns it, all in one statement:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;WITH&lt;/span&gt; &lt;span class="k"&gt;next&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt;
  &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;jobs&lt;/span&gt;
  &lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'pending'&lt;/span&gt;
    &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="n"&gt;run_at&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="n"&gt;now&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
  &lt;span class="k"&gt;ORDER&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;run_at&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt;
  &lt;span class="k"&gt;LIMIT&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
  &lt;span class="k"&gt;FOR&lt;/span&gt; &lt;span class="k"&gt;UPDATE&lt;/span&gt; &lt;span class="n"&gt;SKIP&lt;/span&gt; &lt;span class="n"&gt;LOCKED&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;UPDATE&lt;/span&gt; &lt;span class="n"&gt;jobs&lt;/span&gt; &lt;span class="n"&gt;j&lt;/span&gt;
&lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'running'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;attempts&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;j&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;attempts&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;locked_by&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;claim_token&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;gen_random_uuid&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt;
    &lt;span class="n"&gt;locked_until&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;now&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;interval&lt;/span&gt; &lt;span class="s1"&gt;'5 minutes'&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="k"&gt;next&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;j&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;next&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;
&lt;span class="n"&gt;RETURNING&lt;/span&gt; &lt;span class="n"&gt;j&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;j&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;j&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;payload&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;j&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;claim_token&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Run it from ten workers at once and each gets a different job. Raise the &lt;code&gt;LIMIT&lt;/code&gt; to claim a batch of ten in one round trip. &lt;code&gt;id&lt;/code&gt; in the &lt;code&gt;ORDER BY&lt;/code&gt; is a tie-breaker, so jobs scheduled for the same instant come out in a predictable order.&lt;/p&gt;

&lt;p&gt;One thing to know: &lt;code&gt;SKIP LOCKED&lt;/code&gt; deliberately gives you an inconsistent view of the table, because it pretends locked rows don't exist. That's exactly right for a queue and exactly wrong for almost anything else, such as reports, balances or counts. Keep it on queue tables. The &lt;a href="https://www.postgresql.org/docs/current/sql-select.html#SQL-FOR-UPDATE-SHARE" rel="noopener noreferrer"&gt;PostgreSQL docs on the locking clause&lt;/a&gt; say the same.&lt;/p&gt;

&lt;h2&gt;
  
  
  The First Trap: Holding the Transaction Open
&lt;/h2&gt;

&lt;p&gt;The tempting way to run a job looks like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;BEGIN&lt;/span&gt;
  &lt;span class="n"&gt;claim&lt;/span&gt; &lt;span class="n"&gt;job&lt;/span&gt; &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="n"&gt;SKIP&lt;/span&gt; &lt;span class="n"&gt;LOCKED&lt;/span&gt;
  &lt;span class="k"&gt;index&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;000&lt;/span&gt; &lt;span class="n"&gt;blocks&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;-&lt;/span&gt; &lt;span class="n"&gt;minutes&lt;/span&gt;
  &lt;span class="n"&gt;mark&lt;/span&gt; &lt;span class="n"&gt;job&lt;/span&gt; &lt;span class="n"&gt;done&lt;/span&gt;
&lt;span class="k"&gt;COMMIT&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It feels elegant. If the worker crashes, the transaction rolls back and the row lock is released, so the job becomes claimable again with no extra code.&lt;/p&gt;

&lt;p&gt;It's also one of the most expensive patterns you can run against PostgreSQL:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Every worker pins a database connection for the whole job.&lt;/strong&gt; Ten workers means ten connections sitting idle in a transaction for minutes. With a small pool, the rest of your application starts waiting for connections.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Long transactions hold back vacuum.&lt;/strong&gt; &lt;a href="https://academy.jatinjainsaraf.com/postgresql-in-depth/mvcc" rel="noopener noreferrer"&gt;MVCC&lt;/a&gt; is time travel for rows: old row versions stay around as long as any running transaction might still need to see them. This transaction has already written something (the claim), which gave it a transaction ID. Until it ends, vacuum can't remove row versions that became dead after that ID was assigned, because Postgres has to treat the transaction as one that might still need them. A read-only transaction holds vacuum back the same way while it keeps an old snapshot, for example under &lt;code&gt;REPEATABLE READ&lt;/code&gt;. The effect reaches every table in the database, not just the jobs table. Your hottest tables bloat while the job runs.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Idle-in-transaction timeouts will kill it.&lt;/strong&gt; If you've set &lt;code&gt;idle_in_transaction_session_timeout&lt;/code&gt; (and you should have), long jobs get killed partway through.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The fix is to &lt;strong&gt;claim and commit straight away&lt;/strong&gt; , then run the job outside any transaction. The claim query above does exactly that: it's a single auto-committed statement. The row is no longer locked, but &lt;code&gt;status = 'running'&lt;/code&gt; keeps other workers away from it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Leases Replace Locks
&lt;/h2&gt;

&lt;p&gt;Once the job runs outside a transaction, a crashed worker no longer releases anything automatically. That's what &lt;code&gt;locked_until&lt;/code&gt; is for. It's a lease: the worker owns the job until that time, and then the job is up for grabs again.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why a worker ID isn't enough
&lt;/h3&gt;

&lt;p&gt;Picture this sequence:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Worker A claims job 42.&lt;/li&gt;
&lt;li&gt;Worker A stalls in a long garbage-collection pause, or loses its network for a minute.&lt;/li&gt;
&lt;li&gt;Its lease expires, and the job goes back to pending.&lt;/li&gt;
&lt;li&gt;Worker A's process restarts with the same worker ID and claims job 42 again (or worker B claims it).&lt;/li&gt;
&lt;li&gt;The old, stalled execution of worker A wakes up and marks the job done.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;If ownership checks only compare &lt;code&gt;locked_by&lt;/code&gt;, step 5 succeeds whenever the new claim carries the same worker ID, and a stale execution completes a claim that belongs to someone else. That's why the claim query generates a fresh &lt;code&gt;claim_token&lt;/code&gt; for &lt;strong&gt;every claim&lt;/strong&gt;. The worker ID says who is working on the job. The token says which claim this is.&lt;/p&gt;

&lt;p&gt;A long job extends its lease as it goes (a heartbeat), and only the current claim can do that:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;UPDATE&lt;/span&gt; &lt;span class="n"&gt;jobs&lt;/span&gt;
&lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;locked_until&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;now&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;interval&lt;/span&gt; &lt;span class="s1"&gt;'5 minutes'&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;
  &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="n"&gt;claim_token&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;
  &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'running'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Completion checks the same token:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;UPDATE&lt;/span&gt; &lt;span class="n"&gt;jobs&lt;/span&gt;
&lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'done'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;locked_by&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;claim_token&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;locked_until&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;
  &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="n"&gt;claim_token&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;
  &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'running'&lt;/span&gt;
&lt;span class="n"&gt;RETURNING&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If no row comes back, the claim was lost: another worker now owns the job, or the reaper took it back. The worker must stop processing and must not treat the job as completed. That doesn't undo side effects it has already performed, which is why those need to be idempotent (more on that below). If a heartbeat returns no row, the worker should stop working on the job right away.&lt;/p&gt;

&lt;p&gt;A reaper, which can be a cron job or any worker on a timer, takes back expired leases. It clears the token, so no stale execution can match it afterwards. It also pushes the retry into the future with exponential backoff and some jitter, so a job that fails because a downstream service is down doesn't get retried in lockstep with every other failing job (the &lt;a href="https://academy.jatinjainsaraf.com/system-design-in-depth/timeouts-retries-backoff" rel="noopener noreferrer"&gt;Timeouts, Retries, and Backoff&lt;/a&gt; module explains why jitter matters):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;UPDATE&lt;/span&gt; &lt;span class="n"&gt;jobs&lt;/span&gt;
&lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;CASE&lt;/span&gt; &lt;span class="k"&gt;WHEN&lt;/span&gt; &lt;span class="n"&gt;attempts&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="mi"&gt;5&lt;/span&gt; &lt;span class="k"&gt;THEN&lt;/span&gt; &lt;span class="s1"&gt;'failed'&lt;/span&gt; &lt;span class="k"&gt;ELSE&lt;/span&gt; &lt;span class="s1"&gt;'pending'&lt;/span&gt; &lt;span class="k"&gt;END&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;run_at&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;now&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;make_interval&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
                     &lt;span class="n"&gt;secs&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="n"&gt;least&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
                       &lt;span class="mi"&gt;3600&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                       &lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;power&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;least&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;attempts&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;random&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt;
                     &lt;span class="p"&gt;)&lt;/span&gt;
                   &lt;span class="p"&gt;),&lt;/span&gt;
    &lt;span class="n"&gt;locked_by&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;claim_token&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;locked_until&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'running'&lt;/span&gt;
  &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="n"&gt;locked_until&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="n"&gt;now&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The cap applies after the jitter, so no retry waits longer than an hour. Because the claim query increments &lt;code&gt;attempts&lt;/code&gt; when a job is claimed, the first failed attempt waits 30 to 90 seconds, the second 60 to 180, and so on until the one-hour cap. Use the same backoff when a worker catches an error and gives the job back on purpose. If many leases can expire at once (say, a whole worker fleet restarts), reap in bounded batches with the same &lt;code&gt;SKIP LOCKED&lt;/code&gt; pattern instead of one big update, so recovery doesn't arrive as a burst of WAL and dead tuples.&lt;/p&gt;

&lt;p&gt;Note what actually takes a job away from a worker here. An expired lease only makes the job eligible for recovery. The reaper is what revokes the claim, by clearing its token. So a worker that finishes just after its lease expires, before the reaper has run, can still complete the job, and that's fine: nobody else has claimed it yet. Once the reaper clears the token, the old claim can never match again.&lt;/p&gt;

&lt;h3&gt;
  
  
  At-least-once, not exactly-once
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;A job can still run more than once.&lt;/strong&gt; A worker can finish the work and crash just before marking it done. The lease expires and another worker runs it again. That's not a bug in this design. Every queue that redelivers work behaves this way, including SQS and BullMQ. The queue gives you at-least-once processing, not exactly-once.&lt;/p&gt;

&lt;p&gt;The claim token protects the queue row. It can't stop a stale worker from sending an email or calling a webhook after it has lost its lease. Checking the token right before the side effect makes the window smaller but doesn't close it. Closing it completely needs the receiving system to reject stale writers, which is what fencing tokens do (see &lt;a href="https://academy.jatinjainsaraf.com/system-design-in-depth/distributed-locks-and-coordination" rel="noopener noreferrer"&gt;Distributed Locks and Coordination&lt;/a&gt;). Idempotency has to cover the job's real side effects: for the indexer, that means writing indexed rows with &lt;code&gt;INSERT ... ON CONFLICT DO NOTHING&lt;/code&gt; or an upsert, so indexing the same block range twice gives the same result. For external calls, use idempotency keys or deduplication on the receiving side. I covered the pattern in &lt;a href="https://insight.jatinjainsaraf.com/idempotency-keys-how-to-make-retries-safe-in-distributed-systems" rel="noopener noreferrer"&gt;Idempotency Keys: How to Make Retries Safe in Distributed Systems&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Second Trap: The Queue Table Bloats Itself
&lt;/h2&gt;

&lt;p&gt;A queue table is the worst-case workload for MVCC. Every job is inserted, updated to running, updated again (sometimes several times, for heartbeats) and then marked done or deleted. In PostgreSQL an &lt;code&gt;UPDATE&lt;/code&gt; doesn't change a row in place. It writes a new version and leaves the old one behind as a dead tuple for vacuum to clean up.&lt;/p&gt;

&lt;p&gt;So a table that only ever holds a few hundred pending jobs can contain hundreds of thousands of dead row versions. The production symptom is subtle: &lt;strong&gt;the claim query slowly gets slower&lt;/strong&gt; , even though the queue is short. Each claim has to step over dead index entries and dead heap tuples to find the live pending rows at the front of the queue.&lt;/p&gt;

&lt;p&gt;Three things keep it under control.&lt;/p&gt;

&lt;h3&gt;
  
  
  1. A partial index on exactly what the claim query reads
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;INDEX&lt;/span&gt; &lt;span class="n"&gt;jobs_claimable_idx&lt;/span&gt;
  &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;jobs&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run_at&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'pending'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It matches the claim query's filter and its &lt;code&gt;ORDER BY run_at, id&lt;/code&gt;, so the claim reads the front of the index directly. Because it only covers pending jobs, it's far smaller than an index over the whole table.&lt;/p&gt;

&lt;p&gt;It is not automatically tiny, though. When a job moves from pending to running, its old index entry doesn't vanish. It stays in the index as a dead entry until vacuum removes it. (Why B-tree entries outlive the rows they point to is covered in &lt;a href="https://insight.jatinjainsaraf.com/btree-reads-inserts-updates-deletes-blockchain-indexer" rel="noopener noreferrer"&gt;How a B-Tree Really Reads, Inserts, Updates and Deletes&lt;/a&gt;.) Under heavy churn the index fills up with entries for jobs that were claimed long ago, even when only a handful are pending right now. The partial index makes the claim efficient. It doesn't replace vacuum.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Keep finished jobs out of the hot table
&lt;/h3&gt;

&lt;p&gt;Don't let &lt;code&gt;done&lt;/code&gt; rows build up in the table you claim from. Either delete them when they finish, or move them to a history table on a schedule. If you need to keep history for weeks, partition the history table by time and drop old partitions. Dropping a partition removes its data as a catalog and file operation, without creating a dead tuple per row, while &lt;code&gt;DELETE&lt;/code&gt; of a million rows creates a million dead tuples for vacuum to process. It isn't free: a plain detach or drop takes an &lt;code&gt;ACCESS EXCLUSIVE&lt;/code&gt; lock on the parent table. That mode conflicts with everything, even plain reads, so while the request waits for running queries to finish, every new query on that table queues behind it. If that matters, use &lt;code&gt;ALTER TABLE ... DETACH PARTITION ... CONCURRENTLY&lt;/code&gt; first. It takes a weaker lock, but it can't run inside a transaction block or when the table has a default partition, and if it's interrupted you have to finish it with &lt;code&gt;DETACH PARTITION ... FINALIZE&lt;/code&gt;. The &lt;a href="https://academy.jatinjainsaraf.com/postgresql-in-depth/advanced-partitioning" rel="noopener noreferrer"&gt;Advanced Partitioning&lt;/a&gt; module covers partition maintenance with pg_partman, and &lt;a href="https://insight.jatinjainsaraf.com/case-study/scaling-a-blockchain-database-from-a-single-partition-to-4-6tb" rel="noopener noreferrer"&gt;Scaling a Blockchain Database From a Single Partition to 4.6 TB&lt;/a&gt; shows the same idea on a real indexer.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. Tell autovacuum this table is special
&lt;/h3&gt;

&lt;p&gt;Autovacuum's default trigger is a fraction of the table's estimated size, tuned for ordinary tables. A queue table has a small live set and a high churn rate, so give it a fixed trigger instead:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;jobs&lt;/span&gt; &lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="n"&gt;autovacuum_vacuum_scale_factor&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;autovacuum_vacuum_threshold&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;1000&lt;/span&gt;
&lt;span class="p"&gt;);&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That means: vacuum after roughly every 1,000 dead tuples, however big the table is. The 1,000 is a starting point, not a recommendation for every system. Pick it from the table's update rate, how long a vacuum takes, and how much bloat you can tolerate.&lt;/p&gt;

&lt;p&gt;Leave vacuum throttling (&lt;code&gt;autovacuum_vacuum_cost_delay&lt;/code&gt;) at its default to begin with. Turning it off lets vacuum run flat out, and on a database that's already busy with indexing traffic, that I/O competes with your real queries. If vacuum can't keep up, look at how long it runs, the cost limits, your spare I/O and the competing workload before you touch throttling. The &lt;a href="https://www.postgresql.org/docs/current/routine-vacuuming.html" rel="noopener noreferrer"&gt;routine vacuuming docs&lt;/a&gt; cover these settings in depth, the &lt;a href="https://academy.jatinjainsaraf.com/postgresql-in-depth/autovacuum" rel="noopener noreferrer"&gt;Autovacuum&lt;/a&gt; module walks through the settings people most often get wrong, and I wrote about what autovacuum actually does in &lt;a href="https://insight.jatinjainsaraf.com/postgres-wal-checkpoints-wait-events-vacuum-analyze-explained" rel="noopener noreferrer"&gt;Every Postgres Write Triggers Five Background Processes&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;And remember the first trap: none of this works if a long-running transaction is holding back vacuum. Bloat and long transactions are the same problem seen from two sides.&lt;/p&gt;

&lt;h2&gt;
  
  
  Watching the Queue in Production
&lt;/h2&gt;

&lt;p&gt;A Postgres-backed queue needs watching at two levels.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Queue health:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Pending and running job counts.&lt;/li&gt;
&lt;li&gt;Age of the oldest pending job (the number your users actually feel).&lt;/li&gt;
&lt;li&gt;Job duration, retry rate and failure rate.&lt;/li&gt;
&lt;li&gt;Expired leases per hour, which tells you about crashing or stalling workers.&lt;/li&gt;
&lt;li&gt;Claim query latency.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Database health:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Dead tuple trend and autovacuum frequency for the jobs table, from &lt;code&gt;pg_stat_user_tables&lt;/code&gt;. &lt;code&gt;n_dead_tup&lt;/code&gt; is an estimate, and &lt;code&gt;last_autovacuum&lt;/code&gt; alone doesn't prove vacuum is keeping up, so watch the trend alongside claim latency.&lt;/li&gt;
&lt;li&gt;Connection usage.&lt;/li&gt;
&lt;li&gt;WAL volume and replication lag. Queue churn is all writes, and on an indexer that can push replicas behind (see &lt;a href="https://insight.jatinjainsaraf.com/taming-postgresql-replication-lag-in-real-time-blockchain-indexers" rel="noopener noreferrer"&gt;Taming PostgreSQL Replication Lag in Real-Time Blockchain Indexers&lt;/a&gt;).&lt;/li&gt;
&lt;li&gt;Long-running transactions and sessions stuck idle in transaction, from &lt;code&gt;pg_stat_activity&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A growing queue doesn't automatically mean Postgres is the bottleneck. Jobs can back up because an external API is slow or because you don't have enough workers. The age of the oldest job and the job duration, read together, tell you which.&lt;/p&gt;

&lt;h2&gt;
  
  
  Stop Polling: LISTEN and NOTIFY
&lt;/h2&gt;

&lt;p&gt;Idle workers need to find out when new work arrives. Running the claim query every 100 milliseconds from ten workers is a lot of wasted queries on an empty queue. Running it every 10 seconds adds up to 10 seconds of latency.&lt;/p&gt;

&lt;p&gt;PostgreSQL has a built-in wake-up signal. Workers listen on a channel:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;LISTEN jobs_ready;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Whatever enqueues work sends a notification in the same transaction:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;BEGIN&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;INSERT&lt;/span&gt; &lt;span class="k"&gt;INTO&lt;/span&gt; &lt;span class="n"&gt;jobs&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;payload&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;VALUES&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'index_blocks'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'{"from": 2000000, "to": 2009999}'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="k"&gt;NOTIFY&lt;/span&gt; &lt;span class="n"&gt;jobs_ready&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;COMMIT&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Notifications are delivered only when the transaction commits, so a worker never wakes up for a job it can't see yet. Treat the notification as a hint to run the claim query, not as the job itself. The job lives in the table. The notification is just a tap on the shoulder.&lt;/p&gt;

&lt;p&gt;Three things to know before you rely on it:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Notifications aren't a durable delivery mechanism.&lt;/strong&gt; A session that isn't connected and listening when the notification is sent never gets it. A worker restarting during a deploy will miss some. Keep a slow poll, every 30 seconds or so, as a safety net.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;LISTEN needs a dedicated session.&lt;/strong&gt; It doesn't work through PgBouncer in transaction pooling mode, because your next query may run on a different server connection. Give each worker one direct connection for listening. More on why in &lt;a href="https://insight.jatinjainsaraf.com/connection-pooling-in-the-serverless-era-five-failure-modes" rel="noopener noreferrer"&gt;Connection Pooling in the Serverless Era: Five Failure Modes&lt;/a&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;NOTIFY serializes part of the commit.&lt;/strong&gt; A transaction that has sent a notification takes a global lock near the end of its commit, so transactions that call &lt;code&gt;NOTIFY&lt;/code&gt; queue up behind each other at that step. Commits that don't notify aren't affected by this particular bottleneck. At low rates you'll never notice. If you enqueue thousands of jobs per second, send one notification per batch, not one per row.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;See the docs for &lt;a href="https://www.postgresql.org/docs/current/sql-listen.html" rel="noopener noreferrer"&gt;LISTEN&lt;/a&gt; and &lt;a href="https://www.postgresql.org/docs/current/sql-notify.html" rel="noopener noreferrer"&gt;NOTIFY&lt;/a&gt; for the exact delivery rules.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Feature No Queue Service Can Give You
&lt;/h2&gt;

&lt;p&gt;There's one advantage that's easy to miss: &lt;strong&gt;you can enqueue a job in the same transaction as the business data that caused it.&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;BEGIN&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;INSERT&lt;/span&gt; &lt;span class="k"&gt;INTO&lt;/span&gt; &lt;span class="n"&gt;blocks&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;height&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;hash&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;...)&lt;/span&gt; &lt;span class="k"&gt;VALUES&lt;/span&gt; &lt;span class="p"&gt;(...);&lt;/span&gt;
&lt;span class="k"&gt;INSERT&lt;/span&gt; &lt;span class="k"&gt;INTO&lt;/span&gt; &lt;span class="n"&gt;jobs&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;payload&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;VALUES&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'index_events'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'{"height": 2000123}'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="k"&gt;COMMIT&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If the transaction commits, both rows become visible together. If it rolls back, neither is written. There's no window where the block exists without its job.&lt;/p&gt;

&lt;p&gt;With Redis or SQS you'd write to the database and then publish to the queue, and if the process dies between the two, you either lose the job or queue work for data that was never committed. That's the dual-write problem. A transactional outbox solves it by recording the event in the same transaction as the business data and publishing it to another system afterwards. An outbox and a jobs table are built the same way, a row written atomically with your data, but they differ in purpose: an outbox holds events waiting to be delivered somewhere else, while a jobs table holds work your own workers execute. I covered the outbox in detail in &lt;a href="https://insight.jatinjainsaraf.com/the-dual-write-problem-outbox-sagas-and-2pc" rel="noopener noreferrer"&gt;The Dual-Write Problem&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  When You Really Do Need a Queue Service
&lt;/h2&gt;

&lt;p&gt;Postgres as a queue isn't free and it isn't unlimited. Move to something dedicated when:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Queue traffic starts hurting application traffic.&lt;/strong&gt; Every claim, heartbeat, retry and status change is a write on your primary. Each one produces WAL and dead tuples, and each one competes with your application for CPU, disk, connections and vacuum. There's no universal jobs-per-second number where Postgres stops being a good queue: a job that's claimed once and completed once costs far less than one that heartbeats every 30 seconds and retries three times. Benchmark with your real payloads, batch sizes and heartbeat interval, and watch queue latency, application query latency and replication lag together.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;You need fan-out.&lt;/strong&gt; One event consumed independently by five different services is pub/sub. Kafka, Redis Streams or SNS-to-SQS fit that shape. A jobs table doesn't.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;You need a replayable log.&lt;/strong&gt; If consumers have to re-read last Tuesday's events, you want a retained log like Kafka, not a table you're constantly deleting from.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;You need strict ordering.&lt;/strong&gt; &lt;code&gt;SKIP LOCKED&lt;/code&gt; hands out jobs roughly in order, but job 2 can finish before job 1. If order per key matters (all events for one account, in sequence), you have to design for it, and a partitioned log does that more naturally.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The work crosses service boundaries.&lt;/strong&gt; If the producer and consumer don't share a database, a shared jobs table becomes a coupling you'll regret.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For an indexer backfill, background emails, webhook deliveries, report generation, or most of the jobs a typical backend runs, none of these apply.&lt;/p&gt;

&lt;p&gt;If you do outgrow it, the System Design In-Depth modules on &lt;a href="https://academy.jatinjainsaraf.com/system-design-in-depth/message-queues" rel="noopener noreferrer"&gt;Message Queues&lt;/a&gt;, &lt;a href="https://academy.jatinjainsaraf.com/system-design-in-depth/event-streaming-kafka" rel="noopener noreferrer"&gt;Event Streaming with Kafka&lt;/a&gt; and &lt;a href="https://academy.jatinjainsaraf.com/system-design-in-depth/background-jobs-and-scheduling" rel="noopener noreferrer"&gt;Background Jobs and Scheduling&lt;/a&gt; cover what to move to, and &lt;a href="https://academy.jatinjainsaraf.com/redis-in-depth/bullmq-internals" rel="noopener noreferrer"&gt;BullMQ Internals&lt;/a&gt; shows how the most common Redis queue does the same claim-and-lease dance with Redis data structures.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Checklist
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Claim jobs with &lt;code&gt;FOR UPDATE SKIP LOCKED&lt;/code&gt;, ordered by &lt;code&gt;run_at, id&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Commit straight after claiming. Never hold a transaction open for the whole job.&lt;/li&gt;
&lt;li&gt;Use leases with a fresh claim token per claim, and check the token on every heartbeat and completion.&lt;/li&gt;
&lt;li&gt;Retry with backoff and jitter, and cap the attempts.&lt;/li&gt;
&lt;li&gt;Make the job's side effects idempotent. The queue is at-least-once.&lt;/li&gt;
&lt;li&gt;Index the claim query with a matching partial index.&lt;/li&gt;
&lt;li&gt;Keep finished jobs out of the hot table and tune autovacuum from observed churn.&lt;/li&gt;
&lt;li&gt;Watch the oldest pending job's age, expired leases and dead tuples.&lt;/li&gt;
&lt;li&gt;Keep a polling fallback when you use &lt;code&gt;LISTEN&lt;/code&gt; and &lt;code&gt;NOTIFY&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Benchmark before you move to a dedicated queue.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The Takeaway
&lt;/h2&gt;

&lt;p&gt;A queue is a table where many readers each need to take a different row. PostgreSQL ships with the exact primitive for that. &lt;code&gt;FOR UPDATE SKIP LOCKED&lt;/code&gt; lets parallel workers claim different jobs without waiting on each other. Leases with claim tokens take care of crashed and stalled workers. A partial index plus vacuum settings tuned to the table keep the claim fast. And transactional enqueue gives you a guarantee that an external queue can't.&lt;/p&gt;

&lt;p&gt;Start with the database you already run. Add a queue service when you can name the limit you hit, not before.&lt;/p&gt;

</description>
      <category>postgres</category>
      <category>database</category>
      <category>systemdesign</category>
      <category>concurrency</category>
    </item>
    <item>
      <title>How a B-Tree Really Reads, Inserts, Updates and Deletes (and Where It Fails a Blockchain Indexer)</title>
      <dc:creator>Jatin Jain Saraf</dc:creator>
      <pubDate>Fri, 09 Oct 2026 03:34:03 +0000</pubDate>
      <link>https://dev.to/jatinjainsaraf/how-a-b-tree-really-reads-inserts-updates-and-deletes-and-where-it-fails-a-blockchain-indexer-29g</link>
      <guid>https://dev.to/jatinjainsaraf/how-a-b-tree-really-reads-inserts-updates-and-deletes-and-where-it-fails-a-blockchain-indexer-29g</guid>
      <description>&lt;h1&gt;
  
  
  How a B-Tree Really Reads, Inserts, Updates and Deletes (and Where It Fails a Blockchain Indexer)
&lt;/h1&gt;

&lt;p&gt;Every indexer I have worked on started the same way. Someone creates the &lt;code&gt;transactions&lt;/code&gt; table, adds an index on every column the API might filter by, and ships. For the first hundred million rows, everything is fast.&lt;/p&gt;

&lt;p&gt;Then ingestion starts falling behind the chain. The wallet history endpoint is still quick, but the balance updater takes twice as long per block as it did last month. Disk usage grows faster than row count. A cleanup job deletes old spam events and the table gets no smaller.&lt;/p&gt;

&lt;p&gt;None of that is necessarily a bug. The database is doing exactly what its indexes require. The workload has simply outgrown the assumptions behind the original schema.&lt;/p&gt;

&lt;p&gt;This post walks through what a B-tree physically does on each of the four operations, using PostgreSQL's implementation, and then applies that to the tables of a real blockchain indexer: blocks, transactions, coins, fungible assets, NFTs, digital assets, wallet history and portfolio. For each one, the question is the same: is a B-tree the right tool here, and if not, what is?&lt;/p&gt;

&lt;h2&gt;
  
  
  The Shape: A Dictionary With a Table of Contents
&lt;/h2&gt;

&lt;p&gt;A B-tree is a sorted dictionary with a table of contents stacked on top of it.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Leaf pages&lt;/strong&gt; hold the actual entries: a key (say, a wallet address) and a pointer to the row in the table (Postgres calls this pointer a TID, the physical location of the row). Leaf pages are sorted and linked to their neighbors, left and right, like pages in a book.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Internal pages&lt;/strong&gt; hold no rows. They hold signposts: "keys below X are on this page, keys from X onward are on that one."&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The root&lt;/strong&gt; is the single page at the top. In a small index, the root is itself the only leaf, and internal levels appear as the index grows.
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;                    [root: ... | 0x5a.. | 0xb3.. | ...]
                       / | \
        [internal] [internal] [internal]
         / | \ / | \ / | \
     [leaf]&amp;lt;-&amp;gt;[leaf]&amp;lt;-&amp;gt;[leaf]&amp;lt;-&amp;gt;[leaf]&amp;lt;-&amp;gt;[leaf]&amp;lt;-&amp;gt;[leaf]&amp;lt;-&amp;gt;[leaf] ...

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Each page is 8 KB in Postgres. As an illustrative estimate, a leaf page of &lt;code&gt;bigint&lt;/code&gt; keys holds a few hundred entries once you account for per-entry overhead (the row pointer, the line pointer) and the free space left by the fillfactor. Internal pages fan out by a similar order of magnitude. That fanout is the whole trick. Depending on key size, page occupancy and deduplication, an index with tens of millions of entries may be three or four levels deep, and one with billions may need another level. Depth grows logarithmically with row count, not linearly.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Why this matters in production:&lt;/strong&gt; an index traversal touches only a handful of pages, and the upper levels are touched by every query, so they tend to stay cached. Fetching the matching heap row can still cost extra I/O, but the search itself stays cheap even on enormous tables. This is why "just add an index" works so well for so long.&lt;/p&gt;

&lt;h2&gt;
  
  
  Read: Walk Down, Then Walk Right
&lt;/h2&gt;

&lt;p&gt;A point lookup like &lt;code&gt;WHERE tx_version = 912345678&lt;/code&gt; works like finding a word in a dictionary:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Read the root. Binary search its signposts to pick a child.&lt;/li&gt;
&lt;li&gt;Repeat on each internal page until you reach a leaf.&lt;/li&gt;
&lt;li&gt;Binary search the leaf for the key. It gives you a TID.&lt;/li&gt;
&lt;li&gt;Fetch the row from the table (the heap) at that TID.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Step 4 matters more than people expect. In Postgres, the index does not know whether a row is visible to your transaction. That information lives in the heap. So a normal index scan pays one extra page read per matching row to check visibility. The exception is an &lt;strong&gt;index-only scan&lt;/strong&gt; : if every column the query needs is in the index, and the visibility map says the heap page is all-visible, Postgres skips the heap entirely.&lt;/p&gt;

&lt;p&gt;Range queries are where B-trees really shine. &lt;code&gt;WHERE sender = $1 ORDER BY tx_version DESC LIMIT 50&lt;/code&gt; descends once to the right spot, then walks sideways along the linked leaf pages, reading entries that are already in the order you asked for. No sort step. No scanning unrelated rows.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Why this matters in production:&lt;/strong&gt;"latest 50 transactions for this wallet" is one descent plus one or two leaf pages, whether the table has a million rows or ten billion. That query shape is the B-tree's home ground, and a wallet explorer is mostly made of it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Insert: Find the Page, Make Room, Sometimes Split
&lt;/h2&gt;

&lt;p&gt;An insert starts as a read. Postgres descends to the leaf where the new key belongs, then:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;If the leaf has free space&lt;/strong&gt; , the entry goes in, in sorted position. Done.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;If the leaf is full&lt;/strong&gt; , the page &lt;strong&gt;splits&lt;/strong&gt;. Postgres allocates a new page, moves roughly half the entries into it, links it into the leaf chain, and inserts a new signpost into the parent. If the parent is also full, the parent splits too. If the split climbs all the way up, the root splits and the tree gets one level taller.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For a unique index, there is one extra step: before inserting, Postgres checks the leaf for an existing live entry with the same key. That is how &lt;code&gt;PRIMARY KEY&lt;/code&gt; and &lt;code&gt;ON CONFLICT&lt;/code&gt; work.&lt;/p&gt;

&lt;p&gt;Where the new key lands decides everything:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Increasing keys&lt;/strong&gt; (block height, transaction version, a sequence) always land on the rightmost leaf. Postgres has a fast path for this: it remembers the rightmost leaf and skips the descent when it can. And when the rightmost page splits, it leaves the old page packed full (to the index's fillfactor, 90% by default) instead of half empty, because nothing will ever be inserted to its left again.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Random keys&lt;/strong&gt; (transaction hashes, UUIDv4, wallet addresses) land on a random leaf every time. Insert activity is spread across the whole index instead of one edge, so the working set is much larger, and splits tend to leave pages around half full.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;I covered the random-key case with numbers in &lt;a href="https://insight.jatinjainsaraf.com/your-uuid-primary-keys-are-quietly-taxing-every-insert" rel="noopener noreferrer"&gt;the UUID post&lt;/a&gt;. The short version: the same number of rows can produce a bigger index, and as its working set grows beyond what the cache holds, inserts are more likely to need extra page reads and writes.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Why this matters in production:&lt;/strong&gt; an indexer inserts millions of rows per day into tables keyed by both kinds of value. The &lt;code&gt;tx_version&lt;/code&gt; index stays compact and its hot edge stays cached. The &lt;code&gt;tx_hash&lt;/code&gt; index on the same table touches pages all over the index, and at scale it is a likely candidate for the slowest part of the ingest path.&lt;/p&gt;

&lt;h2&gt;
  
  
  Update: There Is No Such Thing as Updating an Index Entry
&lt;/h2&gt;

&lt;p&gt;This is the part most engineers get wrong, and it is the most important one for an indexer.&lt;/p&gt;

&lt;p&gt;In Postgres, &lt;code&gt;UPDATE&lt;/code&gt; does not modify a row in place. Because of MVCC, it writes a &lt;strong&gt;new version&lt;/strong&gt; of the row somewhere in the heap and marks the old one as expired. The new version has a new physical location. So every index on the table now has an entry pointing at the old location and nothing pointing at the new one.&lt;/p&gt;

&lt;p&gt;The default consequence: &lt;strong&gt;an UPDATE that can't use HOT (below) generally needs new index entries for the new row version in every index on the table&lt;/strong&gt;, including indexes whose values did not change. Updating only &lt;code&gt;amount&lt;/code&gt; on a balances table with five indexes can therefore mean index maintenance across all five. That adds CPU, WAL and write overhead on every update. Depending on how full the target pages are and what cleanup is possible, some of those insertions can also trigger page splits.&lt;/p&gt;

&lt;p&gt;There is one escape hatch, and it is worth designing around: the &lt;strong&gt;HOT update&lt;/strong&gt; (Heap-Only Tuple). If the update changes no column that any index depends on, and the new row version fits on the same heap page as the old one, Postgres chains the new version off the old one and touches no indexes at all. The existing index entries still lead to the right row by following the chain.&lt;/p&gt;

&lt;p&gt;"Depends on" is broader than it sounds. It includes columns used inside expression indexes (&lt;code&gt;lower(name)&lt;/code&gt;) and columns in a partial index's &lt;code&gt;WHERE&lt;/code&gt; clause, not just columns listed directly in an index. (Since Postgres 16, summarizing indexes like BRIN are the exception: they no longer block HOT.)&lt;/p&gt;

&lt;p&gt;Two conditions, both under your control:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Don't index columns that change often.&lt;/strong&gt; One index on &lt;code&gt;amount&lt;/code&gt; disables HOT for every balance change.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Leave free space on heap pages&lt;/strong&gt; with a table fillfactor below 100, so the new version has room on the same page.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Postgres 14 added bottom-up index deletion, which removes the obsolete version entries that non-HOT updates leave in indexes whose values didn't change, often before they force a page split. It reduces the bloat. It does not make the insertions free.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;An index is not free just because no query uses it during an update.&lt;/strong&gt; Its mere presence decides whether Postgres can use HOT and skip touching indexes altogether.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Note also that changing an indexed key never edits the entry. The old entry stays until vacuum removes it, and a new one is inserted where the new value sorts.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Why this matters in production:&lt;/strong&gt; a "current balances" table updated on every transfer is the most write-heavy table in an indexer. Whether those updates are HOT is often the difference between the balance updater keeping up with the chain or not.&lt;/p&gt;

&lt;h2&gt;
  
  
  Delete: Mark Now, Clean Later, Never Shrink
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;DELETE&lt;/code&gt; in Postgres removes nothing immediately. It marks the heap row as expired. The index entries stay exactly where they are, still pointing at the dead row.&lt;/p&gt;

&lt;p&gt;Cleanup happens in layers:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Kill bits.&lt;/strong&gt; When an index scan follows an entry to a row that is dead to everyone, it marks that index entry as dead so future scans skip it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Opportunistic cleanup.&lt;/strong&gt; When an insert finds a leaf full, Postgres first tries removing dead entries from that page before splitting it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;VACUUM.&lt;/strong&gt; The real cleanup. It scans the indexes, removes entries pointing to dead rows, and only then lets the heap space be reused.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;And here is the part that surprises people: &lt;strong&gt;Postgres does not merge partially filled B-tree pages&lt;/strong&gt; just because their occupancy drops. Vacuum removes dead entries and can reclaim pages that become completely empty, but a page that is still 40% full stays a page. Delete a large fraction of rows spread across the key range, and the index will not shrink in proportion. The freed space is reusable for future inserts into those key ranges, but it is not given back.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Why this matters in production:&lt;/strong&gt; pruning old events with &lt;code&gt;DELETE ... WHERE tx_version &amp;lt; X&lt;/code&gt; leaves the table and indexes the same size, generates a burst of WAL, and creates vacuum work. If you plan to drop data by age or range, partition by that range and &lt;code&gt;DROP&lt;/code&gt; or &lt;code&gt;DETACH&lt;/code&gt; the old partition. That removes the table and all of its indexes as files, with no dead rows to clean up.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where B-Trees Hurt
&lt;/h2&gt;

&lt;p&gt;The four operations above give you the failure list directly:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Situation&lt;/th&gt;
&lt;th&gt;Why the B-tree struggles&lt;/th&gt;
&lt;th&gt;Better tool&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Many indexes on a write-heavy table&lt;/td&gt;
&lt;td&gt;Every insert, and every non-HOT update, writes to all of them&lt;/td&gt;
&lt;td&gt;Fewer, composite indexes; drop unused ones&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Frequently updated column is indexed&lt;/td&gt;
&lt;td&gt;Kills HOT, every update touches every index&lt;/td&gt;
&lt;td&gt;Don't index it; serve that query from another table&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Random keys (hashes, UUIDv4) at scale&lt;/td&gt;
&lt;td&gt;Inserts scatter across the whole index, large working set&lt;/td&gt;
&lt;td&gt;Time-ordered keys where you control the key; benchmark a hash index for equality-only lookups&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Huge append-only table, queried by time range&lt;/td&gt;
&lt;td&gt;B-tree stores one entry per row, much larger than needed&lt;/td&gt;
&lt;td&gt;BRIN&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Low-selectivity column (&lt;code&gt;success&lt;/code&gt;, &lt;code&gt;is_spam&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;Matches too many rows to beat a sequential scan&lt;/td&gt;
&lt;td&gt;No index, or a partial index on the rare value&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Containment queries (JSONB, arrays, attributes)&lt;/td&gt;
&lt;td&gt;B-tree can only compare whole values&lt;/td&gt;
&lt;td&gt;GIN&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Mass deletes by age or range&lt;/td&gt;
&lt;td&gt;No page merging, bloat stays&lt;/td&gt;
&lt;td&gt;Partitioning, drop partitions&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Very wide keys (long text)&lt;/td&gt;
&lt;td&gt;Fewer entries per page, taller tree; entries over ~2.7 KB are rejected&lt;/td&gt;
&lt;td&gt;Hash the key into a fixed-size column&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Firehose ingest where writes dominate reads&lt;/td&gt;
&lt;td&gt;Every write pays sort cost up front&lt;/td&gt;
&lt;td&gt;LSM-tree stores (RocksDB, Cassandra, ScyllaDB)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Analytics over the whole history&lt;/td&gt;
&lt;td&gt;Row-by-row index access is the wrong shape&lt;/td&gt;
&lt;td&gt;Columnar stores (ClickHouse) or rollup tables&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;A note on the alternatives, because each has its own cost:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;BRIN&lt;/strong&gt; stores only the min and max value per block of table pages (128 pages by default). When the column's values follow the physical order of the table, it can be thousands of times smaller than a B-tree. When they don't, it is nearly useless. Check the actual correlation (&lt;code&gt;pg_stats.correlation&lt;/code&gt;) before relying on it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Hash indexes&lt;/strong&gt; store a 4-byte hash code per row instead of the key, which can make them smaller than a B-tree on a wide key. But in Postgres they are equality-only, single-column, can't serve ordered or range scans, and &lt;strong&gt;cannot enforce uniqueness&lt;/strong&gt;. Whether they are actually faster for your lookups is something to benchmark, not assume.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;GIN&lt;/strong&gt; handles "does this JSONB contain X." Its write cost is higher than a B-tree's, and it buffers new entries in a pending list that gets merged later.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;LSM-trees&lt;/strong&gt; make writes cheap by deferring the sort to background compaction, and pay for it on reads and in compaction I/O. The &lt;a href="https://insight.jatinjainsaraf.com/why-postgres-and-cassandra-made-opposite-bets-on-storage-engines" rel="noopener noreferrer"&gt;Cassandra post&lt;/a&gt; covers that trade in depth. Aptos nodes use RocksDB, an LSM engine, as part of their storage layer. An indexer serves a different job: turning chain data into relational records that support flexible queries. The two have different requirements, so their storage choices aren't interchangeable.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The Indexer, Table by Table
&lt;/h2&gt;

&lt;p&gt;Here is how this plays out for a Move-chain indexer (Supra, Aptos), with the queries an explorer and wallet API actually serve. I'll assume addresses and hashes are stored as 32-byte &lt;code&gt;bytea&lt;/code&gt;, not 66-character hex text. That choice alone roughly halves the key size of every index that contains one, as covered in &lt;a href="https://insight.jatinjainsaraf.com/your-blockchain-indexer-is-fine-until-it-isnt" rel="noopener noreferrer"&gt;Your Blockchain Indexer Is Fine Until It Isn't&lt;/a&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  Blocks
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;blocks&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="n"&gt;height&lt;/span&gt; &lt;span class="nb"&gt;bigint&lt;/span&gt; &lt;span class="k"&gt;PRIMARY&lt;/span&gt; &lt;span class="k"&gt;KEY&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;-- B-tree, append-only, rightmost inserts&lt;/span&gt;
  &lt;span class="n"&gt;block_hash&lt;/span&gt; &lt;span class="n"&gt;bytea&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;first_version&lt;/span&gt; &lt;span class="nb"&gt;bigint&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;last_version&lt;/span&gt; &lt;span class="nb"&gt;bigint&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;block_time&lt;/span&gt; &lt;span class="n"&gt;timestamptz&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;
&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;UNIQUE&lt;/span&gt; &lt;span class="k"&gt;INDEX&lt;/span&gt; &lt;span class="n"&gt;blocks_hash_uidx&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;blocks&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;block_hash&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="c1"&gt;-- or, if uniqueness is enforced outside the database:&lt;/span&gt;
&lt;span class="c1"&gt;-- CREATE INDEX blocks_hash_idx ON blocks USING hash (block_hash);&lt;/span&gt;
&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;INDEX&lt;/span&gt; &lt;span class="n"&gt;blocks_time_brin&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;blocks&lt;/span&gt; &lt;span class="k"&gt;USING&lt;/span&gt; &lt;span class="n"&gt;brin&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;block_time&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;height&lt;/code&gt;: B-tree.&lt;/strong&gt; Perfect case. Monotonic, never updated, "latest blocks" is a backward walk from the right edge.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;block_hash&lt;/code&gt;: hash index or unique B-tree, depending on what the database must guarantee.&lt;/strong&gt; It is only ever looked up by equality. Note that the &lt;code&gt;height&lt;/code&gt; primary key does &lt;strong&gt;not&lt;/strong&gt; make &lt;code&gt;block_hash&lt;/code&gt; unique. If you want the database to enforce unique block hashes, use &lt;code&gt;CREATE UNIQUE INDEX ... ON blocks (block_hash)&lt;/code&gt;, which has to be a B-tree. If you have decided to trust the chain and your ingestion code for that invariant, a non-unique hash index is worth benchmarking.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;block_time&lt;/code&gt;: BRIN.&lt;/strong&gt; Blocks are inserted in time order, so the physical correlation should be near perfect (verify it in &lt;code&gt;pg_stats&lt;/code&gt;). "Blocks between 10:00 and 11:00" needs nothing more.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Honestly, the blocks table is small enough that the index choices barely matter. They matter on the next one.&lt;/p&gt;

&lt;h3&gt;
  
  
  Transactions
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;transactions&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="k"&gt;version&lt;/span&gt; &lt;span class="nb"&gt;bigint&lt;/span&gt; &lt;span class="k"&gt;PRIMARY&lt;/span&gt; &lt;span class="k"&gt;KEY&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;-- global, increasing&lt;/span&gt;
  &lt;span class="n"&gt;tx_hash&lt;/span&gt; &lt;span class="n"&gt;bytea&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;sender&lt;/span&gt; &lt;span class="n"&gt;bytea&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;block_height&lt;/span&gt; &lt;span class="nb"&gt;bigint&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;success&lt;/span&gt; &lt;span class="nb"&gt;boolean&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;gas_used&lt;/span&gt; &lt;span class="nb"&gt;bigint&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;payload&lt;/span&gt; &lt;span class="n"&gt;jsonb&lt;/span&gt;
&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;UNIQUE&lt;/span&gt; &lt;span class="k"&gt;INDEX&lt;/span&gt; &lt;span class="n"&gt;tx_hash_uidx&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;transactions&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tx_hash&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="c1"&gt;-- or, if uniqueness is enforced outside the database (benchmark first):&lt;/span&gt;
&lt;span class="c1"&gt;-- CREATE INDEX tx_hash_idx ON transactions USING hash (tx_hash);&lt;/span&gt;
&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;INDEX&lt;/span&gt; &lt;span class="n"&gt;tx_sender_ver_idx&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;transactions&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sender&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;version&lt;/span&gt; &lt;span class="k"&gt;DESC&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;INDEX&lt;/span&gt; &lt;span class="n"&gt;tx_block_brin&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;transactions&lt;/span&gt; &lt;span class="k"&gt;USING&lt;/span&gt; &lt;span class="n"&gt;brin&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;block_height&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;version&lt;/code&gt;: B-tree.&lt;/strong&gt; Same as block height. It is also your idempotency key: re-ingesting a batch uses &lt;code&gt;ON CONFLICT (version) DO NOTHING&lt;/code&gt;, so replays don't create duplicate rows. (In the &lt;a href="https://insight.jatinjainsaraf.com/postgres-jsonb-toast-tax-blockchain-indexer" rel="noopener noreferrer"&gt;JSONB TOAST post&lt;/a&gt; I keyed &lt;code&gt;transactions&lt;/code&gt; by &lt;code&gt;tx_hash&lt;/code&gt; to keep the example focused on JSONB. For a real indexer, the increasing version is the better primary key, for exactly the insert reasons above.)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;tx_hash&lt;/code&gt;: a design decision, not a default.&lt;/strong&gt; Version uniqueness and hash uniqueness protect different things. The &lt;code&gt;version&lt;/code&gt; key stops the same transaction version from being stored twice. It does not stop two rows from carrying the same hash. If "one hash, one row" must be enforced by the database, keep &lt;code&gt;CREATE UNIQUE INDEX ... ON transactions (tx_hash)&lt;/code&gt;, which has to be a B-tree. If you trust the chain and ingestion for that invariant and the column only serves "look up this transaction," a hash index is a candidate: hashes are uniformly random, so a B-tree on them gets the random-insert treatment described above, on your largest table, while the hash index stores a 4-byte hash value in its entries (plus tuple and page overhead). That does not guarantee it ends up smaller or faster overall, since B-trees have their own advantages such as deduplication. Benchmark both on your own data before switching, and make sure the API handles a lookup returning more than one row instead of assuming it can't.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;(sender, version DESC)&lt;/code&gt;: composite B-tree.&lt;/strong&gt; This is wallet transaction history, and it is the textbook B-tree query: one descent to the sender, then walk the leaves in version order. Don't create a separate index on &lt;code&gt;sender&lt;/code&gt; alone. The composite one already serves it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;block_height&lt;/code&gt;: BRIN&lt;/strong&gt; , since version and height rise together and rows are inserted in that order (again, verify the correlation, especially after backfills).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;success&lt;/code&gt;: no index.&lt;/strong&gt; Almost every row is &lt;code&gt;true&lt;/code&gt;. If you need failed transactions, &lt;code&gt;CREATE INDEX ... WHERE NOT success&lt;/code&gt; indexes only the rare ones.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;payload&lt;/code&gt;: no index by default.&lt;/strong&gt; If you need to filter by entry function, extract it into its own column at ingest and B-tree that, rather than GIN-indexing the whole payload.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Coin and Fungible Asset Activity (transfers)
&lt;/h3&gt;

&lt;p&gt;Every deposit and withdrawal for both legacy &lt;code&gt;0x1::coin&lt;/code&gt; coins and &lt;code&gt;0x1::fungible_asset&lt;/code&gt; FAs, one row per event. Append-only, never updated, and the biggest table in the database.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;fa_activities&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="k"&gt;version&lt;/span&gt; &lt;span class="nb"&gt;bigint&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;event_index&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="k"&gt;owner&lt;/span&gt; &lt;span class="n"&gt;bytea&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;asset_type&lt;/span&gt; &lt;span class="n"&gt;bytea&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;-- coin type hashed, or FA metadata address&lt;/span&gt;
  &lt;span class="n"&gt;amount&lt;/span&gt; &lt;span class="nb"&gt;numeric&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;kind&lt;/span&gt; &lt;span class="nb"&gt;smallint&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;-- deposit / withdraw / mint / burn&lt;/span&gt;
  &lt;span class="k"&gt;PRIMARY&lt;/span&gt; &lt;span class="k"&gt;KEY&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;version&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;event_index&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;INDEX&lt;/span&gt; &lt;span class="n"&gt;fa_owner_ver_idx&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;fa_activities&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;owner&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;version&lt;/span&gt; &lt;span class="k"&gt;DESC&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;INDEX&lt;/span&gt; &lt;span class="n"&gt;fa_asset_ver_idx&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;fa_activities&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;asset_type&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;version&lt;/span&gt; &lt;span class="k"&gt;DESC&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Wallet transfer history&lt;/strong&gt; : &lt;code&gt;(owner, version DESC)&lt;/code&gt;. B-tree.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Coin transaction history&lt;/strong&gt; ("all USDC transfers, newest first"): &lt;code&gt;(asset_type, version DESC)&lt;/code&gt;. B-tree.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Match the index to your pagination cursor.&lt;/strong&gt; One transaction can emit several events for the same wallet, so a cursor of &lt;code&gt;version&lt;/code&gt; alone is ambiguous. If the API paginates on &lt;code&gt;(version, event_index)&lt;/code&gt;, use &lt;code&gt;(owner, version DESC, event_index DESC)&lt;/code&gt; and &lt;code&gt;WHERE owner = $1 AND (version, event_index) &amp;lt; ($2, $3)&lt;/code&gt; so the index serves both the filter and the order.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Coin type strings&lt;/strong&gt; like &lt;code&gt;0x1::supra_coin::SupraCoin&lt;/code&gt; can be long, and legacy coin types can be very long once generics are involved. Hash them into a fixed 32-byte key with a lookup table, instead of putting a variable-length string into two of your biggest indexes.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;No index on &lt;code&gt;amount&lt;/code&gt;&lt;/strong&gt;. "Largest transfers" is an analytics query. Serve it from a rollup or a columnar store.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Partition by version range&lt;/strong&gt; if you will ever prune or archive. This table is where a &lt;code&gt;DELETE&lt;/code&gt;-based retention job would hurt the most.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Wallet Portfolio (current balances)
&lt;/h3&gt;

&lt;p&gt;This is the table where B-tree update behavior decides whether you keep up with the chain.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;current_balances&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="k"&gt;owner&lt;/span&gt; &lt;span class="n"&gt;bytea&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;asset_type&lt;/span&gt; &lt;span class="n"&gt;bytea&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;amount&lt;/span&gt; &lt;span class="nb"&gt;numeric&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;last_version&lt;/span&gt; &lt;span class="nb"&gt;bigint&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="k"&gt;PRIMARY&lt;/span&gt; &lt;span class="k"&gt;KEY&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;owner&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;asset_type&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;WITH&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fillfactor&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;80&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;(owner, asset_type)&lt;/code&gt; primary key: B-tree.&lt;/strong&gt;"Show this wallet's portfolio" is one descent and a short leaf walk.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Nothing else.&lt;/strong&gt; &lt;code&gt;amount&lt;/code&gt; and &lt;code&gt;last_version&lt;/code&gt; change on every transfer. As long as neither is indexed (directly, in an expression, or in a partial index predicate), and the fillfactor leaves room on each heap page, most balance updates can be HOT and touch zero indexes. Treat 80 as a starting point to measure, not a magic number: watch &lt;code&gt;n_tup_hot_upd&lt;/code&gt; against &lt;code&gt;n_tup_upd&lt;/code&gt; in &lt;code&gt;pg_stat_user_tables&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The trap&lt;/strong&gt; : someone adds &lt;code&gt;(asset_type, amount DESC)&lt;/code&gt; to serve a "top holders" page. From that moment, every balance change is a non-HOT update that inserts into both indexes. On a busy token, this is the table that starts lagging first.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Top holders&lt;/strong&gt; belongs in a separate table refreshed every few minutes, or computed in your analytics store. It is a ranking, and rankings don't need to be correct to the transaction.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Historical Balances (balance at a point in time)
&lt;/h3&gt;

&lt;p&gt;If you support "what did this wallet hold at version X," keep an append-only table:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;balance_history&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="k"&gt;owner&lt;/span&gt; &lt;span class="n"&gt;bytea&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;asset_type&lt;/span&gt; &lt;span class="n"&gt;bytea&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="k"&gt;version&lt;/span&gt; &lt;span class="nb"&gt;bigint&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;amount&lt;/span&gt; &lt;span class="nb"&gt;numeric&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="k"&gt;PRIMARY&lt;/span&gt; &lt;span class="k"&gt;KEY&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;owner&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;asset_type&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;version&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;amount&lt;/span&gt; &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;balance_history&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="k"&gt;owner&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="n"&gt;asset_type&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt; &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="k"&gt;version&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;
&lt;span class="k"&gt;ORDER&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="k"&gt;version&lt;/span&gt; &lt;span class="k"&gt;DESC&lt;/span&gt;
&lt;span class="k"&gt;LIMIT&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This is a B-tree at its best. With &lt;code&gt;owner&lt;/code&gt; and &lt;code&gt;asset_type&lt;/code&gt; fixed by equality, Postgres can use the composite index to find the latest version not greater than X and return that row without scanning the wallet's history. No other structure answers "the last value before this point" as cheaply.&lt;/p&gt;

&lt;h3&gt;
  
  
  NFTs and Digital Assets
&lt;/h3&gt;

&lt;p&gt;Move chains have two NFT standards, and they want different key designs (the &lt;a href="https://insight.jatinjainsaraf.com/the-complete-developers-guide-to-indexing-assets-on-supra-aptos-coins-fas-nfts" rel="noopener noreferrer"&gt;Coins, FAs and NFTs guide&lt;/a&gt; covers the data model itself):&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Legacy &lt;code&gt;0x3&lt;/code&gt; tokens&lt;/strong&gt; are identified by creator, collection name, token name and property version. That is a wide, variable-length text key. Hash it into a 32-byte &lt;code&gt;token_id&lt;/code&gt; at ingest and index that.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;0x4&lt;/code&gt; Digital Assets&lt;/strong&gt; are objects, so the token already has a 32-byte address. Use it directly.
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="c1"&gt;-- current owner of each token: churns on every transfer&lt;/span&gt;
&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;current_token_ownership&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="n"&gt;token_id&lt;/span&gt; &lt;span class="n"&gt;bytea&lt;/span&gt; &lt;span class="k"&gt;PRIMARY&lt;/span&gt; &lt;span class="k"&gt;KEY&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="k"&gt;owner&lt;/span&gt; &lt;span class="n"&gt;bytea&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;amount&lt;/span&gt; &lt;span class="nb"&gt;numeric&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;last_version&lt;/span&gt; &lt;span class="nb"&gt;bigint&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;WITH&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;fillfactor&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;80&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;INDEX&lt;/span&gt; &lt;span class="n"&gt;cto_owner_idx&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;current_token_ownership&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;owner&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="c1"&gt;-- every movement, append-only&lt;/span&gt;
&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;token_activities&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="k"&gt;version&lt;/span&gt; &lt;span class="nb"&gt;bigint&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;event_index&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;token_id&lt;/span&gt; &lt;span class="n"&gt;bytea&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;from_addr&lt;/span&gt; &lt;span class="n"&gt;bytea&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;to_addr&lt;/span&gt; &lt;span class="n"&gt;bytea&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="k"&gt;PRIMARY&lt;/span&gt; &lt;span class="k"&gt;KEY&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;version&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;event_index&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;INDEX&lt;/span&gt; &lt;span class="n"&gt;ta_token_ver_idx&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;token_activities&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;token_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;version&lt;/span&gt; &lt;span class="k"&gt;DESC&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="c1"&gt;-- metadata&lt;/span&gt;
&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;token_data&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="n"&gt;token_id&lt;/span&gt; &lt;span class="n"&gt;bytea&lt;/span&gt; &lt;span class="k"&gt;PRIMARY&lt;/span&gt; &lt;span class="k"&gt;KEY&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;collection_id&lt;/span&gt; &lt;span class="n"&gt;bytea&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;name&lt;/span&gt; &lt;span class="nb"&gt;text&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;attributes&lt;/span&gt; &lt;span class="n"&gt;jsonb&lt;/span&gt;
&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;INDEX&lt;/span&gt; &lt;span class="n"&gt;td_collection_name_idx&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;token_data&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;collection_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;INDEX&lt;/span&gt; &lt;span class="n"&gt;td_attributes_gin&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;token_data&lt;/span&gt; &lt;span class="k"&gt;USING&lt;/span&gt; &lt;span class="n"&gt;gin&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;attributes&lt;/span&gt; &lt;span class="n"&gt;jsonb_path_ops&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;"NFTs in this wallet"&lt;/strong&gt; : B-tree on &lt;code&gt;current_token_ownership (owner)&lt;/code&gt;. Here we do index a changing column, deliberately, because it is the query the wallet page depends on. Accept that ownership changes are non-HOT, and keep this the only secondary index on the table.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;NFT movement history&lt;/strong&gt; : &lt;code&gt;(token_id, version DESC)&lt;/code&gt; on the append-only activity table. B-tree.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Collection listing sorted by name&lt;/strong&gt; : &lt;code&gt;(collection_id, name)&lt;/code&gt;. B-tree.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;"Tokens in this collection with Background = Gold"&lt;/strong&gt; : that is a containment query, &lt;code&gt;attributes @&amp;gt; '{"Background": "Gold"}'&lt;/code&gt;. GIN with &lt;code&gt;jsonb_path_ops&lt;/code&gt; is usually smaller than the default &lt;code&gt;jsonb_ops&lt;/code&gt; and supports &lt;code&gt;@&amp;gt;&lt;/code&gt; (plus the jsonpath operators &lt;code&gt;@?&lt;/code&gt; and &lt;code&gt;@@&lt;/code&gt;), but not the key-existence operators &lt;code&gt;?&lt;/code&gt;, &lt;code&gt;?|&lt;/code&gt; and &lt;code&gt;?&amp;amp;&lt;/code&gt;. If your API asks "has this attribute at all," you need &lt;code&gt;jsonb_ops&lt;/code&gt;. A B-tree cannot help with either.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Analytics: Daily Volume, Unique Wallets, Holder Counts
&lt;/h3&gt;

&lt;p&gt;Don't add B-trees to the OLTP tables for these. Questions that read most of the history are not what a B-tree is for, and every index you add to serve a dashboard is paid for on every ingest. Push activity into a columnar store, or maintain rollup tables updated per block. For distinct counts specifically, a HyperLogLog sketch estimates the number of unique wallets in a few kilobytes, with a tunable accuracy-versus-space trade-off. That is fine for dashboards, and very different from the exact balances an explorer must serve. The technique is discussed in my &lt;a href="https://insight.jatinjainsaraf.com/case-study/exact-analytics-in-12kb-hardening-a-wallet-uniqueness-system" rel="noopener noreferrer"&gt;wallet uniqueness system case study&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Cheat Sheet
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Table / query&lt;/th&gt;
&lt;th&gt;Index&lt;/th&gt;
&lt;th&gt;Why&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Blocks / transactions by height or version&lt;/td&gt;
&lt;td&gt;B-tree PK&lt;/td&gt;
&lt;td&gt;Monotonic, append-only, rightmost inserts&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Lookup by tx hash or block hash&lt;/td&gt;
&lt;td&gt;Unique B-tree if the DB must enforce it; otherwise benchmark a hash index&lt;/td&gt;
&lt;td&gt;Hash indexes can't be unique; height/version keys don't protect the hash&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Blocks or txs by time / height range&lt;/td&gt;
&lt;td&gt;BRIN (verify correlation)&lt;/td&gt;
&lt;td&gt;Rises with insert order, tiny index&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Wallet tx history&lt;/td&gt;
&lt;td&gt;B-tree &lt;code&gt;(sender, version DESC)&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;One descent, then a leaf walk in order&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Coin / FA transfer history&lt;/td&gt;
&lt;td&gt;B-tree &lt;code&gt;(asset_type, version DESC)&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Same shape, per asset&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Wallet portfolio&lt;/td&gt;
&lt;td&gt;B-tree PK &lt;code&gt;(owner, asset_type)&lt;/code&gt; only, fillfactor below 100 (start at 80, measure)&lt;/td&gt;
&lt;td&gt;Keep balance updates HOT&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Balance at version X&lt;/td&gt;
&lt;td&gt;B-tree &lt;code&gt;(owner, asset_type, version)&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;"Last value before X" is a single descent&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;NFTs owned by wallet&lt;/td&gt;
&lt;td&gt;B-tree on &lt;code&gt;owner&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Accept the non-HOT cost, it is the core query&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;NFT movement history&lt;/td&gt;
&lt;td&gt;B-tree &lt;code&gt;(token_id, version DESC)&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Append-only, ordered&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;NFT attribute filters&lt;/td&gt;
&lt;td&gt;GIN &lt;code&gt;jsonb_path_ops&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Containment, B-tree can't do it&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;success&lt;/code&gt;, &lt;code&gt;is_spam&lt;/code&gt;, other booleans&lt;/td&gt;
&lt;td&gt;None, or partial index on the rare value&lt;/td&gt;
&lt;td&gt;Low selectivity&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Top holders, volume, unique wallets&lt;/td&gt;
&lt;td&gt;No OLTP index; rollups or columnar&lt;/td&gt;
&lt;td&gt;Whole-history reads, would tax every write&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Pruning old data&lt;/td&gt;
&lt;td&gt;Partition by version range&lt;/td&gt;
&lt;td&gt;B-trees never shrink after deletes&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  The Takeaway
&lt;/h2&gt;

&lt;p&gt;A B-tree is a sorted dictionary that is cheap to search and expensive to keep sorted. Every read is a descent and a walk. Every insert might split a page. Every update in Postgres is an insert into every index, unless you keep it HOT. Every delete leaves a hole that never closes on its own.&lt;/p&gt;

&lt;p&gt;An indexer is a machine that turns blockchain events into those four operations at high volume. Design each table by asking which of the four it does most, and the index choices mostly make themselves: B-trees for ordered history and enforced uniqueness, hash indexes as a benchmarked option for random equality lookups, BRIN for time, GIN for containment, partitions for retention, and somewhere else entirely for analytics.&lt;/p&gt;

&lt;p&gt;If you want the full page-level detail on nbtree, HOT updates, and the other index types, &lt;a href="https://academy.jatinjainsaraf.com/postgresql-in-depth/indexes" rel="noopener noreferrer"&gt;Module A-5: Index Internals&lt;/a&gt; in the PostgreSQL In-Depth course goes deeper, and &lt;a href="https://academy.jatinjainsaraf.com/system-design-in-depth/storage-engines-btrees-vs-lsm-trees" rel="noopener noreferrer"&gt;Storage Engines: B-Trees vs LSM-Trees&lt;/a&gt; in the System Design course covers the other side of the trade.&lt;/p&gt;

</description>
      <category>postgres</category>
      <category>btree</category>
      <category>indexing</category>
      <category>databaseinternals</category>
    </item>
    <item>
      <title>Turning Your Blockchain Indexer Into a Paid API Without Slowing Down Your Explorer</title>
      <dc:creator>Jatin Jain Saraf</dc:creator>
      <pubDate>Thu, 08 Oct 2026 03:30:41 +0000</pubDate>
      <link>https://dev.to/jatinjainsaraf/turning-your-blockchain-indexer-into-a-paid-api-without-slowing-down-your-explorer-33fg</link>
      <guid>https://dev.to/jatinjainsaraf/turning-your-blockchain-indexer-into-a-paid-api-without-slowing-down-your-explorer-33fg</guid>
      <description>&lt;h1&gt;
  
  
  Turning Your Blockchain Indexer Into a Paid API Without Slowing Down Your Explorer
&lt;/h1&gt;

&lt;p&gt;Your indexer has done the hard part. Every block, transaction, event and balance change on the chain is decoded and stored in Postgres, with indexes that make your explorer fast.&lt;/p&gt;

&lt;p&gt;Sooner or later, someone asks the obvious question: &lt;em&gt;can we get this data through an API?&lt;/em&gt; Wallets want balances. Analytics teams want transaction history. Trading bots want events the moment they land. Some will pay for it.&lt;/p&gt;

&lt;p&gt;The first version is usually easy: point a few REST endpoints at the same database the explorer uses, hand out API keys, done.&lt;/p&gt;

&lt;p&gt;Then a single customer writes a loop that pages through every transaction for a busy contract address, ten requests at a time, all day. The explorer's p99 latency climbs. The indexer's writes start competing with those reads for disk and memory, and the indexer falls behind the chain. Now your explorer shows stale blocks, every API customer gets stale data, and none of it shows up as an error anywhere. (It's the same slow-burn pattern I described in &lt;a href="https://insight.jatinjainsaraf.com/your-blockchain-indexer-is-fine-until-it-isnt" rel="noopener noreferrer"&gt;Your Blockchain Indexer Is Fine Until It Isn't&lt;/a&gt;, this time triggered from outside.)&lt;/p&gt;

&lt;p&gt;Exposing indexer data is not an endpoint problem. It's an isolation problem. This post walks through a design that serves blockchain data to anyone, metered by credits and plans, at low latency, without letting that traffic touch the two things you can't afford to slow down: &lt;strong&gt;ingestion&lt;/strong&gt; and &lt;strong&gt;your explorer&lt;/strong&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Kitchen and the Counter
&lt;/h2&gt;

&lt;p&gt;A restaurant doesn't let customers walk into the kitchen and cook whatever they like. Customers order at the counter, from a menu, and every dish on the menu has a price that reflects how much work it takes.&lt;/p&gt;

&lt;p&gt;Map that onto your system:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The kitchen&lt;/strong&gt; is your primary database and indexer. It has one job: keep up with the chain. Nobody outside eats here.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The counter&lt;/strong&gt; is your API gateway. It checks who you are, what plan you're on, and whether you've got credits left. It takes orders. It doesn't cook. (If you're choosing what that gateway should be, I compared the options in &lt;a href="https://insight.jatinjainsaraf.com/demystifying-api-gateway-patterns-design-choices-real-world-use-cases-when-to-use-what" rel="noopener noreferrer"&gt;Demystifying API Gateway Patterns&lt;/a&gt;.)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The menu&lt;/strong&gt; is a fixed set of endpoints, served by a query service that knows how to cook each dish one way. No arbitrary queries. If it isn't on the menu, you can't order it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The prices&lt;/strong&gt; are credits. A balance lookup is cheap. A page of event history costs more. The price reflects what the request costs &lt;em&gt;you&lt;/em&gt;, not just that it happened.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Every section below is one part of that restaurant.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Architecture at a Glance
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;chain ──▶ [indexer] ──writes──▶ [primary DB]
                                     │ streaming replication
                  ┌──────────────────┼──────────────────────┐
                  ▼ ▼ ▼
        [explorer replicas] [paid API replica pool] [free API replica pool]
                  ▲ ▲ ▲
             [explorer] └──────┬───────────────┘
                                            │
                                   [API query service]
                                   endpoint + cursor validation,
                                   query building, per-plan pools,
                                   timeouts, retries, response shaping
                                            ▲
                                      [API gateway] ◀──── clients + API keys
                                   auth, rate limit, credits,
                                   cache, routing
                                            │
                                 [Redis: limits, credits, cache]

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Four rules hold this together:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;The primary is reserved for ingestion and tightly controlled internal workloads.&lt;/strong&gt; External API and explorer traffic never reads it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The explorer and the API never share a replica.&lt;/strong&gt; An API spike can't slow the explorer, and vice versa.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Every request passes the gateway first.&lt;/strong&gt; Auth, rate limit, credit check and cache happen before a single query runs.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The gateway never talks to the database.&lt;/strong&gt; A separate query service owns every query: it validates the endpoint and cursor, builds the query, picks the replica pool for the caller's plan, enforces timeouts and shapes the response. The gateway decides &lt;em&gt;whether&lt;/em&gt; a request runs. The query service decides &lt;em&gt;how&lt;/em&gt;.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;That last split keeps the gateway small, fast and stateless, and puts everything that touches SQL in one place you can reason about.&lt;/p&gt;

&lt;h2&gt;
  
  
  Layer 1: Isolate Reads on Dedicated Replica Pools
&lt;/h2&gt;

&lt;p&gt;Postgres streaming replication gives you read-only copies of the primary. Under healthy conditions they may be only milliseconds or seconds behind, but that lag depends on the workload: how much WAL the indexer generates, the network, and how busy the replica is replaying it while also serving queries. Give the API its own replica pools, separate from the explorer's.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Why this matters in production:&lt;/strong&gt; read traffic from the API now can't evict the explorer's hot pages from memory or compete with the indexer's writes for disk on the primary. A badly behaved customer can only hurt the replica pool for their plan.&lt;/p&gt;

&lt;p&gt;Two settings decide how that replica behaves under pressure, and they pull in opposite directions:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;hot_standby_feedback = on&lt;/code&gt;&lt;/strong&gt; makes the replica tell the primary which row versions its running queries still need. Your API queries stop getting cancelled, but long-running queries on the replica can now prevent vacuum on the primary from cleaning up those row versions. Dead rows pile up and the tables your indexer writes to start to bloat. That's exactly the coupling you were trying to remove.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;hot_standby_feedback = off&lt;/code&gt;&lt;/strong&gt; with a bounded &lt;code&gt;max_standby_streaming_delay&lt;/code&gt; keeps the primary clean. The cost: when replaying WAL conflicts with a query on the replica, Postgres waits up to that delay and then cancels the query with &lt;code&gt;canceling statement due to conflict with recovery&lt;/code&gt; (SQLSTATE &lt;code&gt;40001&lt;/code&gt;).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;For a public API, I'd generally prefer the second option, &lt;strong&gt;provided the query service is prepared to handle cancelled reads.&lt;/strong&gt; Catch that error, retry once on another replica in the same pool, and if that also fails, return a clean &lt;code&gt;503&lt;/code&gt; with &lt;code&gt;Retry-After&lt;/code&gt; instead of an unexplained &lt;code&gt;500&lt;/code&gt;. Pair it with short statement timeouts (Layer 5): a public API shouldn't be running queries long enough to conflict often in the first place. If it is, that's a menu problem, not a replica setting problem.&lt;/p&gt;

&lt;p&gt;Replicas also lag. A replica serving a heavy query load can fall behind the primary, and an API that silently serves old data is worse than one that admits it. I've written about &lt;a href="https://insight.jatinjainsaraf.com/taming-postgresql-replication-lag-in-real-time-blockchain-indexers" rel="noopener noreferrer"&gt;replication lag in blockchain indexers&lt;/a&gt; separately. For the API, the rule is simple: &lt;strong&gt;tell the client how fresh the data is.&lt;/strong&gt; Return the latest indexed block height with every response. (The &lt;a href="https://academy.jatinjainsaraf.com/postgresql-in-depth/replication-ha" rel="noopener noreferrer"&gt;Replication &amp;amp; High Availability module&lt;/a&gt; in PostgreSQL In-Depth covers how streaming replication, lag and standby conflicts work underneath.)&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;X-Indexed-Height: 18452231
X-Indexed-At: 2026-10-08T09:14:02Z

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Clients can then decide for themselves whether data that's two blocks old is fine for their use case.&lt;/p&gt;

&lt;h2&gt;
  
  
  Layer 2: Serve From Read Models, Not Raw Tables
&lt;/h2&gt;

&lt;p&gt;The tables your indexer writes are shaped for ingestion. The API needs tables shaped for the questions customers ask. The most common one by far is "show me the transactions for this address, newest first."&lt;/p&gt;

&lt;p&gt;Have the indexer maintain a read model built for exactly that question, as part of ingestion:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;address_transactions&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="n"&gt;address&lt;/span&gt; &lt;span class="n"&gt;bytea&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;ledger_version&lt;/span&gt; &lt;span class="nb"&gt;bigint&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;tx_hash&lt;/span&gt; &lt;span class="n"&gt;bytea&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="k"&gt;PRIMARY&lt;/span&gt; &lt;span class="k"&gt;KEY&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;address&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ledger_version&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;);&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Here &lt;code&gt;ledger_version&lt;/code&gt; assumes the chain gives every transaction its own unique, increasing version number, as Aptos-style ledgers do. Block height is &lt;strong&gt;not&lt;/strong&gt; a substitute: a block holds many transactions, so &lt;code&gt;(address, block_height)&lt;/code&gt; would collide as soon as an address appears twice in one block. If your chain has no per-transaction version, use a composite position such as &lt;code&gt;(block_height, tx_index)&lt;/code&gt; in the key instead. And since one transaction can touch the same address more than once (sender and receiver, or several events), have the indexer insert with &lt;code&gt;ON CONFLICT DO NOTHING&lt;/code&gt;, so repeated observations of the same &lt;code&gt;(address, ledger_version)&lt;/code&gt; pair collapse into one read-model row.&lt;/p&gt;

&lt;p&gt;The primary key index on &lt;code&gt;(address, ledger_version)&lt;/code&gt; keeps each address's history in index order, so one address's newest transactions are always a short index range scan, whether the address has 10 transactions or 10 million. The &lt;a href="https://academy.jatinjainsaraf.com/postgresql-in-depth/indexes" rel="noopener noreferrer"&gt;Indexes module&lt;/a&gt; in PostgreSQL In-Depth explains why composite key order decides which queries an index can serve.&lt;/p&gt;

&lt;p&gt;Then serve history with &lt;strong&gt;keyset pagination&lt;/strong&gt; , never &lt;code&gt;OFFSET&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;ledger_version&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tx_hash&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;address_transactions&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;address&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;
  &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="n"&gt;ledger_version&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt; &lt;span class="c1"&gt;-- the cursor from the previous page&lt;/span&gt;
&lt;span class="k"&gt;ORDER&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;ledger_version&lt;/span&gt; &lt;span class="k"&gt;DESC&lt;/span&gt;
&lt;span class="k"&gt;LIMIT&lt;/span&gt; &lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;OFFSET 50000&lt;/code&gt; makes Postgres read and throw away 50,000 rows before returning the page you asked for, so every page gets slower than the last. A cursor jumps straight to the right spot in the index, so page 1,000 costs the same as page 1. (The &lt;a href="https://insight.jatinjainsaraf.com/handling-large-datasets-high-traffic-queries-optimizing-pagination-sorting-filtering" rel="noopener noreferrer"&gt;pagination post&lt;/a&gt; goes deeper on this.) Return the cursor as an opaque string, so clients can't build their own and you can change the format later.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Rule for the whole menu:&lt;/strong&gt; every endpoint must map to an index-backed access path with a bounded result size. No free-form filters, no "sort by any column", no unbounded block ranges. An events endpoint takes a maximum window of blocks per call. If a customer needs more, they page.&lt;/p&gt;

&lt;h2&gt;
  
  
  Layer 3: Cache by Finality
&lt;/h2&gt;

&lt;p&gt;Blockchain data has a property most application data doesn't: &lt;strong&gt;once a block is considered final by the chain's consensus and finality rules, its canonical data can be treated as immutable for caching.&lt;/strong&gt; A finalized transaction from last year can be treated as immutable forever.&lt;/p&gt;

&lt;p&gt;What "final" means differs a lot between chains. Some finalize a block within seconds and never revert it. Others only become safe after a number of confirmations, and recent blocks can still be reorganized. Define finality per chain, and expose it: the indexer should track the latest finalized height separately from the latest indexed height, and the API should only treat data at or below the finalized height as immutable.&lt;/p&gt;

&lt;p&gt;That splits your caching strategy cleanly in two:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Data&lt;/th&gt;
&lt;th&gt;Changes?&lt;/th&gt;
&lt;th&gt;Cache policy&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Finalized blocks, finalized transaction-by-hash responses, historical events&lt;/td&gt;
&lt;td&gt;Never&lt;/td&gt;
&lt;td&gt;Usually safe to cache for a long time, at the CDN and in Redis&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Latest block, current balances, recent history above the finalized height&lt;/td&gt;
&lt;td&gt;Every block, and can reorg on some chains&lt;/td&gt;
&lt;td&gt;Short TTL, about one block time&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Pending, unconfirmed or not-found lookups&lt;/td&gt;
&lt;td&gt;Constantly&lt;/td&gt;
&lt;td&gt;Don't cache, or cache very briefly&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;That last row matters for transaction-by-hash: a lookup for a transaction that's still pending, or that hasn't been indexed yet, must not be cached as "not found" for an hour.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Why this matters in production:&lt;/strong&gt; most traffic on a blockchain API is for things that already happened. If finalized responses are served from a CDN or Redis, a large share of requests never reach a replica at all, and those cache hits are where the low latency comes from.&lt;/p&gt;

&lt;p&gt;Watch out for one failure mode. When a popular address's cached balance expires, a thousand clients can miss the cache at the same moment and send a thousand identical queries to the replica. Use request coalescing (one in-flight query per key, everyone else waits for its result). The &lt;a href="https://academy.jatinjainsaraf.com/redis-in-depth/cache-stampede-avalanche-penetration" rel="noopener noreferrer"&gt;cache stampede module&lt;/a&gt; in Redis In-Depth covers the patterns.&lt;/p&gt;

&lt;h2&gt;
  
  
  Layer 4: Plans, Credits and the Gateway
&lt;/h2&gt;

&lt;p&gt;Now the part that makes this a product: who can call what, how often, and how much.&lt;/p&gt;

&lt;h3&gt;
  
  
  Two limits, not one
&lt;/h3&gt;

&lt;p&gt;A plan needs two separate limits, because they protect against two different problems:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Rate limit (requests per second):&lt;/strong&gt; protects your &lt;em&gt;infrastructure&lt;/em&gt; from bursts. A customer with plenty of monthly credits still can't send 5,000 requests in one second.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Credit quota (per month):&lt;/strong&gt; protects your &lt;em&gt;business&lt;/em&gt;. It's what the customer is paying for.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Credits priced by cost
&lt;/h3&gt;

&lt;p&gt;Charging "one request = one credit" is unfair to you. A balance lookup and a 100-row event scan are not the same amount of work. Price each endpoint by what it actually costs to serve:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Endpoint&lt;/th&gt;
&lt;th&gt;Credits&lt;/th&gt;
&lt;th&gt;Why&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;GET /blocks/latest&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;One row, almost always cached&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;GET /tx/{hash}&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;Point lookup; finalized results are immutable and highly cacheable&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;GET /accounts/{addr}/balance&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;Point lookup, short cache life&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;GET /accounts/{addr}/transactions&lt;/code&gt; (up to 100)&lt;/td&gt;
&lt;td&gt;5&lt;/td&gt;
&lt;td&gt;Index range scan&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;GET /events?from=&amp;amp;to=&lt;/code&gt; (bounded window)&lt;/td&gt;
&lt;td&gt;20&lt;/td&gt;
&lt;td&gt;Larger scan, rarely cached&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;These numbers are illustrative. Derive your own from measured query costs, and keep the price &lt;strong&gt;fixed per endpoint and page size, decided before the query runs.&lt;/strong&gt; Customers can then predict their bill, and you never have to run a query just to find out what to charge for it.&lt;/p&gt;

&lt;h3&gt;
  
  
  Plans
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Free&lt;/th&gt;
&lt;th&gt;Pro&lt;/th&gt;
&lt;th&gt;Enterprise&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Requests per second&lt;/td&gt;
&lt;td&gt;5&lt;/td&gt;
&lt;td&gt;50&lt;/td&gt;
&lt;td&gt;Custom&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Monthly credits&lt;/td&gt;
&lt;td&gt;100K&lt;/td&gt;
&lt;td&gt;10M&lt;/td&gt;
&lt;td&gt;Custom&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Max page size&lt;/td&gt;
&lt;td&gt;25&lt;/td&gt;
&lt;td&gt;100&lt;/td&gt;
&lt;td&gt;100+&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;History depth&lt;/td&gt;
&lt;td&gt;Recent only&lt;/td&gt;
&lt;td&gt;Full&lt;/td&gt;
&lt;td&gt;Full&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Real-time streams&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Replica pool&lt;/td&gt;
&lt;td&gt;Shared free pool&lt;/td&gt;
&lt;td&gt;Paid pool&lt;/td&gt;
&lt;td&gt;Dedicated replica&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Again, these numbers are for illustration. The structure is what matters: plans differ in more than just credits. Page size, history depth and which replica pool you land on are all levers.&lt;/p&gt;

&lt;h3&gt;
  
  
  Checking limits without touching the database
&lt;/h3&gt;

&lt;p&gt;The gateway runs this check on every request, so it can't afford a database write per call. Keep both counters in Redis and make the check atomic with a Lua script, so that two simultaneous requests can't both spend the last credit:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight lua"&gt;&lt;code&gt;&lt;span class="c1"&gt;-- KEYS[1] = per-second window key, e.g. rl:{keyId}:{unixSecond}&lt;/span&gt;
&lt;span class="c1"&gt;-- KEYS[2] = billing-period credits key, e.g. cr:{keyId}:{yyyymm}&lt;/span&gt;
&lt;span class="c1"&gt;-- ARGV[1] = requests/sec limit, ARGV[2] = cost of this request&lt;/span&gt;
&lt;span class="c1"&gt;-- ARGV[3] = period quota, ARGV[4] = unix time the key may expire&lt;/span&gt;
&lt;span class="kd"&gt;local&lt;/span&gt; &lt;span class="n"&gt;reqs&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;redis&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'INCR'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;KEYS&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;reqs&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="k"&gt;then&lt;/span&gt; &lt;span class="n"&gt;redis&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'EXPIRE'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;KEYS&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;end&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;reqs&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;tonumber&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ARGV&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="k"&gt;then&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'rate_limited'&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="k"&gt;end&lt;/span&gt;

&lt;span class="kd"&gt;local&lt;/span&gt; &lt;span class="n"&gt;used&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;tonumber&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;redis&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'GET'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;KEYS&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="s1"&gt;'0'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;used&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="nb"&gt;tonumber&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ARGV&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;tonumber&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ARGV&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="k"&gt;then&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'quota_exhausted'&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="k"&gt;end&lt;/span&gt;

&lt;span class="n"&gt;used&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;redis&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'INCRBY'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;KEYS&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;ARGV&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;used&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="nb"&gt;tonumber&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ARGV&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="k"&gt;then&lt;/span&gt;
  &lt;span class="n"&gt;redis&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'EXPIREAT'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;KEYS&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;ARGV&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
&lt;span class="k"&gt;end&lt;/span&gt;
&lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;tonumber&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ARGV&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;used&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The whole script runs as one atomic step inside Redis, so there's no race between "check the balance" and "spend the credits." The &lt;a href="https://academy.jatinjainsaraf.com/redis-in-depth/atomic-counters-rate-limiters" rel="noopener noreferrer"&gt;atomic counters and rate limiters module&lt;/a&gt; explains why that matters.&lt;/p&gt;

&lt;p&gt;Some honest notes about this script, because each one is a product decision, not just an implementation detail:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Rejected requests count toward the rate window.&lt;/strong&gt; The counter is incremented before the limit check, so a client hammering at 10x its limit keeps its own window full, and a request that passes the rate check but fails the quota check has still used a rate slot. That's deliberate here: it makes ignoring 429s pointless. If your product should count only admitted requests, decrement on rejection or use a different limiter design.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The credits key expires at an absolute time, not after a duration.&lt;/strong&gt; The gateway passes the end of the billing period &lt;em&gt;plus&lt;/em&gt; your reconciliation grace window (say, a few days) to &lt;code&gt;EXPIREAT&lt;/code&gt;. A relative TTL set on the first request of the month would expire at a different moment for every customer, possibly before billing has read the final count.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;It's a fixed window&lt;/strong&gt; per second, so a client can fit up to twice its limit into the instant where one second ends and the next begins. If that matters for your infrastructure, use a token bucket instead. The &lt;a href="https://academy.jatinjainsaraf.com/system-design-in-depth/rate-limiting-and-quotas" rel="noopener noreferrer"&gt;rate limiting and quotas module&lt;/a&gt; compares the algorithms.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Redis admits, Postgres bills
&lt;/h3&gt;

&lt;p&gt;Credits represent money, so be precise about who owns what. &lt;strong&gt;Redis is the authoritative low-latency counter for admission:&lt;/strong&gt; it decides, in microseconds, whether this request runs. &lt;strong&gt;The durable billing system in Postgres remains the source of truth&lt;/strong&gt; for what a customer used and owes.&lt;/p&gt;

&lt;p&gt;Every minute or so, a background job reads the Redis counters and records usage in a Postgres billing table. Make that sync idempotent (keyed by API key and time window), so a retried sync never double-charges. Same idea as &lt;a href="https://insight.jatinjainsaraf.com/idempotency-keys-how-to-make-retries-safe-in-distributed-systems" rel="noopener noreferrer"&gt;idempotency keys&lt;/a&gt;, applied to your own billing.&lt;/p&gt;

&lt;p&gt;Then plan for the obvious question: &lt;strong&gt;what happens if Redis loses the counters?&lt;/strong&gt; A failover or restart without persistence resets usage to zero, and every customer suddenly has a full quota. Two safeguards:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;When a credits key is missing mid-period, seed it from the last usage recorded in Postgres before admitting the request, rather than starting from zero.&lt;/li&gt;
&lt;li&gt;Reconcile the two regularly. The reconciliation gap is bounded by the sync interval, assuming the billing sync itself is durable and complete. A failed or delayed sync widens it, so alert when a sync is late. Don't refund or bill from Redis directly.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The API key's plan itself (limits, quota, replica pool) should be cached in the gateway's memory for a short time, so authenticating a request costs zero database queries. Store only a &lt;strong&gt;hash&lt;/strong&gt; of each API key, the same way you'd store a password, and log the key's ID, never the key.&lt;/p&gt;

&lt;h3&gt;
  
  
  Tell clients where they stand
&lt;/h3&gt;

&lt;p&gt;Every response should include the limits, and every rejection should say why and when to retry:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight http"&gt;&lt;code&gt;&lt;span class="err"&gt;RateLimit-Policy: "per-second";q=50;w=1
RateLimit: "per-second";r=37;t=1
X-Credits-Remaining: 8421337
X-Credits-Cost: 5

&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;And when a request is rejected:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight http"&gt;&lt;code&gt;&lt;span class="k"&gt;HTTP&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="m"&gt;1.1&lt;/span&gt; &lt;span class="m"&gt;429&lt;/span&gt; &lt;span class="ne"&gt;Too Many Requests&lt;/span&gt;
&lt;span class="na"&gt;RateLimit-Policy&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;"per-second";q=50;w=1&lt;/span&gt;
&lt;span class="na"&gt;RateLimit&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;"per-second";r=0;t=1&lt;/span&gt;
&lt;span class="na"&gt;Retry-After&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;1&lt;/span&gt;

{"error": "rate_limited"}

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;RateLimit-Policy&lt;/code&gt; describes the limit (a quota &lt;code&gt;q&lt;/code&gt; of 50 per window &lt;code&gt;w&lt;/code&gt; of 1 second). &lt;code&gt;RateLimit&lt;/code&gt; reports the current state (&lt;code&gt;r&lt;/code&gt; requests remaining, resetting in &lt;code&gt;t&lt;/code&gt; seconds). These come from the IETF HTTPAPI working group's RateLimit header fields draft. It's still a draft, not a published RFC, and earlier draft versions used separate &lt;code&gt;RateLimit-Limit&lt;/code&gt;, &lt;code&gt;RateLimit-Remaining&lt;/code&gt; and &lt;code&gt;RateLimit-Reset&lt;/code&gt; fields, so you'll see both those and the older &lt;code&gt;X-RateLimit-*&lt;/code&gt; convention in real APIs. Pick one, document it, and keep it stable. Credits are your own application-level concept, so they keep their own &lt;code&gt;X-Credits-*&lt;/code&gt; headers.&lt;/p&gt;

&lt;p&gt;Clients that can see their limits back off on their own. Clients that can't just retry harder.&lt;/p&gt;

&lt;h2&gt;
  
  
  Layer 5: Bulkheads So Free Users Can't Hurt Paying Ones
&lt;/h2&gt;

&lt;p&gt;Plans aren't just billing. They're also a way to keep different kinds of traffic from interfering with each other.&lt;/p&gt;

&lt;p&gt;The query service routes each tier to its own replica pool, and connects to each with that tier's own database role and timeout:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;ROLE&lt;/span&gt; &lt;span class="n"&gt;api_free&lt;/span&gt; &lt;span class="n"&gt;LOGIN&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;ROLE&lt;/span&gt; &lt;span class="n"&gt;api_free&lt;/span&gt; &lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;statement_timeout&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'2s'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;ROLE&lt;/span&gt; &lt;span class="n"&gt;api_paid&lt;/span&gt; &lt;span class="n"&gt;LOGIN&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;ROLE&lt;/span&gt; &lt;span class="n"&gt;api_paid&lt;/span&gt; &lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;statement_timeout&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'5s'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Role-level settings apply when a connection logs in. If you use PgBouncer, give each tier its own pool that logs in as that tier's role, and keep each pool small. A pool's configured connection limit is a hard ceiling on how many database sessions that tier can have executing work at the same time, no matter how many requests arrive. Pool sizing has its own traps, covered in &lt;a href="https://insight.jatinjainsaraf.com/connection-pooling-in-the-serverless-era-five-failure-modes" rel="noopener noreferrer"&gt;Connection Pooling in the Serverless Era&lt;/a&gt; and the &lt;a href="https://academy.jatinjainsaraf.com/postgresql-in-depth/connection-pooling-failure-modes" rel="noopener noreferrer"&gt;Connection Pooling Failure Modes module&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Why this matters in production:&lt;/strong&gt; a viral free-tier app that suddenly sends 100x traffic fills the free pool and starts getting 429s. Paid customers, on a different pool and replica, don't notice. This is the bulkhead pattern from &lt;a href="https://insight.jatinjainsaraf.com/the-three-layers-of-failure-isolation-timeouts-circuit-breakers-and-load-shedding" rel="noopener noreferrer"&gt;the three layers of failure isolation&lt;/a&gt;, applied to pricing tiers.&lt;/p&gt;

&lt;p&gt;When the system as a whole is under pressure (replica lag above a threshold, CPU pinned), shed load &lt;strong&gt;in plan order&lt;/strong&gt; : free tier first, then lower paid tiers. Return &lt;code&gt;503&lt;/code&gt; with &lt;code&gt;Retry-After&lt;/code&gt;, and keep the explorer and the indexer out of it entirely. The &lt;a href="https://academy.jatinjainsaraf.com/system-design-in-depth/backpressure-and-load-shedding" rel="noopener noreferrer"&gt;backpressure and load shedding module&lt;/a&gt; covers how to pick those thresholds.&lt;/p&gt;

&lt;h2&gt;
  
  
  Layer 6: Don't Make Clients Poll for New Blocks
&lt;/h2&gt;

&lt;p&gt;A trading bot that wants every new event will poll &lt;code&gt;/events?from=latest&lt;/code&gt; once per block time. A thousand bots doing that means a thousand identical queries per block, all returning the same rows.&lt;/p&gt;

&lt;p&gt;Flip it around. The indexer already knows the moment a block is indexed. Have it publish new blocks and events once, to a fan-out layer (Redis Streams, a message broker, or your own WebSocket service), and push them to subscribers. The indexed data is fetched once for the fan-out path, instead of once per subscriber.&lt;/p&gt;

&lt;p&gt;Two things to get right:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Charge for streams per message or per subscription&lt;/strong&gt; , not per request, or a busy contract's event stream becomes free unlimited data.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Let clients resume.&lt;/strong&gt; Every message carries its block height. When a client reconnects, it sends the last height it saw and gets the gap filled from the API, so a dropped connection never means missed events.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Layer 7: Bulk Data Doesn't Belong in the API
&lt;/h2&gt;

&lt;p&gt;Some customers don't want an API. They want &lt;em&gt;everything&lt;/em&gt;: every transaction since genesis, to load into their own warehouse. If the only tool you give them is a paginated endpoint, they will page through your entire chain, one request at a time, for days.&lt;/p&gt;

&lt;p&gt;Give them a better door: periodic exports of finalized data (for example, daily Parquet files per table) in object storage, sold as their own plan. Exports are produced once, off a replica, during quiet hours, and downloaded any number of times at no cost to your databases.&lt;/p&gt;

&lt;h2&gt;
  
  
  Observability: Know Who Is Expensive
&lt;/h2&gt;

&lt;p&gt;You can't price or protect what you can't see. Track, per API key:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Requests and credits spent, by endpoint&lt;/li&gt;
&lt;li&gt;Cache hit rate&lt;/li&gt;
&lt;li&gt;p50/p99 latency and error rate&lt;/li&gt;
&lt;li&gt;Which keys are hitting rate limits or timeouts&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Tag queries so a slow one on the replica can be traced back to a customer. Postgres's &lt;code&gt;application_name&lt;/code&gt; connection setting, or a comment in the SQL text, can carry the &lt;strong&gt;key ID&lt;/strong&gt; (never the key). When a replica slows down, you want to know in seconds whether it's one customer or everyone.&lt;/p&gt;

&lt;h2&gt;
  
  
  A Checklist Before You Launch
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Does any external API or explorer traffic still read the primary, or share a replica pool across the two?&lt;/li&gt;
&lt;li&gt;Does the gateway stay out of the database, with every query owned by the query service?&lt;/li&gt;
&lt;li&gt;Does every endpoint map to an index, with a bounded result size and cursor pagination?&lt;/li&gt;
&lt;li&gt;Is "final" defined per chain, and is only finalized data cached as immutable?&lt;/li&gt;
&lt;li&gt;Are replica query cancellations retried and turned into clean 503s?&lt;/li&gt;
&lt;li&gt;Are rate limits and credits checked in Redis, atomically, with zero database queries per request?&lt;/li&gt;
&lt;li&gt;Is the billing sync idempotent, and can lost Redis counters be re-seeded from Postgres?&lt;/li&gt;
&lt;li&gt;Does each tier have its own pool, role and statement timeout?&lt;/li&gt;
&lt;li&gt;Does every response include the indexed height and remaining credits?&lt;/li&gt;
&lt;li&gt;Is there a plan for real-time data that isn't polling, and for bulk data that isn't pagination?&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The Takeaway
&lt;/h2&gt;

&lt;p&gt;A blockchain indexer that serves only your explorer is a database. One that serves anyone, on plans, is a product, and a product needs a counter in front of the kitchen.&lt;/p&gt;

&lt;p&gt;The design comes down to one idea: &lt;strong&gt;external traffic should only ever reach things that are built to absorb it.&lt;/strong&gt; Replicas instead of the primary. A query service instead of SQL in the gateway. Read models instead of raw tables. Caches instead of replicas, wherever the data is final. Redis instead of the database for every limit check, with Postgres still keeping the books. Separate pools so one tier's spike stays its own problem.&lt;/p&gt;

&lt;p&gt;Get that right, and selling your data stops being a risk to the explorer that made it worth buying.&lt;/p&gt;

</description>
      <category>blockchain</category>
      <category>systemdesign</category>
      <category>apidesign</category>
      <category>postgres</category>
    </item>
    <item>
      <title>Your Postgres Indexes Can Break Without a Single Line of Code Changing</title>
      <dc:creator>Jatin Jain Saraf</dc:creator>
      <pubDate>Wed, 07 Oct 2026 03:47:27 +0000</pubDate>
      <link>https://dev.to/jatinjainsaraf/your-postgres-indexes-can-break-without-a-single-line-of-code-changing-437g</link>
      <guid>https://dev.to/jatinjainsaraf/your-postgres-indexes-can-break-without-a-single-line-of-code-changing-437g</guid>
      <description>&lt;h1&gt;
  
  
  Your Postgres Indexes Can Break Without a Single Line of Code Changing
&lt;/h1&gt;

&lt;p&gt;The ticket says a customer can't log in. Support swears the account exists. You run the query yourself:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;users&lt;/span&gt; &lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;username&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'jo-ann'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Zero rows.&lt;/p&gt;

&lt;p&gt;You open the admin panel, which happens to page through the table with a sequential scan, and there she is. Same username, same spelling, no trailing spaces.&lt;/p&gt;

&lt;p&gt;A week later a second problem shows up: two rows with the same username, in a column with a &lt;code&gt;UNIQUE&lt;/code&gt; index on it. The constraint that was supposed to make that impossible simply let it happen.&lt;/p&gt;

&lt;p&gt;Nobody deployed anything. No migration ran. The only change in the last month was a routine one: the Postgres container moved to a newer base image, or the replica was rebuilt on a newer OS, or the database was restored onto a fresh host with different locale data.&lt;/p&gt;

&lt;p&gt;That routine change is the whole story. The operating system changed the collation rules used to sort text, and B-tree indexes using that collation were built under the old rules.&lt;/p&gt;

&lt;h2&gt;
  
  
  A Dictionary Sorted by One Alphabet, Searched by Another
&lt;/h2&gt;

&lt;p&gt;A B-tree index is a dictionary. The entries are sorted, and finding a word means opening to the middle, deciding "my word comes before this page" or "after this page," and repeating until you land on the right page.&lt;/p&gt;

&lt;p&gt;That decision, "before or after," is the only thing the search depends on. Postgres doesn't scan the dictionary. It trusts the order. (If you want the full layout of root, internal and leaf pages, &lt;a href="https://academy.jatinjainsaraf.com/postgresql-in-depth/indexes" rel="noopener noreferrer"&gt;Module A-5: Index Internals&lt;/a&gt; in the PostgreSQL In-Depth course walks through it. It's the same leaf-page structure that makes &lt;a href="https://insight.jatinjainsaraf.com/your-uuid-primary-keys-are-quietly-taxing-every-insert" rel="noopener noreferrer"&gt;random UUID primary keys tax every insert&lt;/a&gt;.)&lt;/p&gt;

&lt;p&gt;Now imagine the dictionary was printed under one set of alphabet rules, where hyphens are ignored so &lt;code&gt;jo-ann&lt;/code&gt; sits next to &lt;code&gt;joann&lt;/code&gt;. Years later, someone hands the reader a new rulebook where a hyphen sorts before every letter. The pages haven't moved. Every word is still printed. But the reader, following the new rules, decides &lt;code&gt;jo-ann&lt;/code&gt; must come before &lt;code&gt;joan&lt;/code&gt;, flips to that section, doesn't find it, and confidently reports that the word isn't in the dictionary.&lt;/p&gt;

&lt;p&gt;That is the failure mode: the index pages were ordered under the old rules, while the comparison rules used to navigate them have changed.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Why this matters in production:&lt;/strong&gt; Postgres does not detect this during normal operation. There is no error and no failed query. Lookups quietly miss rows, and unique indexes quietly accept duplicates, because both operations rely on the B-tree walking to the right leaf page.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where the Sort Order Actually Lives
&lt;/h2&gt;

&lt;p&gt;For a database using a libc collation such as &lt;code&gt;en_US.UTF-8&lt;/code&gt;, which is still what most installs and container images default to, Postgres doesn't carry its own idea of alphabetical order. It delegates locale-aware comparison to the operating system's C library. On a glibc-based Linux system, that means glibc's collation implementation.&lt;/p&gt;

&lt;p&gt;So "is &lt;code&gt;jo-ann&lt;/code&gt; less than &lt;code&gt;joan&lt;/code&gt;?" is answered by whatever version of glibc is installed on the machine running Postgres at that moment. Not the version that was installed when the index was built.&lt;/p&gt;

&lt;p&gt;Most glibc upgrades change little or nothing in collation. The famous exception is glibc 2.28, released in 2018, which brought its locale data in line with a newer ISO 14651 standard and changed the relative order of a large number of strings, especially ones containing punctuation and spaces. The PostgreSQL wiki's test case is a one-liner you can run on any Linux box:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="o"&gt;(&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"1-1"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"11"&lt;/span&gt; &lt;span class="o"&gt;)&lt;/span&gt; | &lt;span class="nv"&gt;LC_COLLATE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;en_US.UTF-8 &lt;span class="nb"&gt;sort&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Those two strings come out in a different order depending on which side of 2.28 you're on. That change shipped in Debian 10 (buster), Ubuntu 18.10, and RHEL 8, which means a lot of databases crossed that line during routine OS upgrades.&lt;/p&gt;

&lt;p&gt;Postgres has no way to know which glibc changes matter and which don't. So the safe assumption is that any change to the C library under an existing data directory might have changed the sort order.&lt;/p&gt;

&lt;h2&gt;
  
  
  Reproducing It
&lt;/h2&gt;

&lt;p&gt;You can't easily swap glibc under a running server on a laptop, so I simulated it on PostgreSQL 18.6. I created two ICU collations that differ in one rule, whether punctuation is ignored, which is the same kind of rule glibc 2.28 changed:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;COLLATION&lt;/span&gt; &lt;span class="n"&gt;old_rules&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;provider&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;icu&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;locale&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'en-u-ka-shifted'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;COLLATION&lt;/span&gt; &lt;span class="n"&gt;new_rules&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;provider&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;icu&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;locale&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'en'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Same four strings, sorted under each:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="err"&gt;old_rules:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;ab&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;a&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;c&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;a-c&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;ad&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="err"&gt;new_rules:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;a&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;c&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;a-c&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;ab&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;ad&lt;/span&gt;&lt;span class="w"&gt;

&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Under the old rules, &lt;code&gt;a-c&lt;/code&gt; is treated roughly like &lt;code&gt;ac&lt;/code&gt;, so it lands after &lt;code&gt;ab&lt;/code&gt;. Under the new rules, the hyphen and the space sort before letters, so &lt;code&gt;a-c&lt;/code&gt; jumps ahead of &lt;code&gt;ab&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Then I built a real table: 200,000 usernames (a third of them hyphenated) plus &lt;code&gt;jo-ann&lt;/code&gt;, &lt;code&gt;joan&lt;/code&gt; and &lt;code&gt;johnny&lt;/code&gt;, with a unique index, all under &lt;code&gt;old_rules&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;users&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="nb"&gt;bigint&lt;/span&gt; &lt;span class="k"&gt;GENERATED&lt;/span&gt; &lt;span class="n"&gt;ALWAYS&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="k"&gt;IDENTITY&lt;/span&gt; &lt;span class="k"&gt;PRIMARY&lt;/span&gt; &lt;span class="k"&gt;KEY&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;username&lt;/span&gt; &lt;span class="nb"&gt;text&lt;/span&gt; &lt;span class="k"&gt;COLLATE&lt;/span&gt; &lt;span class="n"&gt;old_rules&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;
&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="c1"&gt;-- ... load 200,003 rows ...&lt;/span&gt;
&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;UNIQUE&lt;/span&gt; &lt;span class="k"&gt;INDEX&lt;/span&gt; &lt;span class="n"&gt;users_username_key&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;users&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;username&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;At this point everything is healthy. &lt;code&gt;amcheck&lt;/code&gt; passes and &lt;code&gt;WHERE username = 'jo-ann'&lt;/code&gt; returns one row.&lt;/p&gt;

&lt;p&gt;Then the "OS upgrade": I pointed the column and the index at &lt;code&gt;new_rules&lt;/code&gt; directly in the system catalog. The index pages on disk are byte for byte what the old rules built. Only the comparison function changed. That's precisely the situation after a glibc upgrade.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Don't try this on a real database.&lt;/strong&gt; Editing &lt;code&gt;pg_index&lt;/code&gt; and &lt;code&gt;pg_attribute&lt;/code&gt; by hand is how you &lt;em&gt;create&lt;/em&gt; corruption, not how you fix it. I did it only on a disposable local test cluster, to fake the broken state an OS upgrade leaves behind. It is not part of any remediation. The real fix is further down.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h3&gt;
  
  
  Symptom 1: the row vanishes from indexed lookups
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;EXPLAIN&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;COSTS&lt;/span&gt; &lt;span class="k"&gt;OFF&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;users&lt;/span&gt; &lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;username&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'jo-ann'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;


                  &lt;span class="n"&gt;QUERY&lt;/span&gt; &lt;span class="n"&gt;PLAN&lt;/span&gt;
&lt;span class="c1"&gt;----------------------------------------------&lt;/span&gt;
 &lt;span class="k"&gt;Index&lt;/span&gt; &lt;span class="n"&gt;Scan&lt;/span&gt; &lt;span class="k"&gt;using&lt;/span&gt; &lt;span class="n"&gt;users_username_key&lt;/span&gt; &lt;span class="k"&gt;on&lt;/span&gt; &lt;span class="n"&gt;users&lt;/span&gt;
   &lt;span class="k"&gt;Index&lt;/span&gt; &lt;span class="n"&gt;Cond&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;username&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'jo-ann'&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="nb"&gt;text&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;


&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;users&lt;/span&gt; &lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;username&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'jo-ann'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;


 &lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="n"&gt;username&lt;/span&gt;
&lt;span class="c1"&gt;----+----------&lt;/span&gt;
&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt; &lt;span class="k"&gt;rows&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Turn off index scans and read the table directly, and it's right there. (Those &lt;code&gt;enable_*&lt;/code&gt; switches are planner debugging tools, covered in &lt;a href="https://academy.jatinjainsaraf.com/postgresql-in-depth/query-planning" rel="noopener noreferrer"&gt;Module A-6: Query Planning&lt;/a&gt;. Never set them globally in production.)&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;enable_indexscan&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;off&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;enable_bitmapscan&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;off&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;users&lt;/span&gt; &lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;username&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'jo-ann'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;


   &lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="n"&gt;username&lt;/span&gt;
&lt;span class="c1"&gt;--------+----------&lt;/span&gt;
 &lt;span class="mi"&gt;200001&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="n"&gt;jo&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;ann&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This is the "customer can't log in" ticket. The data is fine. The path to the data is wrong.&lt;/p&gt;

&lt;h3&gt;
  
  
  Symptom 2: the unique index accepts a duplicate
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;INSERT&lt;/span&gt; &lt;span class="k"&gt;INTO&lt;/span&gt; &lt;span class="n"&gt;users&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;username&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;VALUES&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'jo-ann'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;RETURNING&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;username&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;


   &lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="n"&gt;username&lt;/span&gt;
&lt;span class="c1"&gt;--------+----------&lt;/span&gt;
 &lt;span class="mi"&gt;200004&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="n"&gt;jo&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;ann&lt;/span&gt;

&lt;span class="k"&gt;INSERT&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A unique index enforces uniqueness by walking to the leaf page where the new key belongs and checking whether it's already there. Under the new rules it walks to a different leaf, finds nothing, and inserts. Every guarantee in &lt;a href="https://academy.jatinjainsaraf.com/postgresql-in-depth/constraints-and-data-integrity" rel="noopener noreferrer"&gt;Constraints and Data Integrity&lt;/a&gt; that's backed by an index is only as good as the index's sort order.&lt;/p&gt;

&lt;p&gt;And the duplicate hides itself. Ask through the index and you get one:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt; index_says
------------
          1

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Ask the table and you get two:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt; table_says
------------
          2

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Why this matters in production:&lt;/strong&gt; this is the dangerous one. Missing lookups are visible and get reported. Duplicates in a column your application assumes is unique are silent, and they spread: two accounts, two payment records, a join that now returns double. By the time anyone notices, the bad rows have been written for weeks.&lt;/p&gt;

&lt;h3&gt;
  
  
  Symptom 3: the fix fails too
&lt;/h3&gt;

&lt;p&gt;The obvious move is to rebuild the index:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;REINDEX&lt;/span&gt; &lt;span class="k"&gt;INDEX&lt;/span&gt; &lt;span class="n"&gt;CONCURRENTLY&lt;/span&gt; &lt;span class="n"&gt;users_username_key&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;


&lt;span class="n"&gt;ERROR&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;could&lt;/span&gt; &lt;span class="k"&gt;not&lt;/span&gt; &lt;span class="k"&gt;create&lt;/span&gt; &lt;span class="k"&gt;unique&lt;/span&gt; &lt;span class="k"&gt;index&lt;/span&gt; &lt;span class="nv"&gt;"users_username_key_ccnew"&lt;/span&gt;
&lt;span class="n"&gt;DETAIL&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="k"&gt;Key&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;username&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;jo&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;ann&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;is&lt;/span&gt; &lt;span class="n"&gt;duplicated&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The rebuild sorts everything under the new rules, notices the duplicate the broken index let in, and refuses. Worse, a failed &lt;code&gt;REINDEX CONCURRENTLY&lt;/code&gt; leaves its half-built index behind. Run the detection query below and you'll see &lt;code&gt;users_username_key_ccnew&lt;/code&gt; sitting there, marked invalid, still costing write overhead on every insert until you drop it. It's the same invalid-index trap that catches failed &lt;code&gt;CREATE INDEX CONCURRENTLY&lt;/code&gt; runs, covered in &lt;a href="https://academy.jatinjainsaraf.com/postgresql-in-depth/zero-downtime-schema-migrations" rel="noopener noreferrer"&gt;Zero-Downtime Schema Migrations&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where This Bites in Real Systems
&lt;/h2&gt;

&lt;p&gt;The common thread is any path that &lt;strong&gt;keeps the existing index files&lt;/strong&gt; while &lt;strong&gt;changing the C library underneath them&lt;/strong&gt; :&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Changing the Docker base image.&lt;/strong&gt; Bumping &lt;code&gt;postgres:15-bullseye&lt;/code&gt; to &lt;code&gt;postgres:15-bookworm&lt;/code&gt; keeps the Postgres major version and your data volume, and swaps glibc from 2.31 to 2.36. Moving a data directory between an Alpine image (musl, which doesn't do locale-aware sorting) and a Debian image is a much bigger jump. If your Postgres lives on a &lt;a href="https://academy.jatinjainsaraf.com/docker-in-depth/stateful-containers-and-storage" rel="noopener noreferrer"&gt;Docker volume&lt;/a&gt;, the volume outlives the image, which is exactly what makes this easy to miss.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Physical replicas on a different OS.&lt;/strong&gt; &lt;a href="https://academy.jatinjainsaraf.com/postgresql-in-depth/replication-ha" rel="noopener noreferrer"&gt;Streaming replication&lt;/a&gt; ships the primary's index pages as-is. A replica running a different glibc answers queries against those pages with its own sort rules. It returns wrong results while it's a replica, and if it's promoted, it starts writing new entries into indexes it misreads.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;pg_upgrade&lt;/code&gt; combined with an OS upgrade.&lt;/strong&gt; &lt;code&gt;pg_upgrade&lt;/code&gt; is fast precisely because it reuses the index files. If the new server runs on a newer OS, those files are now walked with new rules.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Restoring a file-level backup onto a newer host.&lt;/strong&gt; Same mechanism as above.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Safer migration paths: &lt;code&gt;pg_dump&lt;/code&gt;/restore and logical replication rebuild indexes on the target, under the target's rules, rather than carrying the existing index files across. That's one strong argument for &lt;a href="https://academy.jatinjainsaraf.com/postgresql-in-depth/logical-replication-cdc-pipelines" rel="noopener noreferrer"&gt;logical replication&lt;/a&gt; in OS migrations: you pay for it in setup and time, and in exchange the collation problem disappears. I covered the full playbook for that kind of move in &lt;a href="https://insight.jatinjainsaraf.com/cloudtocloud-database-migration-a-practical-guide-for-large-postgresql-systems" rel="noopener noreferrer"&gt;Cloud-to-Cloud Database Migration&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Range partitioning on a text key has the same exposure, by the way. Partition bounds are compared with the same collation, so rows can be routed to a different partition than the one that "should" hold them under the old rules.&lt;/p&gt;

&lt;h2&gt;
  
  
  Detection
&lt;/h2&gt;

&lt;h3&gt;
  
  
  The warning you probably never saw
&lt;/h3&gt;

&lt;p&gt;PostgreSQL 18 records a database-level collation version and checks it against the version currently provided by the system on every new connection. Here's what it looks like after a move from Debian bullseye to bookworm (the message format is exactly what Postgres prints; the version numbers are illustrative, since my laptop reproduction reports different ones):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;WARNING: database "app" has a collation version mismatch
DETAIL: The database was created using collation version 2.31, but the operating system provides version 2.36.
HINT: Rebuild all objects in this database that use the default collation and run ALTER DATABASE app REFRESH COLLATION VERSION, or build PostgreSQL with the right library version.

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That's a good warning. The problem is where it goes. It's sent to the connecting client, and your application's database driver almost certainly discards it. Look for it in a &lt;code&gt;psql&lt;/code&gt; session after any OS change. It may be the only early signal you'll get.&lt;/p&gt;

&lt;h3&gt;
  
  
  Find the indexes at risk
&lt;/h3&gt;

&lt;p&gt;Indexes whose indexed expressions use collatable data can depend on collation, including indexes on &lt;code&gt;text&lt;/code&gt;, &lt;code&gt;varchar&lt;/code&gt;, &lt;code&gt;char&lt;/code&gt;, &lt;code&gt;citext&lt;/code&gt;, and expressions such as &lt;code&gt;lower(email)&lt;/code&gt;. Your &lt;code&gt;bigint&lt;/code&gt; and &lt;code&gt;uuid&lt;/code&gt; keys are unaffected. This lists the ones that are exposed (the &lt;code&gt;CROSS JOIN LATERAL&lt;/code&gt; turns each index's collation list into rows, the pattern from &lt;a href="https://insight.jatinjainsaraf.com/the-power-of-lateral-joins-turning-sql-into-a-foreach-loop" rel="noopener noreferrer"&gt;The Power of LATERAL Joins&lt;/a&gt;):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="k"&gt;DISTINCT&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;indrelid&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="n"&gt;regclass&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="k"&gt;table_name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;indexrelid&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="n"&gt;regclass&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;index_name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="k"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;collname&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                &lt;span class="k"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;collprovider&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;pg_index&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;
&lt;span class="k"&gt;CROSS&lt;/span&gt; &lt;span class="k"&gt;JOIN&lt;/span&gt; &lt;span class="k"&gt;LATERAL&lt;/span&gt; &lt;span class="k"&gt;unnest&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;indcollation&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="n"&gt;oid&lt;/span&gt;&lt;span class="p"&gt;[])&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;ic&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;coll&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;JOIN&lt;/span&gt; &lt;span class="n"&gt;pg_collation&lt;/span&gt; &lt;span class="k"&gt;c&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="k"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;oid&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ic&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;coll&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="k"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;collprovider&lt;/span&gt; &lt;span class="k"&gt;IN&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'d'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'c'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'i'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="k"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;collname&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;IN&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'C'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'POSIX'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;ORDER&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;d&lt;/code&gt; is the database default, &lt;code&gt;c&lt;/code&gt; is libc and &lt;code&gt;i&lt;/code&gt; is ICU. If your database's default provider is &lt;code&gt;builtin&lt;/code&gt; (more on that below), the &lt;code&gt;d&lt;/code&gt; rows are safe and you can ignore them.&lt;/p&gt;

&lt;h3&gt;
  
  
  Prove it with amcheck
&lt;/h3&gt;

&lt;p&gt;&lt;code&gt;amcheck&lt;/code&gt; validates the B-tree's structural and ordering invariants under the current comparison rules. It's the B-tree equivalent of asking the reader to check every page of the dictionary against the new rulebook:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="n"&gt;EXTENSION&lt;/span&gt; &lt;span class="n"&gt;IF&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;EXISTS&lt;/span&gt; &lt;span class="n"&gt;amcheck&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;bt_index_check&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'users_username_key'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;heapallindexed&lt;/span&gt; &lt;span class="o"&gt;=&amp;gt;&lt;/span&gt; &lt;span class="k"&gt;true&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On the broken index:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ERROR: item order invariant violated for index "users_username_key"
DETAIL: Lower index tid=(116,4) (points to index tid=(338,1)) higher index tid=(116,5) (points to index tid=(449,1)) page lsn=0/473D668.

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;bt_index_check&lt;/code&gt; takes &lt;code&gt;AccessShareLock&lt;/code&gt;, the same lock mode a plain &lt;code&gt;SELECT&lt;/code&gt; takes, so reads and writes carry on while it runs and it's safe on a live system. (Its stricter sibling, &lt;code&gt;bt_index_parent_check&lt;/code&gt;, takes a &lt;code&gt;ShareLock&lt;/code&gt; that blocks writes, so keep that one for maintenance windows. Lock modes and what they block are covered in &lt;a href="https://academy.jatinjainsaraf.com/postgresql-in-depth/locking-internals" rel="noopener noreferrer"&gt;Module A-13: Locking Internals&lt;/a&gt;.) It isn't free though. It reads the entire index, and &lt;code&gt;heapallindexed =&amp;gt; true&lt;/code&gt; also reads the entire table, so on a large database it's real I/O you'll want to schedule. For a whole database at once, the &lt;code&gt;pg_amcheck&lt;/code&gt; command-line tool that ships with Postgres runs the same checks in parallel:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;pg_amcheck &lt;span class="nt"&gt;--heapallindexed&lt;/span&gt; &lt;span class="nt"&gt;--install-missing&lt;/span&gt; &lt;span class="nt"&gt;--jobs&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;4 &lt;span class="nt"&gt;-d&lt;/span&gt; app

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  The Fix, in the Right Order
&lt;/h2&gt;

&lt;p&gt;The order matters more than any single step.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. Find duplicates through the table, not the index.&lt;/strong&gt; Any query that uses the broken index will hide them, so force sequential scans for the session:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;enable_indexscan&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;off&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;enable_bitmapscan&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;off&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;enable_indexonlyscan&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;off&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;username&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;count&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;users&lt;/span&gt;
&lt;span class="k"&gt;GROUP&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;username&lt;/span&gt;
&lt;span class="k"&gt;HAVING&lt;/span&gt; &lt;span class="k"&gt;count&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;2. Resolve them.&lt;/strong&gt; This is a business decision, not a SQL one. Two "identical" accounts might each have orders attached. In the demo I kept the oldest row, but in production you'd merge or reassign first.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Clean up any failed rebuilds,&lt;/strong&gt; then rebuild:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;DROP&lt;/span&gt; &lt;span class="k"&gt;INDEX&lt;/span&gt; &lt;span class="n"&gt;CONCURRENTLY&lt;/span&gt; &lt;span class="n"&gt;IF&lt;/span&gt; &lt;span class="k"&gt;EXISTS&lt;/span&gt; &lt;span class="n"&gt;users_username_key_ccnew&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;REINDEX&lt;/span&gt; &lt;span class="k"&gt;INDEX&lt;/span&gt; &lt;span class="n"&gt;CONCURRENTLY&lt;/span&gt; &lt;span class="n"&gt;users_username_key&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;4. Verify.&lt;/strong&gt; &lt;code&gt;bt_index_check&lt;/code&gt; returns cleanly and &lt;code&gt;WHERE username = 'jo-ann'&lt;/code&gt; returns exactly one row again.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;5. Only then, acknowledge the new version:&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;DATABASE&lt;/span&gt; &lt;span class="n"&gt;app&lt;/span&gt; &lt;span class="n"&gt;REFRESH&lt;/span&gt; &lt;span class="k"&gt;COLLATION&lt;/span&gt; &lt;span class="k"&gt;VERSION&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Here is the trap. That command does not rebuild or check anything. It updates the recorded version number so the warning stops. Run it first and you've silenced the only alarm while every index is still broken. Treat it as the last line of the runbook, a signature that says "I have rebuilt everything," never as the fix itself.&lt;/p&gt;

&lt;h2&gt;
  
  
  Making Collation Drift Boring
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Pin the C library with the Postgres version
&lt;/h3&gt;

&lt;p&gt;The cheapest protection is process. Treat the base OS image as part of the database version. &lt;code&gt;postgres:16.4-bookworm&lt;/code&gt;, not &lt;code&gt;postgres:16&lt;/code&gt;. Changing the OS image then becomes a database migration that goes through the same review as a major version upgrade, with "reindex text indexes, then amcheck" written into the checklist.&lt;/p&gt;

&lt;h3&gt;
  
  
  Use a collation that can't drift with the OS
&lt;/h3&gt;

&lt;p&gt;PostgreSQL 18 ships a &lt;code&gt;builtin&lt;/code&gt; collation provider. Its &lt;code&gt;C.UTF-8&lt;/code&gt; locale uses Unicode code point ordering implemented inside PostgreSQL itself, so ordinary OS upgrades don't change its ordering rules:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;DATABASE&lt;/span&gt; &lt;span class="n"&gt;app&lt;/span&gt;
  &lt;span class="n"&gt;LOCALE_PROVIDER&lt;/span&gt; &lt;span class="n"&gt;builtin&lt;/span&gt;
  &lt;span class="n"&gt;BUILTIN_LOCALE&lt;/span&gt; &lt;span class="s1"&gt;'C.UTF-8'&lt;/span&gt;
  &lt;span class="k"&gt;TEMPLATE&lt;/span&gt; &lt;span class="n"&gt;template0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The honest cost: code point order isn't what humans expect. Every uppercase letter sorts before every lowercase one, so &lt;code&gt;Zebra&lt;/code&gt; comes before &lt;code&gt;apple&lt;/code&gt;, and accented characters like &lt;code&gt;é&lt;/code&gt; sort after &lt;code&gt;z&lt;/code&gt;. For identifiers, emails, usernames, SKUs and other machine-oriented keys, code point ordering is often a good trade-off, because the application usually cares about deterministic equality and lookup semantics rather than human alphabetical order. For display sorting, apply a linguistic collation at query time:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt; &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;products&lt;/span&gt; &lt;span class="k"&gt;ORDER&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt; &lt;span class="k"&gt;COLLATE&lt;/span&gt; &lt;span class="nv"&gt;"en-x-icu"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That sort won't be served from an index unless you create one with that collation, and an index like that brings back the versioning problem, just isolated to the one index you chose.&lt;/p&gt;

&lt;p&gt;One caveat, and it matters for anyone planning a major-version upgrade. Code point &lt;em&gt;ordering&lt;/em&gt; is fixed, but &lt;code&gt;builtin&lt;/code&gt; takes its &lt;em&gt;case mapping&lt;/em&gt; (what &lt;code&gt;lower()&lt;/code&gt; and &lt;code&gt;upper()&lt;/code&gt; return) from Unicode tables compiled into Postgres, and the docs only promise those are stable within a major version. PostgreSQL 18 moved to Unicode 16.0, and PostgreSQL 19 (in beta as I write this) moves to Unicode 17.0. The rest of what changes in 19 is in &lt;a href="https://insight.jatinjainsaraf.com/postgresql-19-vs-18-what-actually-changed-feature-by-feature" rel="noopener noreferrer"&gt;PostgreSQL 19 vs 18: What Actually Changed&lt;/a&gt;. An expression index such as &lt;code&gt;lower(email)&lt;/code&gt; is therefore worth an &lt;code&gt;amcheck&lt;/code&gt; after a &lt;code&gt;pg_upgrade&lt;/code&gt; across major versions, even with no OS change at all. For the same reason, the PostgreSQL 18 release notes recommend reindexing &lt;a href="https://academy.jatinjainsaraf.com/postgresql-in-depth/full-text-search" rel="noopener noreferrer"&gt;full-text search&lt;/a&gt; and &lt;code&gt;pg_trgm&lt;/code&gt; indexes after &lt;code&gt;pg_upgrade&lt;/code&gt; on clusters whose default provider isn't libc. If you need case-insensitive matching, PostgreSQL 18 also adds &lt;code&gt;PG_UNICODE_FAST&lt;/code&gt;, a builtin locale with full Unicode case mapping that still sorts by code point.&lt;/p&gt;

&lt;p&gt;You also can't switch an existing database's default locale in place. Moving to &lt;code&gt;builtin&lt;/code&gt; means a new database, filled by dump and restore or by logical replication, which, conveniently, is also the safer way to do an OS migration.&lt;/p&gt;

&lt;h3&gt;
  
  
  ICU is better, not immune
&lt;/h3&gt;

&lt;p&gt;ICU collations (&lt;code&gt;provider = icu&lt;/code&gt;) decouple sorting from glibc and give you consistent behavior across Linux and macOS. Postgres tracks an ICU collation's version and warns when it changes. But ICU's rules do change between ICU major versions, and in most Linux images ICU comes from an OS package that upgrades along with the base image. ICU makes the drift visible and less frequent. It doesn't make it go away.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Takeaway
&lt;/h2&gt;

&lt;p&gt;An index is a promise that the data is sorted. With text, the meaning of "sorted" belongs to a library you didn't write, versioned by an OS image you might upgrade without thinking about the database at all.&lt;/p&gt;

&lt;p&gt;So the rule is short. Any time the C library under an existing data directory changes, through a new base image, a replica on a newer OS, or &lt;code&gt;pg_upgrade&lt;/code&gt; on a new host, assume every index on a libc or ICU collation is suspect until &lt;code&gt;amcheck&lt;/code&gt; says otherwise. Rebuild, verify, and only then tell Postgres the new version is fine.&lt;/p&gt;

&lt;p&gt;The OS doesn't know it's touching your database. You have to.&lt;/p&gt;

&lt;h2&gt;
  
  
  Go Deeper
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;a href="https://academy.jatinjainsaraf.com/postgresql-in-depth" rel="noopener noreferrer"&gt;PostgreSQL In-Depth&lt;/a&gt;: the full course, from storage engine to production operations&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://academy.jatinjainsaraf.com/postgresql-in-depth/indexes" rel="noopener noreferrer"&gt;Module A-5: Index Internals&lt;/a&gt;: B-tree, Hash, GIN and BRIN, and how the planner sees them&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://academy.jatinjainsaraf.com/system-design-in-depth/storage-engines-btrees-vs-lsm-trees" rel="noopener noreferrer"&gt;Storage Engines: B-Trees vs LSM Trees&lt;/a&gt;, from System Design In-Depth&lt;/li&gt;
&lt;li&gt;&lt;a href="https://insight.jatinjainsaraf.com/why-postgres-and-cassandra-made-opposite-bets-on-storage-engines" rel="noopener noreferrer"&gt;Why Postgres and Cassandra Made Opposite Bets on Storage Engines&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://academy.jatinjainsaraf.com/docker-in-depth/building-production-grade-images" rel="noopener noreferrer"&gt;Building Production-Grade Images&lt;/a&gt;, for pinning the base image your database runs on&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  References
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;PostgreSQL wiki: &lt;a href="https://wiki.postgresql.org/wiki/Locale_data_changes" rel="noopener noreferrer"&gt;Locale data changes&lt;/a&gt; (glibc 2.28, affected distributions, the &lt;code&gt;sort&lt;/code&gt; test)&lt;/li&gt;
&lt;li&gt;PostgreSQL 18 docs: &lt;a href="https://www.postgresql.org/docs/18/collation.html" rel="noopener noreferrer"&gt;Collation support&lt;/a&gt; and &lt;a href="https://www.postgresql.org/docs/18/locale.html#LOCALE-PROVIDERS" rel="noopener noreferrer"&gt;Locale providers&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;PostgreSQL 18 docs: &lt;a href="https://www.postgresql.org/docs/18/sql-altercollation.html#SQL-ALTERCOLLATION-NOTES" rel="noopener noreferrer"&gt;ALTER COLLATION, notes on collation versions&lt;/a&gt; and &lt;a href="https://www.postgresql.org/docs/18/sql-alterdatabase.html" rel="noopener noreferrer"&gt;ALTER DATABASE&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;PostgreSQL 18 docs: &lt;a href="https://www.postgresql.org/docs/18/amcheck.html" rel="noopener noreferrer"&gt;amcheck&lt;/a&gt; and &lt;a href="https://www.postgresql.org/docs/18/app-pgamcheck.html" rel="noopener noreferrer"&gt;pg_amcheck&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;PostgreSQL 18 docs: &lt;a href="https://www.postgresql.org/docs/18/sql-reindex.html" rel="noopener noreferrer"&gt;REINDEX&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.postgresql.org/docs/release/18.0/" rel="noopener noreferrer"&gt;PostgreSQL 18 release notes&lt;/a&gt; (Unicode 16.0, &lt;code&gt;PG_UNICODE_FAST&lt;/code&gt;, full-text search reindex advice)&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.postgresql.org/docs/19/release-19.html" rel="noopener noreferrer"&gt;PostgreSQL 19 release notes&lt;/a&gt; (beta; Unicode 17.0)&lt;/li&gt;
&lt;li&gt;Debian packages: &lt;a href="https://packages.debian.org/bullseye/libc6" rel="noopener noreferrer"&gt;libc6 in bullseye&lt;/a&gt; (2.31) and &lt;a href="https://packages.debian.org/bookworm/libc6" rel="noopener noreferrer"&gt;libc6 in bookworm&lt;/a&gt; (2.36)&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>postgres</category>
      <category>postgresinternals</category>
      <category>databaseinternals</category>
      <category>indexing</category>
    </item>
    <item>
      <title>Your UUID Primary Keys Are Quietly Taxing Every Insert</title>
      <dc:creator>Jatin Jain Saraf</dc:creator>
      <pubDate>Tue, 06 Oct 2026 03:05:50 +0000</pubDate>
      <link>https://dev.to/jatinjainsaraf/your-uuid-primary-keys-are-quietly-taxing-every-insert-392b</link>
      <guid>https://dev.to/jatinjainsaraf/your-uuid-primary-keys-are-quietly-taxing-every-insert-392b</guid>
      <description>&lt;p&gt;Nobody gets paged because of a primary key type.&lt;/p&gt;

&lt;p&gt;The problem shows up as a slow drift. Insert latency that was 2ms last quarter is now 6ms. WAL volume grows faster than the data you write. Replica lag spikes after a checkpoint. The cache hit ratio slides from 99.9% to 97% while the queries stay exactly the same.&lt;/p&gt;

&lt;p&gt;Then someone runs &lt;code&gt;EXPLAIN&lt;/code&gt; on the slow query, finds nothing obviously wrong, and blames the disk.&lt;/p&gt;

&lt;p&gt;Sometimes the decision was made years earlier, in a line like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="n"&gt;uuid&lt;/span&gt; &lt;span class="k"&gt;PRIMARY&lt;/span&gt; &lt;span class="k"&gt;KEY&lt;/span&gt; &lt;span class="k"&gt;DEFAULT&lt;/span&gt; &lt;span class="n"&gt;gen_random_uuid&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Random UUIDs (version 4) are a reasonable default for a lot of reasons. But a B-tree index pays for that randomness on every single insert, and the bill grows with the table.&lt;/p&gt;

&lt;h2&gt;
  
  
  A Library Where Every New Book Goes on a Random Shelf
&lt;/h2&gt;

&lt;p&gt;Picture two libraries.&lt;/p&gt;

&lt;p&gt;In the first, new books always go at the end of the last shelf. The librarian only needs to know where that one shelf is. When it fills up, they start a new shelf next to it. Every other shelf can sit untouched for years.&lt;/p&gt;

&lt;p&gt;In the second, every new book has to go on a specific shelf chosen at random. To add a book, the librarian walks to a random aisle. Any shelf could be next, so every shelf has to stay reachable. When a full shelf receives a book, it gets split in half to make room.&lt;/p&gt;

&lt;p&gt;A B-tree index on a &lt;code&gt;bigint&lt;/code&gt; identity column is the first library. A B-tree on &lt;code&gt;gen_random_uuid()&lt;/code&gt; is the second.&lt;/p&gt;

&lt;p&gt;With sequential keys, every insert goes to the rightmost leaf page. PostgreSQL even has a fast path for this: since version 11 it caches the rightmost leaf and skips the tree descent entirely when the new key belongs there. The hot part of the index is one page plus the path to it.&lt;/p&gt;

&lt;p&gt;With random keys, inserts are spread across the leaf level of the index. The upper levels of the tree are small and stay hot, but leaf pages make up well over 99% of a large B-tree, and all of them become the working set. Once that leaf working set grows much larger than the useful portion of &lt;code&gt;shared_buffers&lt;/code&gt;, inserts are increasingly likely to need a page that isn't cached, and random key generation turns into extra I/O.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Why this matters in production:&lt;/strong&gt; the trouble starts not at a particular row count but when the leaf working set outgrows memory. A table can run happily for a year, then degrade over a few weeks as the index gets bigger than your cache can comfortably hold. Nothing in your code changed.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Benchmark
&lt;/h2&gt;

&lt;p&gt;I loaded 10 million rows into three identical tables on PostgreSQL 18.6. The only difference was the primary key:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;t_bigint&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="nb"&gt;bigint&lt;/span&gt; &lt;span class="k"&gt;GENERATED&lt;/span&gt; &lt;span class="n"&gt;ALWAYS&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="k"&gt;IDENTITY&lt;/span&gt; &lt;span class="k"&gt;PRIMARY&lt;/span&gt; &lt;span class="k"&gt;KEY&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;payload&lt;/span&gt; &lt;span class="nb"&gt;text&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;t_v4&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="n"&gt;uuid&lt;/span&gt; &lt;span class="k"&gt;PRIMARY&lt;/span&gt; &lt;span class="k"&gt;KEY&lt;/span&gt; &lt;span class="k"&gt;DEFAULT&lt;/span&gt; &lt;span class="n"&gt;gen_random_uuid&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="n"&gt;payload&lt;/span&gt; &lt;span class="nb"&gt;text&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;t_v7&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="n"&gt;uuid&lt;/span&gt; &lt;span class="k"&gt;PRIMARY&lt;/span&gt; &lt;span class="k"&gt;KEY&lt;/span&gt; &lt;span class="k"&gt;DEFAULT&lt;/span&gt; &lt;span class="n"&gt;uuidv7&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="n"&gt;payload&lt;/span&gt; &lt;span class="nb"&gt;text&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Rows went in as 1,000 committed batches of 10,000, with &lt;code&gt;shared_buffers = 128MB&lt;/code&gt; so the indexes would outgrow memory the way a real production index eventually does. Before each run I issued a &lt;code&gt;CHECKPOINT&lt;/code&gt; and reset the WAL statistics.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;bigint identity&lt;/th&gt;
&lt;th&gt;UUIDv4&lt;/th&gt;
&lt;th&gt;UUIDv7&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Load time&lt;/td&gt;
&lt;td&gt;22.2s&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;84.2s&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;23.2s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;WAL generated&lt;/td&gt;
&lt;td&gt;1,581 MB&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;2,523 MB&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;1,715 MB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Full-page images&lt;/td&gt;
&lt;td&gt;45&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;121,490&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;45&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Primary key index size&lt;/td&gt;
&lt;td&gt;214 MB&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;387 MB&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;301 MB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Avg leaf density&lt;/td&gt;
&lt;td&gt;90.0%&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;70.1%&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;90.0%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Index blocks read from outside shared_buffers&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;3,151,242&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;In this workload, UUIDv4 took 3.8x as long as bigint and 3.6x as long as UUIDv7, generated 60% more WAL than bigint, and produced an index 29% larger than UUIDv7. UUIDv4 and UUIDv7 are the same 16-byte data type, with the same row count on the same hardware. The only difference is the order the keys arrive in.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Benchmark scope:&lt;/strong&gt; PostgreSQL generated the UUIDv7 values itself, from a single backend, so its near-perfect locality reflects PostgreSQL's native, backend-local monotonic generation. A workload where many application servers generate UUIDv7 independently will see more interleaving, because their clocks and generators differ.&lt;/p&gt;

&lt;p&gt;Three separate mechanisms produce that gap.&lt;/p&gt;

&lt;h2&gt;
  
  
  Cost 1: The Index Stops Fitting in Memory
&lt;/h2&gt;

&lt;p&gt;Look at the last row. The bigint and UUIDv7 indexes were read from outside shared buffers once in total. The UUIDv4 index was read 3.15 million times.&lt;/p&gt;

&lt;p&gt;Those reads are consistent with inserts repeatedly needing leaf pages that were no longer resident in shared buffers, forcing PostgreSQL to fetch them again. The workload only ever adds new rows, yet the buffer cache is thrashing, because those new rows land on a randomly distributed set of leaf pages.&lt;/p&gt;

&lt;p&gt;On my laptop most of those reads were served by the operating system's page cache and a local NVMe drive. On cloud block storage, where a miss is a network round trip, I'd expect the gap to be wider, not narrower.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Why this matters in production:&lt;/strong&gt; the hit ratio drop also hurts your read queries. Random-key inserts push the pages your SELECTs need out of shared buffers, so lookups that were fast get slower even though nobody touched them.&lt;/p&gt;

&lt;h2&gt;
  
  
  Cost 2: Half-Empty Pages
&lt;/h2&gt;

&lt;p&gt;Leaf density tells the second story. Sequential inserts grow the right edge of the index. PostgreSQL's default B-tree fillfactor is 90%, and that target is used when building an index and when splitting the rightmost page. The page left behind is never touched again, so it stays about 90% full.&lt;/p&gt;

&lt;p&gt;Random inserts eventually split pages in the middle of the tree, and for those PostgreSQL divides the entries roughly evenly between the two pages. Each split leaves two half-full pages, and future random inserts only partly refill them. Under a sustained random-insert workload, average occupancy settles well below the fillfactor; here it landed at 70.1%.&lt;/p&gt;

&lt;p&gt;That's the extra index size: the same keys spread across about 30% more pages. More pages means more memory needed to cache the index, which makes Cost 1 worse. The two feed each other.&lt;/p&gt;

&lt;h2&gt;
  
  
  Cost 3: Full-Page Writes Inflate WAL
&lt;/h2&gt;

&lt;p&gt;This is the one most people never connect to their key type.&lt;/p&gt;

&lt;p&gt;PostgreSQL writes every change to the write-ahead log before touching the data file. That's the aircraft black box that makes crash recovery possible. But there's a catch. If the server crashes halfway through writing an 8KB page to disk, the page could be half old and half new, and a small WAL record can't repair a torn page.&lt;/p&gt;

&lt;p&gt;So with &lt;code&gt;full_page_writes = on&lt;/code&gt; (the default, and it should stay on), the &lt;strong&gt;first&lt;/strong&gt; modification of any page after a checkpoint writes the entire 8KB page image into WAL. Later changes to that page in the same checkpoint cycle only log the small delta.&lt;/p&gt;

&lt;p&gt;The benchmark recorded 45 full-page images in each of the bigint and UUIDv7 runs, across 10 million rows. The point isn't that sequential inserts produce zero full-page images. What matters is how many &lt;em&gt;distinct&lt;/em&gt; existing pages get modified for the first time after a checkpoint. Sequential workloads keep modifying a small hot set of pages at the right edge. Random UUIDv4 inserts spread those first modifications across the whole leaf level.&lt;/p&gt;

&lt;p&gt;The UUIDv4 run recorded 121,490 full-page images, compared with 45 in the bigint run. At 8 KiB per uncompressed page image, the difference represents roughly 950 MiB of page-image payload, remarkably close to the roughly 942 MB difference in total WAL between UUIDv4 and bigint. The two numbers aren't expected to match exactly, since &lt;code&gt;wal_fpi&lt;/code&gt; counts full-page images across the whole cluster and WAL also carries record headers and other records. But the three runs were otherwise identical, so the result strongly indicates that extra full-page images are the major contributor to the WAL gap.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Why this matters in production:&lt;/strong&gt; WAL has to be generated and persisted according to the server's durability settings. In a replicated or archived system it also adds to the data that must be shipped, replayed, and retained. So random keys don't just affect the primary. In a write-heavy workload they can increase replication traffic, replay work on replicas, and WAL archive storage. The effect peaks right after each checkpoint, when every page becomes "first touch" again, which is one reason lag charts on these systems often have a sawtooth shape.&lt;/p&gt;

&lt;h2&gt;
  
  
  UUIDv7: Time First, Random Second
&lt;/h2&gt;

&lt;p&gt;UUIDv7 (standardized in RFC 9562) keeps the 128-bit UUID format but changes the layout. The first 48 bits are a Unix timestamp in milliseconds. After the version and variant bits, the rest is random (or, in some implementations including PostgreSQL's, partly extra timestamp precision).&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;uuidv7&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
&lt;span class="c1"&gt;-- 01a10f1f-702c-75ef-9838-f47c570df364&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Because the timestamp occupies the most significant portion of the UUID, UUIDv7 values generated later generally sort later when they come from the same clock source. To a B-tree that looks almost exactly like a sequence: inserts go to the right edge, pages fill densely, and very few existing pages are touched per checkpoint. The benchmark shows it: UUIDv7 landed within a few percent of bigint on load time, matched it on full-page images and leaf density, and its only real penalty is a larger index because each key is 16 bytes instead of 8.&lt;/p&gt;

&lt;p&gt;You still keep everything that made UUIDs attractive in the first place. Application servers, mobile clients, or separate services can generate IDs without asking the database. Different services or shards can generate IDs independently with an extremely low probability of collision. You can create the ID before the INSERT, which makes idempotent retries and outbox patterns simpler.&lt;/p&gt;

&lt;p&gt;PostgreSQL 18 ships &lt;code&gt;uuidv7()&lt;/code&gt; natively, so you no longer need an extension or application-side generation. Its implementation uses the 12 bits immediately after the millisecond timestamp for sub-millisecond precision, helping UUIDs generated by the same backend remain ordered even when many are created within the same millisecond. It also pairs with &lt;code&gt;uuid_extract_timestamp()&lt;/code&gt;, which pulls the creation time back out of the ID.&lt;/p&gt;

&lt;h2&gt;
  
  
  What UUIDv7 Still Costs
&lt;/h2&gt;

&lt;p&gt;It's much better, but it isn't free, and pretending otherwise is how teams get surprised later.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;It leaks creation time.&lt;/strong&gt; Anyone who sees a UUIDv7 can recover the time value encoded in it, with millisecond precision in the standard layout, which reveals roughly when the record was created. If your IDs show up in public URLs, a competitor can estimate your signup rate by creating two accounts a week apart. For user-facing identifiers where that matters, expose a separate random token and keep the v7 key internal.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;It's still 16 bytes.&lt;/strong&gt; The benchmark's heap was 805 MB for both UUID tables against 730 MB for bigint, and the UUIDv7 index was 41% larger than bigint's. That cost repeats in every foreign key column and every index that includes the key. A table referenced by five others pays it six times.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The right edge is a single hot spot.&lt;/strong&gt; Sequential keys of any kind concentrate inserts on one leaf page. At very high concurrency, sessions can queue for the lock on that page. This affects bigint identity too, and it takes serious write concurrency before it shows up, but it's the trade you make for locality.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Order is not guaranteed across independent generators.&lt;/strong&gt; If several application servers generate IDs, clock skew between them means IDs are roughly time-ordered, not strictly ordered. Inserts land near the right edge instead of exactly on it. For an index that's fine. For logic that assumes "larger ID means created later," it isn't.&lt;/p&gt;

&lt;h2&gt;
  
  
  Which Key Should You Use?
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;bigint GENERATED ALWAYS AS IDENTITY&lt;/code&gt;&lt;/strong&gt; : one database generates every ID and the IDs never need to be unguessable. Smallest, fastest, and the default unless you have a reason not to.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;UUIDv7&lt;/strong&gt; : IDs are created outside the database (clients, multiple services, offline-first apps), data from several shards or regions needs to merge, or you need the ID before the row exists.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;UUIDv4&lt;/strong&gt; : as a separate, opaque public identifier where unpredictability helps, such as resource URLs or share links, stored in a column with its own index. Not as the primary key of a high-write table. And not as an authentication token: keep those as dedicated security credentials.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Already Running on UUIDv4?
&lt;/h2&gt;

&lt;p&gt;You don't need to rewrite the table. Because UUIDv4 is just a 128-bit value, you can switch the default:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;orders&lt;/span&gt; &lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;COLUMN&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="k"&gt;DEFAULT&lt;/span&gt; &lt;span class="n"&gt;uuidv7&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Old rows keep their random IDs and still sort among them. New rows all carry timestamps from right now, so they fall into one narrow slice of the key range and concentrate their inserts on a small group of pages instead of the whole index. The existing index won't shrink or repack itself. If you want the density back, rebuild it in a quiet window without blocking writes:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;REINDEX&lt;/span&gt; &lt;span class="k"&gt;INDEX&lt;/span&gt; &lt;span class="n"&gt;CONCURRENTLY&lt;/span&gt; &lt;span class="n"&gt;orders_pkey&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It needs extra disk space and takes longer than a plain &lt;code&gt;REINDEX&lt;/code&gt;, and if it fails partway it leaves an invalid index behind that you have to drop. Measure WAL volume and full-page images before and after so you know what you actually gained.&lt;/p&gt;

&lt;p&gt;To see where your own tables stand today:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="c1"&gt;-- How full are your primary key leaf pages?&lt;/span&gt;
&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="n"&gt;EXTENSION&lt;/span&gt; &lt;span class="n"&gt;IF&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;EXISTS&lt;/span&gt; &lt;span class="n"&gt;pgstattuple&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;avg_leaf_density&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;leaf_fragmentation&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;pgstatindex&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'orders_pkey'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="c1"&gt;-- How much of your WAL is full-page images?&lt;/span&gt;
&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;wal_records&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;wal_fpi&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;pg_size_pretty&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;wal_bytes&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;pg_stat_wal&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="c1"&gt;-- Is the index being served from memory?&lt;/span&gt;
&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;idx_blks_hit&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;idx_blks_read&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;pg_statio_user_indexes&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;indexrelname&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'orders_pkey'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A leaf density near 70% on a primary key, unusually high &lt;code&gt;wal_fpi&lt;/code&gt; compared with a sequential-key baseline, and a growing &lt;code&gt;idx_blks_read&lt;/code&gt; on an insert-heavy table are the fingerprint of random keys.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Takeaway
&lt;/h2&gt;

&lt;p&gt;A primary key is not just an identifier. It decides where every new row lands in the index, and that decides how much of the index has to stay in memory, how full its pages stay, and how much WAL every insert writes.&lt;/p&gt;

&lt;p&gt;Random keys spread that work across the whole index. Time-ordered keys keep it in one place. With &lt;code&gt;uuidv7()&lt;/code&gt; built into PostgreSQL 18, you no longer have to choose between distributed ID generation and an index that ages well.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Benchmark setup: PostgreSQL 18.6 on an Apple M2 Pro (16 GB RAM, local NVMe) in a fresh, otherwise idle cluster, &lt;code&gt;shared_buffers=128MB&lt;/code&gt;, &lt;code&gt;max_wal_size=1GB&lt;/code&gt;, defaults for everything else (including &lt;code&gt;full_page_writes&lt;/code&gt;, &lt;code&gt;wal_compression&lt;/code&gt;, &lt;code&gt;synchronous_commit&lt;/code&gt; and fillfactor), 1,000 committed batches of 10,000 rows, &lt;code&gt;CHECKPOINT&lt;/code&gt; and WAL stats reset before each run. Absolute times will vary on other hardware. The ratios are the point.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>postgres</category>
      <category>database</category>
      <category>performance</category>
      <category>indexing</category>
    </item>
    <item>
      <title>The TOAST Tax: How Unbounded JSONB Arrays Cost My Blockchain Indexer 5TB</title>
      <dc:creator>Jatin Jain Saraf</dc:creator>
      <pubDate>Mon, 05 Oct 2026 03:33:07 +0000</pubDate>
      <link>https://dev.to/jatinjainsaraf/the-toast-tax-how-unbounded-jsonb-arrays-cost-my-blockchain-indexer-5tb-33f1</link>
      <guid>https://dev.to/jatinjainsaraf/the-toast-tax-how-unbounded-jsonb-arrays-cost-my-blockchain-indexer-5tb-33f1</guid>
      <description>&lt;p&gt;&lt;em&gt;How PostgreSQL TOAST turned unbounded JSONB arrays into a 5TB storage problem, and how we moved the hot path out without taking the indexer offline.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Our blockchain indexer's biggest PostgreSQL table looked healthy. Row counts made sense, the queries were simple and the indexes were in place. Then we looked at disk usage, and the numbers didn't add up. The table was taking terabytes more than its rows could explain.&lt;/p&gt;

&lt;p&gt;The space wasn't in the table's own pages at all. It was in a second table we had never created, never queried and never named: &lt;code&gt;pg_toast.pg_toast_&amp;lt;oid&amp;gt;&lt;/code&gt;. That's where PostgreSQL had been putting our JSONB arrays and objects. At its peak, &lt;code&gt;pg_total_relation_size()&lt;/code&gt; for that TOAST relation was &lt;strong&gt;over 5TB&lt;/strong&gt; , counting live data, dead tuples and its index.&lt;/p&gt;

&lt;h2&gt;
  
  
  TL;DR
&lt;/h2&gt;

&lt;p&gt;Large JSONB documents don't just live in your table. PostgreSQL may compress them and move them into a separate TOAST table. That's usually invisible, until your documents get large and your workload keeps querying individual fields inside them.&lt;/p&gt;

&lt;p&gt;In our indexer, unbounded arrays made this much worse. A bigger JSONB index wasn't the fix. We pulled the hot fields into normal columns, moved one-to-many data into child tables, and kept JSONB for the flexible data that actually benefits from it.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;What actually caused the 5TB?&lt;/strong&gt; Not TOAST. TOAST was PostgreSQL doing exactly what it was designed to do. The cause was putting unbounded one-to-many data inside a single JSONB value, then rewriting that value again and again. TOAST hid the problem. The schema created it.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  The Setup: Why JSONB Felt Like the Right Call
&lt;/h2&gt;

&lt;p&gt;If you've built an indexer for a Move-based chain, you know the pull. Resources have different shapes. Event payloads change by event type. A single transaction can carry a list of arguments, a list of events and a list of state changes, and each list can be any length.&lt;/p&gt;

&lt;p&gt;Modeling all of that relationally means dozens of tables before you've shipped anything. JSONB lets you put the whole decoded transaction in one column and move on:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;transactions&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="n"&gt;tx_hash&lt;/span&gt; &lt;span class="n"&gt;bytea&lt;/span&gt; &lt;span class="k"&gt;PRIMARY&lt;/span&gt; &lt;span class="k"&gt;KEY&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;block_height&lt;/span&gt; &lt;span class="nb"&gt;bigint&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;payload&lt;/span&gt; &lt;span class="n"&gt;jsonb&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt; &lt;span class="c1"&gt;-- sender, function, args[], events[], changes[]&lt;/span&gt;
&lt;span class="p"&gt;);&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;I wrote about this tradeoff in &lt;a href="https://insight.jatinjainsaraf.com/your-blockchain-indexer-is-fine-until-it-isnt" rel="noopener noreferrer"&gt;Your Blockchain Indexer Is Fine, Until It Isn't&lt;/a&gt;. JSONB really is useful for an indexer. What I underestimated was what happens when the documents inside it get big, and arrays are exactly what makes them big.&lt;/p&gt;

&lt;h2&gt;
  
  
  TOAST in One Analogy: The Off-Site Storage Unit
&lt;/h2&gt;

&lt;p&gt;PostgreSQL stores rows in 8KB pages, and a row (a heap tuple, in PostgreSQL terms) can't span pages. When a row is wider than &lt;code&gt;TOAST_TUPLE_THRESHOLD&lt;/code&gt; (roughly &lt;strong&gt;2KB&lt;/strong&gt; on a default 8KB page), PostgreSQL hands it to TOAST (The Oversized-Attribute Storage Technique) to shrink it.&lt;/p&gt;

&lt;p&gt;Think of it as an &lt;strong&gt;off-site storage unit&lt;/strong&gt; :&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;First, PostgreSQL tries to &lt;strong&gt;compress&lt;/strong&gt; the large values so the row fits again. That's like vacuum-packing your clothes so they still fit in the closet.&lt;/li&gt;
&lt;li&gt;If the row is still too big, PostgreSQL &lt;strong&gt;moves values out of the row&lt;/strong&gt; into a separate TOAST table, cut into chunks of about 2KB each.&lt;/li&gt;
&lt;li&gt;The row keeps a small &lt;strong&gt;TOAST pointer&lt;/strong&gt; (18 bytes on disk), which works like a claim ticket for the storage unit.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;A common misconception is that TOAST only kicks in above 8KB. It doesn't. The roughly 2KB threshold applies to the whole row, so one fat &lt;code&gt;events[]&lt;/code&gt; array is often enough to push a transaction over it.&lt;/p&gt;

&lt;p&gt;This is great for keeping rows small, and it's why PostgreSQL can store values far bigger than a page at all. The catch is what happens when you need something back from the storage unit.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Want the full internals: chunk layout, storage strategies, and how the toaster picks which column to move first? The &lt;a href="https://academy.jatinjainsaraf.com/postgresql-in-depth/storage-engine" rel="noopener noreferrer"&gt;Storage Engine module&lt;/a&gt; of the PostgreSQL course covers them.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Cost #1: Reading One Key Can Mean Detoasting the Document
&lt;/h2&gt;

&lt;p&gt;This was the query we ran most often:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;tx_hash&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;block_height&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;transactions&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;payload&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&amp;gt;&lt;/span&gt;&lt;span class="s1"&gt;'sender'&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;
&lt;span class="k"&gt;ORDER&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;block_height&lt;/span&gt; &lt;span class="k"&gt;DESC&lt;/span&gt;
&lt;span class="k"&gt;LIMIT&lt;/span&gt; &lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It looks cheap because we only want one small field. But to evaluate &lt;code&gt;payload-&amp;gt;&amp;gt;'sender'&lt;/code&gt;, PostgreSQL generally has to &lt;strong&gt;detoast&lt;/strong&gt; the JSONB value first. For a large, compressed, out-of-line document, that can mean reading its TOAST chunks through the TOAST table's index, rebuilding the value and decompressing it before JSONB can look at its binary layout.&lt;/p&gt;

&lt;p&gt;JSONB's binary format really is fast to search, but only once the value is in memory. PostgreSQL doesn't jump straight to the bytes for &lt;code&gt;sender&lt;/code&gt; inside the TOAST chunks. The trip to the storage unit happens first, for every row the filter touches. In &lt;a href="https://www.snowflake.com/en/blog/engineering/postgres-jsonb-columns-and-toast/" rel="noopener noreferrer"&gt;one benchmark from Snowflake's engineering team&lt;/a&gt;, filtering on one JSONB key in a 10,000-row table took under 10ms when the documents were 40 bytes, and about 500ms (40x slower) when they were 40KB and TOASTed. Your numbers will differ, but the direction matches what we saw.&lt;/p&gt;

&lt;p&gt;Two details make it worse in practice:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Projecting several fields adds up.&lt;/strong&gt; Every JSONB expression still has to work with the underlying value, so pulling five fields out of a large document can add a lot of CPU and memory work compared to reading five plain columns.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The planner is guessing.&lt;/strong&gt; PostgreSQL keeps no statistics on keys inside a JSONB document. For &lt;code&gt;payload-&amp;gt;&amp;gt;'sender' = $1&lt;/code&gt;, it falls back to a hardcoded selectivity estimate, so its row count predictions can be badly wrong, and so can the plans built on them.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;What this looks like in production:&lt;/strong&gt; queries that filter on one field are slow even when they return a handful of rows, CPU stays high on a read replica with no obvious hot query, and an index on &lt;code&gt;block_height&lt;/code&gt; doesn't help as much as you expect.&lt;/p&gt;

&lt;h2&gt;
  
  
  Cost #2: Updating One JSONB Key Can Rewrite the Whole Value
&lt;/h2&gt;

&lt;p&gt;Indexers update data. A reorg rewrites a block, a backfill adds a decoded field, a bug fix changes a format. For us it usually looked like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;UPDATE&lt;/span&gt; &lt;span class="n"&gt;transactions&lt;/span&gt;
&lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;payload&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;jsonb_set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;payload&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'{status}'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;'"finalized"'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;block_height&lt;/span&gt; &lt;span class="k"&gt;BETWEEN&lt;/span&gt; &lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;jsonb_set&lt;/code&gt; looks like a partial update, and at the SQL level it is. Physically, though, PostgreSQL doesn't patch the changed bytes in place. Changing one key in a large TOASTed document produces a &lt;strong&gt;brand-new value&lt;/strong&gt; , so you get new TOAST chunks, new TOAST index entries and new WAL for all of it. The old chunks stay behind as dead tuples until vacuum gets to them.&lt;/p&gt;

&lt;p&gt;One nuance worth knowing: if an &lt;code&gt;UPDATE&lt;/code&gt; &lt;strong&gt;doesn't touch&lt;/strong&gt; the TOASTed column, PostgreSQL reuses the existing pointer and leaves the value alone. Updating &lt;code&gt;block_height&lt;/code&gt; by itself is cheap. Changing &lt;code&gt;payload&lt;/code&gt; in any way is not.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What this looks like in production:&lt;/strong&gt; WAL volume spikes during backfills, replica lag climbs (the same problem I described in &lt;a href="https://insight.jatinjainsaraf.com/taming-postgresql-replication-lag-in-real-time-blockchain-indexers" rel="noopener noreferrer"&gt;Taming PostgreSQL Replication Lag in Real-Time Blockchain Indexers&lt;/a&gt;), and disk keeps growing after the backfill is done.&lt;/p&gt;

&lt;h2&gt;
  
  
  Cost #3: The Storage Unit Needs Its Own Cleaning Crew
&lt;/h2&gt;

&lt;p&gt;The TOAST table is a real table. It gets dead tuples, it needs autovacuum, it bloats, and it has its own index that bloats along with it. Because it's hidden, nobody watches it. Your dashboards say &lt;code&gt;transactions&lt;/code&gt; is healthy while &lt;code&gt;pg_toast_16421&lt;/code&gt; quietly grows by hundreds of gigabytes. (For how vacuum and WAL work in the background, see &lt;a href="https://insight.jatinjainsaraf.com/postgres-wal-checkpoints-wait-events-vacuum-analyze-explained" rel="noopener noreferrer"&gt;Every Postgres Write Triggers Five Background Processes&lt;/a&gt;.)&lt;/p&gt;

&lt;p&gt;That's how our TOAST relation got past 5TB. No single decision did it. It was every array that grew a little, every backfill that rewrote documents, and every vacuum cycle that couldn't keep up.&lt;/p&gt;

&lt;h2&gt;
  
  
  How to Check Whether You Have the Same Problem
&lt;/h2&gt;

&lt;p&gt;You can run all of these on your own database.&lt;/p&gt;

&lt;h3&gt;
  
  
  1. Find your biggest TOAST tables
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt;
  &lt;span class="k"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;relname&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="k"&gt;table_name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;pg_size_pretty&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pg_relation_size&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;oid&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;main_heap&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;pg_size_pretty&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pg_indexes_size&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;oid&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;main_indexes&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;pg_size_pretty&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pg_relation_size&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;reltoastrelid&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;toast_heap&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;pg_size_pretty&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pg_indexes_size&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;reltoastrelid&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;toast_indexes&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;pg_size_pretty&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pg_total_relation_size&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;reltoastrelid&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;toast_total&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;pg_size_pretty&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pg_total_relation_size&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;oid&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;total&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;pg_class&lt;/span&gt; &lt;span class="k"&gt;c&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="k"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;relkind&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'r'&lt;/span&gt;
  &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="k"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;reltoastrelid&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&amp;gt;&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
&lt;span class="k"&gt;ORDER&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;pg_total_relation_size&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;reltoastrelid&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;DESC&lt;/span&gt;
&lt;span class="k"&gt;LIMIT&lt;/span&gt; &lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The TOAST table has its own index, so &lt;code&gt;toast_total&lt;/code&gt; is the number that shows the real off-site footprint. If &lt;code&gt;toast_total&lt;/code&gt; is close to or larger than &lt;code&gt;main_heap&lt;/code&gt;, a big share of the table's physical footprint lives out of line.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Measure your JSONB sizes
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;percentile_cont&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ARRAY&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;9&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;99&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
       &lt;span class="n"&gt;WITHIN&lt;/span&gt; &lt;span class="k"&gt;GROUP&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;ORDER&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;pg_column_size&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;payload&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;p50_p90_p99_bytes&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;transactions&lt;/span&gt; &lt;span class="n"&gt;TABLESAMPLE&lt;/span&gt; &lt;span class="k"&gt;SYSTEM&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;pg_column_size()&lt;/code&gt; reports the size of the value as stored, after any compression. That makes it useful for seeing how big your documents really are on disk. Don't treat 2KB as a hard "TOAST or no TOAST" line for a single column, though. TOAST looks at the size of the whole row, the column's storage strategy, and whether compression can bring the row back under the target size. Look at these percentiles together with &lt;code&gt;toast_total&lt;/code&gt; from query 1 to see what's actually happening.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. Check compression
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;pg_column_compression&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;payload&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="k"&gt;method&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;count&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;transactions&lt;/span&gt; &lt;span class="n"&gt;TABLESAMPLE&lt;/span&gt; &lt;span class="k"&gt;SYSTEM&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;GROUP&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;NULL&lt;/code&gt; means the value isn't compressed. Otherwise you'll see &lt;code&gt;pglz&lt;/code&gt; or &lt;code&gt;lz4&lt;/code&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  4. EXPLAIN ANALYZE, measured honestly
&lt;/h3&gt;

&lt;p&gt;Run &lt;code&gt;EXPLAIN (ANALYZE, BUFFERS)&lt;/code&gt; on your hot queries. When the JSONB is used in a filter, TOAST reads show up in the buffer counts. There's a gotcha: plain &lt;code&gt;EXPLAIN ANALYZE&lt;/code&gt; doesn't detoast columns you only &lt;em&gt;return&lt;/em&gt;, so &lt;code&gt;SELECT payload ...&lt;/code&gt; looks faster than it is. On PostgreSQL 17 and later, add &lt;code&gt;SERIALIZE&lt;/code&gt; to include that cost:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;EXPLAIN&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;ANALYZE&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;BUFFERS&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;SERIALIZE&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;payload&lt;/span&gt; &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;transactions&lt;/span&gt; &lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;block_height&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;1234567&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  The Fixes, From Cheapest to Most Work
&lt;/h2&gt;

&lt;h3&gt;
  
  
  1. Switch to lz4
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;transactions&lt;/span&gt; &lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;COLUMN&lt;/span&gt; &lt;span class="n"&gt;payload&lt;/span&gt; &lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;COMPRESSION&lt;/span&gt; &lt;span class="n"&gt;lz4&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;lz4&lt;/code&gt; decompresses much faster than the older &lt;code&gt;pglz&lt;/code&gt; default, at a similar compression ratio. PostgreSQL 19 changes the default TOAST compression to &lt;code&gt;lz4&lt;/code&gt; on builds with LZ4 support, as I covered in &lt;a href="https://insight.jatinjainsaraf.com/postgresql-19-vs-18-what-actually-changed-feature-by-feature" rel="noopener noreferrer"&gt;PostgreSQL 19 vs. 18&lt;/a&gt;. It's the cheapest change here, but it &lt;strong&gt;only affects newly written values&lt;/strong&gt;. Existing rows keep their old compression until they're rewritten. Each trip to the storage unit gets faster, but you still make the trip.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Extract hot keys into real columns
&lt;/h3&gt;

&lt;p&gt;This is the fix that matters most. If you filter, sort or join on &lt;code&gt;sender&lt;/code&gt; all the time, it shouldn't live inside a document.&lt;/p&gt;

&lt;p&gt;On a small or medium table, a stored generated column is the cleanest option:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;transactions&lt;/span&gt;
  &lt;span class="k"&gt;ADD&lt;/span&gt; &lt;span class="k"&gt;COLUMN&lt;/span&gt; &lt;span class="n"&gt;sender&lt;/span&gt; &lt;span class="nb"&gt;text&lt;/span&gt; &lt;span class="k"&gt;GENERATED&lt;/span&gt; &lt;span class="n"&gt;ALWAYS&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;payload&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&amp;gt;&lt;/span&gt;&lt;span class="s1"&gt;'sender'&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;STORED&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;INDEX&lt;/span&gt; &lt;span class="n"&gt;CONCURRENTLY&lt;/span&gt; &lt;span class="n"&gt;idx_tx_sender&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;transactions&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sender&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;PostgreSQL keeps the column in sync, it's normally stored inline with the row, it gets real planner statistics, and reading the small extracted value doesn't require detoasting the original JSONB document. In the same Snowflake benchmark, filtering on a stored generated column took 5ms, against 500ms for the same filter on the TOASTed JSONB.&lt;/p&gt;

&lt;p&gt;It isn't free, though. Every write to &lt;code&gt;payload&lt;/code&gt; now also computes and stores &lt;code&gt;sender&lt;/code&gt;, and the column and its index take disk. You're paying a small write cost for a big read saving, which is the right trade for a key you read constantly.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;PostgreSQL 18 gotcha:&lt;/strong&gt; generated columns are now &lt;code&gt;VIRTUAL&lt;/code&gt; by default, which means the value is computed on read. If your goal is to get a frequently queried JSONB key out of TOAST, write &lt;code&gt;STORED&lt;/code&gt; explicitly. A virtual column detoasts the document every time it's read, which is exactly what you're trying to avoid.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h3&gt;
  
  
  3. Expression indexes for known lookups
&lt;/h3&gt;

&lt;p&gt;If you can't add a column yet, an expression index stores the extracted value in the index itself:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;INDEX&lt;/span&gt; &lt;span class="n"&gt;CONCURRENTLY&lt;/span&gt; &lt;span class="n"&gt;idx_tx_payload_sender&lt;/span&gt;
  &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;transactions&lt;/span&gt; &lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;payload&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&amp;gt;&lt;/span&gt;&lt;span class="s1"&gt;'sender'&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A lookup by sender can now find matching rows through the index without opening the storage unit, as long as you don't also &lt;code&gt;SELECT payload&lt;/code&gt;. The index fixes the lookup path, not the cost of fetching the document itself. PostgreSQL also collects statistics on indexed expressions, so the planner gets real data to estimate with instead of a hardcoded guess. The downsides are the same as any index (write overhead and disk), plus the query has to use &lt;strong&gt;exactly&lt;/strong&gt; the same expression for the index to be used.&lt;/p&gt;

&lt;p&gt;This works best for exact matches. For a prefix search like &lt;code&gt;LIKE 'a%'&lt;/code&gt;, PostgreSQL uses the index to narrow down the rows, then re-applies the original condition to each one, which means detoasting again. Snowflake measured exactly this: 80ms through an expression index, against 5ms on a real column.&lt;/p&gt;

&lt;h3&gt;
  
  
  4. GIN, for what it's good at
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;INDEX&lt;/span&gt; &lt;span class="n"&gt;CONCURRENTLY&lt;/span&gt; &lt;span class="n"&gt;idx_tx_payload_gin&lt;/span&gt;
  &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;transactions&lt;/span&gt; &lt;span class="k"&gt;USING&lt;/span&gt; &lt;span class="n"&gt;GIN&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;payload&lt;/span&gt; &lt;span class="n"&gt;jsonb_path_ops&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;tx_hash&lt;/span&gt; &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;transactions&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;payload&lt;/span&gt; &lt;span class="o"&gt;@&amp;gt;&lt;/span&gt; &lt;span class="s1"&gt;'{"function": "0x1::coin::transfer"}'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;GIN is good at containment questions like "which documents contain this key and value?" It doesn't make &lt;strong&gt;extracting&lt;/strong&gt; a value any cheaper. &lt;code&gt;jsonb_path_ops&lt;/code&gt; stores hashes, so candidate rows get rechecked against the real document, and that recheck detoasts. On large documents, GIN indexes also get very large themselves. Use GIN when you really don't know the query shape ahead of time, not as a replacement for real columns on hot keys.&lt;/p&gt;

&lt;h3&gt;
  
  
  5. Normalize unbounded arrays
&lt;/h3&gt;

&lt;p&gt;Everything above works around the actual mistake. &lt;strong&gt;An array that can grow without limit doesn't belong in a single column value.&lt;/strong&gt; &lt;code&gt;events[]&lt;/code&gt; is a one-to-many relationship, and relational databases already have a tool for that:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;tx_events&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="n"&gt;tx_hash&lt;/span&gt; &lt;span class="n"&gt;bytea&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;event_index&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;event_type&lt;/span&gt; &lt;span class="nb"&gt;text&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="k"&gt;data&lt;/span&gt; &lt;span class="n"&gt;jsonb&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="c1"&gt;-- small, per-event, usually well under 2KB&lt;/span&gt;
  &lt;span class="k"&gt;PRIMARY&lt;/span&gt; &lt;span class="k"&gt;KEY&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tx_hash&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;event_index&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;INDEX&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;tx_events&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;event_type&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Each event is now its own small row. A query like "all transfer events for this coin" becomes an index scan instead of unpacking thousands of documents. JSONB is still there, but sized for what it's good at: a small, flexible bag of fields per event, not a whole transaction's history in one value.&lt;/p&gt;

&lt;p&gt;If you still need the full raw payload for replays or debugging, move it out of the hot table entirely, into a narrow &lt;code&gt;tx_raw (tx_hash, payload)&lt;/code&gt; table or object storage. The hot table stays small, and the big documents are read only when someone actually needs them.&lt;/p&gt;

&lt;p&gt;Here's the shape before and after:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;BEFORE
transactions
├── tx_hash
├── block_height
└── payload (JSONB)
    ├── sender
    ├── function
    ├── args[]
    ├── events[] ← grows without limit
    └── changes[] ← grows without limit
           │
           ▼
      TOAST table (5TB+)

AFTER
transactions
├── tx_hash
├── block_height ← indexed
├── sender ← indexed, hot path, inline
└── payload (JSONB) ← cold, flexible fields only
           │
           ▼
      TOAST table (small)

tx_events
├── tx_hash
├── event_index
├── event_type ← indexed
└── data (JSONB) ← small, usually stays inline

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  The Zero-Downtime Migration
&lt;/h2&gt;

&lt;p&gt;At 5TB, don't casually add a stored generated column with &lt;code&gt;ALTER TABLE&lt;/code&gt;. PostgreSQL has to rewrite the whole table and its indexes, and the &lt;code&gt;ACCESS EXCLUSIVE&lt;/code&gt; lock that &lt;code&gt;ALTER TABLE ... ADD COLUMN&lt;/code&gt; takes is held until the transaction commits. The rewrite copies every value, TOAST included, and needs free disk for that second copy until it finishes. At this size, that's a long stretch (realistically hours) where the indexer can neither read nor write.&lt;/p&gt;

&lt;p&gt;So we did it in steps that are each cheap or non-blocking:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;add&lt;/span&gt; &lt;span class="k"&gt;nullable&lt;/span&gt; &lt;span class="k"&gt;column&lt;/span&gt; &lt;span class="err"&gt;→&lt;/span&gt; &lt;span class="n"&gt;install&lt;/span&gt; &lt;span class="k"&gt;trigger&lt;/span&gt; &lt;span class="err"&gt;→&lt;/span&gt; &lt;span class="n"&gt;backfill&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;batches&lt;/span&gt; &lt;span class="err"&gt;→&lt;/span&gt; &lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;INDEX&lt;/span&gt; &lt;span class="n"&gt;CONCURRENTLY&lt;/span&gt; &lt;span class="err"&gt;→&lt;/span&gt; &lt;span class="n"&gt;switch&lt;/span&gt; &lt;span class="k"&gt;reads&lt;/span&gt;


&lt;span class="c1"&gt;-- Defining the trigger function takes no lock on the table&lt;/span&gt;
&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;FUNCTION&lt;/span&gt; &lt;span class="n"&gt;set_tx_sender&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="k"&gt;RETURNS&lt;/span&gt; &lt;span class="k"&gt;trigger&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="err"&gt;$$&lt;/span&gt;
&lt;span class="k"&gt;BEGIN&lt;/span&gt;
  &lt;span class="k"&gt;NEW&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sender&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;NEW&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;payload&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&amp;gt;&lt;/span&gt;&lt;span class="s1"&gt;'sender'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="k"&gt;RETURN&lt;/span&gt; &lt;span class="k"&gt;NEW&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;END&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="err"&gt;$$&lt;/span&gt; &lt;span class="k"&gt;LANGUAGE&lt;/span&gt; &lt;span class="n"&gt;plpgsql&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="c1"&gt;-- Steps 1 and 2 need brief table locks: fail fast instead of queueing, then retry&lt;/span&gt;
&lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;lock_timeout&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'5s'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="c1"&gt;-- 1. Instant: a nullable column with no default is a metadata-only change&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;transactions&lt;/span&gt; &lt;span class="k"&gt;ADD&lt;/span&gt; &lt;span class="k"&gt;COLUMN&lt;/span&gt; &lt;span class="n"&gt;sender&lt;/span&gt; &lt;span class="nb"&gt;text&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="c1"&gt;-- 2. Keep new and changed rows in sync from now on&lt;/span&gt;
&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;TRIGGER&lt;/span&gt; &lt;span class="n"&gt;trg_tx_sender&lt;/span&gt;
  &lt;span class="k"&gt;BEFORE&lt;/span&gt; &lt;span class="k"&gt;INSERT&lt;/span&gt; &lt;span class="k"&gt;OR&lt;/span&gt; &lt;span class="k"&gt;UPDATE&lt;/span&gt; &lt;span class="k"&gt;OF&lt;/span&gt; &lt;span class="n"&gt;payload&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;transactions&lt;/span&gt;
  &lt;span class="k"&gt;FOR&lt;/span&gt; &lt;span class="k"&gt;EACH&lt;/span&gt; &lt;span class="k"&gt;ROW&lt;/span&gt; &lt;span class="k"&gt;EXECUTE&lt;/span&gt; &lt;span class="k"&gt;FUNCTION&lt;/span&gt; &lt;span class="n"&gt;set_tx_sender&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;

&lt;span class="k"&gt;RESET&lt;/span&gt; &lt;span class="n"&gt;lock_timeout&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="c1"&gt;-- 3. Backfill history in small block ranges, one committed transaction per range&lt;/span&gt;
&lt;span class="k"&gt;UPDATE&lt;/span&gt; &lt;span class="n"&gt;transactions&lt;/span&gt;
&lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;sender&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;payload&lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&amp;gt;&lt;/span&gt;&lt;span class="s1"&gt;'sender'&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;block_height&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="n"&gt;block_height&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;
  &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="n"&gt;sender&lt;/span&gt; &lt;span class="k"&gt;IS&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="c1"&gt;-- 4. Index without blocking writes&lt;/span&gt;
&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;INDEX&lt;/span&gt; &lt;span class="n"&gt;CONCURRENTLY&lt;/span&gt; &lt;span class="n"&gt;idx_tx_sender&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;transactions&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sender&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A few things make this work at scale:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;"Instant" still needs a lock.&lt;/strong&gt; &lt;code&gt;ALTER TABLE&lt;/code&gt; needs a brief &lt;code&gt;ACCESS EXCLUSIVE&lt;/code&gt; lock and &lt;code&gt;CREATE TRIGGER&lt;/code&gt; a &lt;code&gt;SHARE ROW EXCLUSIVE&lt;/code&gt; one. If a long-running query is holding the table, the DDL waits, and every query that arrives after it queues behind it. A short &lt;code&gt;lock_timeout&lt;/code&gt; makes it fail fast so you can retry. More on that trap in &lt;a href="https://insight.jatinjainsaraf.com/why-a-single-alter-table-can-take-down-your-whole-database" rel="noopener noreferrer"&gt;Why a Single ALTER TABLE Can Take Down Your Whole Database&lt;/a&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Order matters.&lt;/strong&gt; The trigger goes in before the backfill starts, so every row written during the migration is already correct. The backfill only has to cover history.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The backfill doesn't rewrite the existing TOAST values.&lt;/strong&gt; It reads each document from TOAST once, and after that, the &lt;code&gt;sender&lt;/code&gt; lookup no longer needs to detoast the original payload. Because it never modifies &lt;code&gt;payload&lt;/code&gt;, PostgreSQL reuses the existing TOAST pointers, and the trigger (which only fires on &lt;code&gt;UPDATE OF payload&lt;/code&gt;) stays out of the way.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Commit every batch.&lt;/strong&gt; Run the backfill from a script, one transaction per block range, never as one giant &lt;code&gt;UPDATE&lt;/code&gt;. A multi-hour transaction stops vacuum from cleaning up any row that dies while it runs, across the whole database, and if it fails at hour five, all of that work rolls back.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The heap still churns.&lt;/strong&gt; The &lt;code&gt;UPDATE&lt;/code&gt; itself (not the TOAST reads) gives every backfilled row a new tuple version in the main table, and all of that is WAL-logged. Let autovacuum (or a manual &lt;code&gt;VACUUM&lt;/code&gt;) catch up between batches, and size the batches by what you can observe, WAL volume and replica lag, rather than a round row count.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Switch reads last.&lt;/strong&gt; Once the index is built, change the application from &lt;code&gt;payload-&amp;gt;&amp;gt;'sender'&lt;/code&gt; to &lt;code&gt;sender&lt;/code&gt;, check the new plan with &lt;code&gt;EXPLAIN (ANALYZE, BUFFERS)&lt;/code&gt;, and only then think about dropping any expression index you added as a stopgap.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The same pattern (new structure, trigger for new writes, batched backfill, concurrent index, cut over reads) also works for moving &lt;code&gt;events[]&lt;/code&gt; into &lt;code&gt;tx_events&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  When JSONB Is Still the Right Answer
&lt;/h2&gt;

&lt;p&gt;None of this means JSONB is bad. It works well when:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Documents usually stay small enough that rows stay inline&lt;/li&gt;
&lt;li&gt;You read the &lt;strong&gt;whole document&lt;/strong&gt; at once, so the trip to TOAST is worth making&lt;/li&gt;
&lt;li&gt;The structure really is unknown or user-defined, and you rarely filter on it&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;It goes wrong when a document grows without limit and you keep querying single keys inside it. In many indexers, both of those show up sooner than you expect. JSONB isn't the mistake. Using it in place of modeling hot fields and unbounded relationships is.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Rule I Use Now
&lt;/h2&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;JSONB is for the shape you don't know yet. Columns are for the keys you query. Child tables are for anything that's a list.&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;PostgreSQL won't warn you when your rows start relying heavily on TOAST. It quietly starts building a storage unit, hands each row a claim ticket and keeps going. Run the TOAST size query above on your largest tables this week. If the off-site number is bigger than the table itself, you already know which fix to start with.&lt;/p&gt;

&lt;h2&gt;
  
  
  Related Articles
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://insight.jatinjainsaraf.com/your-blockchain-indexer-is-fine-until-it-isnt" rel="noopener noreferrer"&gt;Your Blockchain Indexer Is Fine, Until It Isn't&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://insight.jatinjainsaraf.com/taming-postgresql-replication-lag-in-real-time-blockchain-indexers" rel="noopener noreferrer"&gt;Taming PostgreSQL Replication Lag in Real-Time Blockchain Indexers&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://insight.jatinjainsaraf.com/why-a-single-alter-table-can-take-down-your-whole-database" rel="noopener noreferrer"&gt;Why a Single ALTER TABLE Can Take Down Your Whole Database&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://insight.jatinjainsaraf.com/postgres-wal-checkpoints-wait-events-vacuum-analyze-explained" rel="noopener noreferrer"&gt;Every Postgres Write Triggers Five Background Processes&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://insight.jatinjainsaraf.com/postgresql-async-io-engine-explained" rel="noopener noreferrer"&gt;PostgreSQL's Async I/O Engine: Why Your Sequential Scans Just Got 2-3x Faster&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://insight.jatinjainsaraf.com/postgresql-19-vs-18-what-actually-changed-feature-by-feature" rel="noopener noreferrer"&gt;PostgreSQL 19 vs. 18: What Actually Changed, Feature by Feature&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>postgres</category>
      <category>database</category>
      <category>performance</category>
      <category>jsonb</category>
    </item>
    <item>
      <title>Every Postgres Write Triggers Five Background Processes. Here's What Each One Actually Does.</title>
      <dc:creator>Jatin Jain Saraf</dc:creator>
      <pubDate>Mon, 14 Sep 2026 05:05:17 +0000</pubDate>
      <link>https://dev.to/jatinjainsaraf/every-postgres-write-triggers-five-background-processes-heres-what-each-one-actually-does-3a1n</link>
      <guid>https://dev.to/jatinjainsaraf/every-postgres-write-triggers-five-background-processes-heres-what-each-one-actually-does-3a1n</guid>
      <description>&lt;p&gt;An &lt;code&gt;UPDATE&lt;/code&gt; statement returns in two milliseconds. From where you're sitting, that's the whole story: the row changed, the client got an acknowledgment, done. Underneath, Postgres just did five separate jobs to make that write durable, recoverable, reusable, and still fast for the next query to plan correctly. Most of the time you never see any of them. Then a dashboard shows periodic latency spikes with no slow query in sight, or &lt;code&gt;pg_stat_activity&lt;/code&gt; shows a backend "stuck" on something with a name you've never had to learn, and you're suddenly debugging a part of Postgres that has been running the entire time.&lt;/p&gt;

&lt;p&gt;This is a walkthrough of that background layer: the Write-Ahead Log, checkpoints, the wait events that make both of them visible, and the vacuum/analyze cycle that cleans up after every write and keeps the query planner honest.&lt;/p&gt;

&lt;h2&gt;
  
  
  WAL: the black box that makes crash recovery possible
&lt;/h2&gt;

&lt;p&gt;Postgres never writes a change directly to a table's data file first. It writes a record of the change to the Write-Ahead Log (WAL) first, and only later, asynchronously, does the actual data file get updated. This is the same idea as an aircraft's flight data recorder: it doesn't prevent anything from going wrong, but if the plane goes down, the black box is what lets you reconstruct exactly what happened, in order, right up to the last recorded moment.&lt;/p&gt;

&lt;p&gt;If Postgres crashes between "WAL record written" and "data file updated," the data file is allowed to be behind. On restart, Postgres replays the WAL from the last confirmed checkpoint forward and reconstructs every change that was durably logged but not yet applied to the heap. That's the entire crash-recovery guarantee, and it's also the entire replication guarantee: a replica is, at its core, a process continuously replaying WAL records shipped from the primary.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Why this matters in production:&lt;/strong&gt; &lt;code&gt;synchronous_commit&lt;/code&gt; controls whether a transaction waits for its WAL record to actually hit disk before returning success to the client. Turn it off (or set it to a weaker level like &lt;code&gt;local&lt;/code&gt;) and writes get faster, because you're no longer waiting on an fsync. You're also now willing to lose the last few transactions if the server crashes before that WAL hits disk. That's a real trade-off teams make deliberately for high-throughput, loss-tolerant write paths, and a real trade-off teams make by accident when they copy a config from a benchmark blog post.&lt;/p&gt;

&lt;h2&gt;
  
  
  Checkpoints: the point WAL gets allowed to forget the past
&lt;/h2&gt;

&lt;p&gt;WAL can't grow forever, and replaying an unbounded WAL after a crash would make recovery take unbounded time. A checkpoint is Postgres periodically flushing every dirty page sitting in shared buffers out to the actual data files, so that everything the WAL described up to that point is now durably reflected on disk. Once a checkpoint completes, the WAL segments before it are no longer needed for crash recovery and can be recycled or removed (subject to what replication and archiving still need them for).&lt;/p&gt;

&lt;p&gt;Checkpoints run on a schedule controlled by two settings: &lt;code&gt;checkpoint_timeout&lt;/code&gt; (time-based, five minutes by default) and &lt;code&gt;max_wal_size&lt;/code&gt; (a soft cap on how much WAL can accumulate before a checkpoint is forced early). Whichever threshold is hit first triggers the checkpoint.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Why this matters in production:&lt;/strong&gt; a checkpoint means writing out every dirty page at once, which is a burst of I/O. If that burst happens all at once, you get the classic "checkpoint spike," periodic latency jumps that correlate with wall-clock time, not query volume. &lt;code&gt;checkpoint_completion_target&lt;/code&gt; exists specifically to spread that I/O out over the checkpoint interval instead of dumping it all at once. If your latency graphs show a sawtooth pattern with a period matching your &lt;code&gt;checkpoint_timeout&lt;/code&gt;, that's usually the first setting worth checking, not the query itself.&lt;/p&gt;

&lt;h2&gt;
  
  
  Wait events: the window into both of the above
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;pg_stat_activity&lt;/code&gt; has two columns, &lt;code&gt;wait_event_type&lt;/code&gt; and &lt;code&gt;wait_event&lt;/code&gt;, that tell you what a backend is actually blocked on right now, if it's blocked on anything. This is the diagnostic surface that turns "the query is slow" into "the query is waiting on something specific":&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;pid&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;wait_event_type&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;wait_event&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;state&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;query&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;pg_stat_activity&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;wait_event&lt;/span&gt; &lt;span class="k"&gt;IS&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;wait_event_type&lt;/code&gt; column groups events into categories: &lt;code&gt;Lock&lt;/code&gt; (waiting on a row, table, or advisory lock held by another transaction), &lt;code&gt;LWLock&lt;/code&gt; (waiting on an internal lightweight lock protecting shared memory structures), &lt;code&gt;IO&lt;/code&gt; (waiting on an actual disk read or write), &lt;code&gt;BufferPin&lt;/code&gt;, &lt;code&gt;IPC&lt;/code&gt;, &lt;code&gt;Timeout&lt;/code&gt;, and a few others. A backend showing &lt;code&gt;IO&lt;/code&gt; wait events tied to WAL activity, or an &lt;code&gt;LWLock&lt;/code&gt; wait tied to the WAL insert path, is a backend that's been caught behind the exact mechanism described above: it's trying to get its change durably logged and something (usually disk throughput, sometimes a checkpoint mid-flush) is making it wait its turn.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Why this matters in production:&lt;/strong&gt; the specific wait event names attached to WAL and checkpoint activity have changed across major Postgres versions as the I/O statistics system was reworked, so don't memorize a table of exact strings from a blog post (including this one) and assume it matches your version. Instead, learn the query above, run it against your own instance during a slow period, and read whatever &lt;code&gt;wait_event_type&lt;/code&gt;/&lt;code&gt;wait_event&lt;/code&gt; pair comes back against the current docs for your version. That habit outlives any specific release.&lt;/p&gt;

&lt;h2&gt;
  
  
  Vacuum: cleaning up after MVCC, not just reclaiming disk
&lt;/h2&gt;

&lt;p&gt;Postgres never overwrites a row in place on &lt;code&gt;UPDATE&lt;/code&gt; or &lt;code&gt;DELETE&lt;/code&gt;. MVCC (multi-version concurrency control) means an &lt;code&gt;UPDATE&lt;/code&gt; writes a new row version and marks the old one dead, and a &lt;code&gt;DELETE&lt;/code&gt; just marks a row dead, because other transactions that started before yours might still need to see the old version. Those dead row versions don't disappear on their own. &lt;code&gt;VACUUM&lt;/code&gt; is the process that finds them, confirms no transaction still needs them, and marks that space reusable by future inserts on the same table.&lt;/p&gt;

&lt;p&gt;Plain &lt;code&gt;VACUUM&lt;/code&gt; does not shrink the file on disk (only &lt;code&gt;VACUUM FULL&lt;/code&gt; does, by rewriting the whole table, which takes an exclusive lock). It reclaims space &lt;em&gt;within&lt;/em&gt; the existing file for future rows to reuse. &lt;code&gt;autovacuum&lt;/code&gt; runs this automatically, table by table, once the number of dead tuples crosses a threshold (&lt;code&gt;autovacuum_vacuum_threshold&lt;/code&gt; plus a percentage of the table's row count, &lt;code&gt;autovacuum_vacuum_scale_factor&lt;/code&gt;, which defaults to 20%). Vacuum also has a second job unrelated to space at all: it's what prevents transaction ID wraparound, a much rarer but far more serious failure mode where an un-vacuumed table's transaction ID counter runs out of room entirely.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Why this matters in production:&lt;/strong&gt; vacuum generates its own WAL records, since it's modifying pages, so a large autovacuum run on a big table shows up as real write volume, not a free background chore. It also competes for the same I/O bandwidth a checkpoint is using, which is why an aggressive autovacuum kicking off right as a checkpoint is flushing dirty pages is a common, specific cause of a latency spike that doesn't correlate with any query change. &lt;a href="https://dev.to/blog/database-efficiency-101-understanding-bloat-vacuum-and-the-power-of-pg_repack"&gt;Database Efficiency 101&lt;/a&gt; goes deeper on the bloat side of this; the piece worth adding here is that vacuum isn't an isolated background job, it's sharing the same WAL and I/O path as everything above.&lt;/p&gt;

&lt;h2&gt;
  
  
  ANALYZE: the step that keeps the planner honest
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;ANALYZE&lt;/code&gt; is a separate operation from vacuum, even though &lt;code&gt;autovacuum&lt;/code&gt; triggers both and people often say "autovacuum" to mean either. &lt;code&gt;ANALYZE&lt;/code&gt; samples a table's rows and updates the planner's statistics (row counts, most common values, distribution histograms) stored in &lt;code&gt;pg_statistic&lt;/code&gt;. The query planner uses those statistics, not a live count, to estimate how many rows a &lt;code&gt;WHERE&lt;/code&gt; clause will match and decide whether a sequential scan, index scan, or bitmap heap scan will be cheapest.&lt;/p&gt;

&lt;p&gt;Stale statistics don't cause a query to return wrong results. They cause the planner to misjudge how many rows it's dealing with, and a bad row-count estimate is one of the most common root causes behind "this query used to be fast and now it isn't," with no schema change and no data corruption involved, just a planner making a good decision on outdated information. &lt;code&gt;autovacuum_analyze_threshold&lt;/code&gt; and &lt;code&gt;autovacuum_analyze_scale_factor&lt;/code&gt; (10% by default) control when this runs automatically, similarly to vacuum's thresholds, and an insert-heavy, delete-free table (an append-only events table, for instance) still needs &lt;code&gt;ANALYZE&lt;/code&gt; on a schedule even though it may rarely need vacuuming for dead tuples.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Why this matters in production:&lt;/strong&gt; after a large bulk load, either via a migration or an initial data import, running &lt;code&gt;ANALYZE&lt;/code&gt; explicitly before serving real traffic is worth the few seconds it costs. Waiting for autovacuum to notice on its own schedule means the planner may make its first few hours of decisions on statistics from before the table had any real data in it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Putting the five back together
&lt;/h2&gt;

&lt;p&gt;None of these are separate systems bolted onto Postgres. They're one pipeline: WAL makes every write durable and replicable, checkpoints periodically settle that WAL's guarantees into the actual data files and bound how long crash recovery takes, wait events are the live diagnostic surface showing you when a backend is stuck somewhere in that path, vacuum reclaims the space MVCC leaves behind and prevents wraparound, and &lt;code&gt;ANALYZE&lt;/code&gt; keeps the planner's model of your data close enough to reality to keep picking good plans. A slow query is sometimes a bad query. Just as often, in a system that's been running fine for months, it's one of these five processes falling behind, and &lt;code&gt;pg_stat_activity&lt;/code&gt; is the first place that tells you which one.&lt;/p&gt;

</description>
      <category>postgres</category>
      <category>database</category>
      <category>wal</category>
      <category>vacuum</category>
    </item>
    <item>
      <title>Your ORM Isn't Lying to You. It's Just Not Telling You Everything.</title>
      <dc:creator>Jatin Jain Saraf</dc:creator>
      <pubDate>Thu, 03 Sep 2026 13:32:46 +0000</pubDate>
      <link>https://dev.to/jatinjainsaraf/your-orm-isnt-lying-to-you-its-just-not-telling-you-everything-35k4</link>
      <guid>https://dev.to/jatinjainsaraf/your-orm-isnt-lying-to-you-its-just-not-telling-you-everything-35k4</guid>
      <description>&lt;p&gt;An ORM is a translator sitting in a business meeting. It's fluent, it's fast, and after a while you stop paying attention to the original language entirely. You trust it. Most days that trust is fine. Then one sentence gets mistranslated, the deal falls apart, and you realize you never actually learned the language yourself. You just learned to rely on someone who did.&lt;/p&gt;

&lt;p&gt;That's what Prisma, Drizzle, TypeORM, and Sequelize do to your relationship with Postgres. They don't lie. They just don't tell you everything, and the gaps are exactly where production incidents live.&lt;/p&gt;

&lt;p&gt;This post is not "ORMs are bad, use raw SQL." Most apps should use one. It's a map of the specific places where the ORM's model of your database and Postgres's actual behavior diverge, so you know where to look when something is slow, wrong, or gone.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why teams reach for an ORM in the first place
&lt;/h2&gt;

&lt;p&gt;Before the shortcomings, the honest case for using one:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Types generated from your schema.&lt;/strong&gt; Rename a column, your build breaks at compile time instead of at 2am in production.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Migrations as code.&lt;/strong&gt; Schema changes are version-controlled, reviewable in a pull request, and reproducible across environments.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Boilerplate CRUD disappears.&lt;/strong&gt; Most application code touching the database is &lt;code&gt;SELECT&lt;/code&gt;, &lt;code&gt;INSERT&lt;/code&gt;, &lt;code&gt;UPDATE&lt;/code&gt;, &lt;code&gt;DELETE&lt;/code&gt; on one or two tables. An ORM turns that into a few lines instead of a hand-written query and a row-mapping function, every time.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;None of that is wrong. The problem is what it costs you when you stop thinking about the SQL underneath.&lt;/p&gt;

&lt;h2&gt;
  
  
  The N+1 query, hiding in plain sight
&lt;/h2&gt;

&lt;p&gt;This is the failure mode that catches the most teams, because the ORM makes it look identical to the correct version.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;users&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;prisma&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;user&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;findMany&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
&lt;span class="k"&gt;for &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;user&lt;/span&gt; &lt;span class="k"&gt;of&lt;/span&gt; &lt;span class="nx"&gt;users&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;posts&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;prisma&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;post&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;findMany&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;where&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;authorId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nx"&gt;user&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="p"&gt;});&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That's 1 query to fetch users, plus N more queries, one per user, to fetch their posts. With 10 users you might not notice. With 10,000 users, those extra round trips can turn an otherwise simple endpoint into a timeout.&lt;/p&gt;

&lt;p&gt;The dangerous part isn't that this pattern exists. Every engineer knows N+1 queries are bad. The dangerous part is that the ORM's relational syntax looks the same whether it's doing this or a single join:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;users&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;prisma&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;user&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;findMany&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
  &lt;span class="na"&gt;include&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;posts&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt; &lt;span class="p"&gt;},&lt;/span&gt;
&lt;span class="p"&gt;});&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Relation loading isn't necessarily equivalent to the join you would write by hand. Depending on the ORM and its configuration, that one line can compile down to a single query with a join, or to a batch of separate queries stitched together in application code. Prisma, for example, defaults to the second approach for &lt;code&gt;include&lt;/code&gt; on one-to-many relations: it batches the related rows with a SQL &lt;code&gt;IN&lt;/code&gt; clause rather than emitting a native &lt;code&gt;JOIN&lt;/code&gt;, unless you're on a relation mode or raw query that says otherwise. You can't reliably tell which one you got just by reading the application code. Turn on query logging and look.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Why this matters in production:&lt;/strong&gt; N+1 doesn't necessarily mean N new connections. Queries can reuse the pool. The problem is the number of database round trips. Under concurrent traffic, many requests running their own N+1 loops can keep pooled connections busy for longer, increasing pool contention and request latency. This is where an ORM's convenience directly creates the incident.&lt;/p&gt;

&lt;h2&gt;
  
  
  The SQL your ORM actually generates is not the SQL you'd write
&lt;/h2&gt;

&lt;p&gt;Turn on query logging for a week on a real Prisma or TypeORM project and read what comes out. It is rarely what you'd write by hand.&lt;/p&gt;

&lt;p&gt;With nested &lt;code&gt;include&lt;/code&gt;s, some ORMs, Prisma among them, can issue several separate queries and stitch the results together in application code instead of one query with joins. That's not a bug, it's a design trade-off, and it varies by ORM, version, and configuration. But it means &lt;code&gt;EXPLAIN ANALYZE&lt;/code&gt; on the query you think you're running and the query Postgres actually saw can tell two different stories, and only one of them is visible in your codebase.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Why this matters in production:&lt;/strong&gt; if you've read through the &lt;a href="https://dev.to/courses/postgresql-in-depth"&gt;query planning module&lt;/a&gt; of the PostgreSQL In-Depth course, you know Postgres picks between a sequential scan, an index scan, and a bitmap heap scan based on cost estimates for the &lt;em&gt;specific query it receives&lt;/em&gt;. An ORM that reshapes your intent into three smaller queries instead of one joined query gives the planner three separate decisions to make, rather than one plan for the operation as a whole. You lose the planner's ability to optimize across the whole operation.&lt;/p&gt;

&lt;p&gt;The fix isn't abandoning the ORM. It's treating query logging as mandatory, not optional, and periodically running &lt;code&gt;EXPLAIN ANALYZE&lt;/code&gt; on queries that matter to your application's hot paths or have noticeable latency.&lt;/p&gt;

&lt;h2&gt;
  
  
  Migrations: the auto-diff can silently drop data
&lt;/h2&gt;

&lt;p&gt;Most ORMs generate migrations by diffing your schema file against the last known state and producing SQL. This works well for additive changes. It gets dangerous on renames.&lt;/p&gt;

&lt;p&gt;Rename a column in your schema file, and a schema diff doesn't always have enough information to know that you intended a rename rather than a drop-and-add. Some ORMs can detect renames, while others may generate a drop-and-add migration depending on the schema change and migration workflow. The generated migration can be:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;users&lt;/span&gt; &lt;span class="k"&gt;DROP&lt;/span&gt; &lt;span class="k"&gt;COLUMN&lt;/span&gt; &lt;span class="n"&gt;full_name&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;users&lt;/span&gt; &lt;span class="k"&gt;ADD&lt;/span&gt; &lt;span class="k"&gt;COLUMN&lt;/span&gt; &lt;span class="n"&gt;display_name&lt;/span&gt; &lt;span class="nb"&gt;text&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;instead of the safe version:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;users&lt;/span&gt; &lt;span class="k"&gt;RENAME&lt;/span&gt; &lt;span class="k"&gt;COLUMN&lt;/span&gt; &lt;span class="n"&gt;full_name&lt;/span&gt; &lt;span class="k"&gt;TO&lt;/span&gt; &lt;span class="n"&gt;display_name&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The first version runs without error. It also silently discards every value in that column on deploy.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Why this matters in production:&lt;/strong&gt; a migration that runs clean and drops a column's worth of data doesn't announce itself. Nobody gets an error. You find out when a support ticket comes in asking why a field is suddenly empty. This is one of the highest-leverage things to check before merging any ORM-generated migration: read the actual SQL it produces, not just the schema diff you wrote.&lt;/p&gt;

&lt;h2&gt;
  
  
  Connection pooling: every ORM has different defaults, and it matters more than ever in serverless
&lt;/h2&gt;

&lt;p&gt;If you've read &lt;a href="https://dev.to/blog/connection-pooling-in-the-serverless-era-five-failure-modes"&gt;Connection Pooling in the Serverless Era&lt;/a&gt;, you already know that each connection Postgres accepts consumes server resources, whether or not it's doing anything, and enough concurrent connections becomes a significant resource cost. ORMs each ship their own default pool size, their own idle-timeout behavior, and their own assumptions about how long your process lives.&lt;/p&gt;

&lt;p&gt;Those defaults were mostly written assuming a long-lived server process. In a serverless or edge function, where every invocation can be a fresh process, an ORM's default pool of, say, 10 connections means every cold start tries to open 10 connections to Postgres. Multiply that by concurrent invocations during a traffic spike and you can exhaust your database's max connection limit in seconds, well before you exhaust CPU or memory on the compute side.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Why this matters in production:&lt;/strong&gt; this is a common cause of "it works locally, it falls over under load" for teams deploying ORM-based apps to serverless platforms. The fix usually involves an external pooler (PgBouncer, RDS Proxy, or a managed equivalent) sitting between your functions and Postgres, plus explicitly tuning the ORM's own pool size down, not trusting the default. One trap worth knowing before you reach for a pooler: PgBouncer's transaction-mode pooling, the mode most serverless setups need, doesn't preserve a session across queries, and ORMs that rely on prepared statements (Prisma and TypeORM both do, by default) can fail against it in ways that only show up under load. Check your pooler's mode against your ORM's prepared-statement behavior before you assume adding one fixes everything.&lt;/p&gt;

&lt;h2&gt;
  
  
  Write-heavy systems: where the ORM's convenience becomes the bottleneck
&lt;/h2&gt;

&lt;p&gt;Everything above assumes a fairly typical application: reads dominate, writes happen when a user submits a form or an admin edits a record. That's most CRUD apps, and it's exactly where an ORM's ergonomics pay for themselves. It is not a blockchain indexer, an event pipeline, or anything else ingesting a continuous, high-volume stream of writes. Those systems are where an ORM's row-at-a-time model of the world stops being a convenience and starts being the bottleneck.&lt;/p&gt;

&lt;p&gt;A blockchain indexer is a clean example because the write pattern is unforgiving: every new block can carry hundreds or thousands of events (transfers, contract calls, state changes), and all of them need to land in Postgres before the indexer can call itself caught up. Three places an ORM adds real cost in that path:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Row-by-row &lt;code&gt;INSERT&lt;/code&gt; instead of &lt;code&gt;COPY&lt;/code&gt;.&lt;/strong&gt; Postgres's &lt;code&gt;COPY&lt;/code&gt; protocol is built specifically for loading large volumes of rows in one streamed operation, and it is meaningfully faster than even a well-batched multi-row &lt;code&gt;INSERT&lt;/code&gt;. Most ORMs don't expose &lt;code&gt;COPY&lt;/code&gt; at all. Their bulk-insert APIs (&lt;code&gt;createMany&lt;/code&gt;, &lt;code&gt;bulkCreate&lt;/code&gt;, and similar) are a real improvement over inserting one row per statement, but under the hood they're still building &lt;code&gt;INSERT&lt;/code&gt; statements with many value tuples, not streaming through &lt;code&gt;COPY&lt;/code&gt;. For an indexer processing thousands of events per block, that gap compounds every single block.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Object hydration on the way in and out.&lt;/strong&gt; An ORM that maps every row to and from a model instance is doing real work per row: validation, type coercion, applying defaults. That cost is invisible at CRUD volumes and very visible at ingestion volumes, where it adds CPU and allocation overhead to every event in the stream, not just the ones a human is waiting on.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;More, smaller transactions than the workload needs.&lt;/strong&gt; An ORM's ergonomic defaults tend toward one transaction per logical operation. At indexer-scale throughput, that can mean far more transaction and WAL overhead than a design that deliberately batches writes into fewer, larger transactions per block.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Why this matters in production:&lt;/strong&gt; all three of these show up as the same symptom eventually, insert throughput that can't keep up with the source it's indexing, and the indexer falls further behind with every block. If you've read &lt;a href="https://dev.to/blog/taming-postgresql-replication-lag-in-real-time-blockchain-indexers"&gt;Taming PostgreSQL Replication Lag in Real-Time Blockchain Indexers&lt;/a&gt;, this is the same category of problem from a different angle: it's not always replication configuration that causes an indexer to lag, sometimes it's the write path itself, one ORM abstraction away from the bulk-loading tools Postgres actually gives you for this job.&lt;/p&gt;

&lt;p&gt;None of this means an ORM is disqualified from a write-heavy system. It means the write path deserves the same scrutiny as any other hot path in this article: measure it, and if it's the bottleneck, that's exactly the place to drop to &lt;code&gt;COPY&lt;/code&gt;, hand-batched multi-row &lt;code&gt;INSERT&lt;/code&gt;s, or a dedicated bulk-loading library, and keep the ORM everywhere else it's not in the way.&lt;/p&gt;

&lt;h2&gt;
  
  
  When the abstraction stops being useful
&lt;/h2&gt;

&lt;p&gt;Every ORM's query builder is designed around the common case: filter, sort, join, paginate. The moment your requirement moves past that, you're negotiating with the abstraction instead of using it. In practice, that means:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Window functions&lt;/strong&gt; (running totals, ranking within groups) — most ORM query builders don't model these at all.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Recursive CTEs&lt;/strong&gt; (org charts, threaded comments, category trees) — almost always raw SQL, even in projects that otherwise avoid it entirely.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Bulk operations&lt;/strong&gt; — an ORM's &lt;code&gt;.update()&lt;/code&gt; in a loop is N statements; a single &lt;code&gt;UPDATE ... WHERE id = ANY($1)&lt;/code&gt; is one. The difference compounds fast on large batches.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;ON CONFLICT&lt;/code&gt; (upsert) and &lt;code&gt;RETURNING&lt;/code&gt;&lt;/strong&gt; — supported by some ORMs, but often with a narrower API than the SQL clause itself allows, especially bulk upserts (&lt;code&gt;ON CONFLICT DO UPDATE&lt;/code&gt; across many rows at once) and the newer &lt;code&gt;OLD&lt;/code&gt;/&lt;code&gt;NEW&lt;/code&gt; aliases in &lt;code&gt;RETURNING&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;SKIP LOCKED&lt;/code&gt; for queue-style workloads&lt;/strong&gt; — the standard pattern for letting multiple workers pull from the same table without blocking on each other's locked rows. Most ORM query builders have no vocabulary for it at all, so it's raw SQL from the start.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Partial and expression indexes&lt;/strong&gt; — PostgreSQL can use indexes defined with a &lt;code&gt;WHERE&lt;/code&gt; clause or an expression, but the generated SQL still needs to satisfy the conditions that make those indexes usable. An ORM won't necessarily make that obvious from the application code.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;JSONB queries with GIN indexes&lt;/strong&gt; — the operators that make these fast (&lt;code&gt;@&amp;gt;&lt;/code&gt;, &lt;code&gt;?&lt;/code&gt;, &lt;code&gt;#&amp;gt;&amp;gt;&lt;/code&gt;) are Postgres-specific and rarely have first-class ORM support.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Row-level and advisory locking&lt;/strong&gt; — &lt;code&gt;SELECT ... FOR UPDATE&lt;/code&gt;, &lt;code&gt;pg_advisory_lock&lt;/code&gt;, and friends are concurrency primitives most ORMs expose thinly or not at all.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Reading &lt;code&gt;EXPLAIN ANALYZE&lt;/code&gt; output&lt;/strong&gt; — no ORM will do this for you. It's the one skill that stays entirely yours no matter which tool you pick.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;None of this means avoid the ORM for these cases. It means know, ahead of time, that you'll drop to raw SQL here, and treat that as normal rather than a sign something went wrong.&lt;/p&gt;

&lt;h2&gt;
  
  
  Is your ORM a black box? It depends which one
&lt;/h2&gt;

&lt;p&gt;So what should you actually use?&lt;/p&gt;

&lt;p&gt;Not all ORMs hide the same amount, and the trade-off usually runs opposite to how much boilerplate they remove:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Approach&lt;/th&gt;
&lt;th&gt;Abstraction&lt;/th&gt;
&lt;th&gt;SQL visibility&lt;/th&gt;
&lt;th&gt;Best fit&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Prisma&lt;/td&gt;
&lt;td&gt;High&lt;/td&gt;
&lt;td&gt;Medium&lt;/td&gt;
&lt;td&gt;CRUD-heavy product development&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;TypeORM&lt;/td&gt;
&lt;td&gt;High&lt;/td&gt;
&lt;td&gt;Medium&lt;/td&gt;
&lt;td&gt;Traditional TypeScript applications, decorator-based models&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Sequelize&lt;/td&gt;
&lt;td&gt;High&lt;/td&gt;
&lt;td&gt;Medium&lt;/td&gt;
&lt;td&gt;Mature Node.js codebases already built around it&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Drizzle&lt;/td&gt;
&lt;td&gt;Medium&lt;/td&gt;
&lt;td&gt;High&lt;/td&gt;
&lt;td&gt;Type-safe apps that still want SQL-shaped queries&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;pg&lt;/code&gt; / &lt;code&gt;node-postgres&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Low&lt;/td&gt;
&lt;td&gt;Full&lt;/td&gt;
&lt;td&gt;SQL-heavy code and hot paths&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;There's no winner in that table on purpose. A team shipping CRUD-heavy internal tools benefits enormously from Prisma's or Sequelize's ergonomics and rarely hits the edges above. A team running a write-heavy, high-traffic public API needs to know exactly which SQL is running on every hot path, and might be better served by Drizzle or hand-written queries where it matters most. Plenty of production codebases mix approaches deliberately: an ORM for the CRUD majority, raw SQL for the handful of queries that are actually hot.&lt;/p&gt;

&lt;h2&gt;
  
  
  The actual takeaway
&lt;/h2&gt;

&lt;p&gt;The ORM isn't the problem. Forgetting that an ORM is generating database operations is the problem. It doesn't remove the need to understand what Postgres is doing, it just moves the moment you're forced to learn it, from day one of the project to the incident where your database is drowning in unnecessary queries because the ORM issued forty small &lt;code&gt;UPDATE&lt;/code&gt; statements instead of one bulk operation.&lt;/p&gt;

&lt;p&gt;Use the ORM for what it's good at: the 80% of your code that's routine CRUD, type-safe and fast to write. But keep query logging on, read the SQL your ORM generates for anything on a hot path, and read the actual migration SQL before it hits production, not just the schema diff. The ORM is a very good translator. It is still worth knowing enough of the language yourself to catch it when it gets a sentence wrong.&lt;/p&gt;

</description>
      <category>orm</category>
      <category>postgres</category>
      <category>database</category>
      <category>backenddevelopment</category>
    </item>
    <item>
      <title>A Reliability-Hardening Sprint: Isolation, Guardrails, and a Query Plan That Went From 45 Million to 206</title>
      <dc:creator>Jatin Jain Saraf</dc:creator>
      <pubDate>Tue, 01 Sep 2026 06:58:59 +0000</pubDate>
      <link>https://dev.to/jatinjainsaraf/a-reliability-hardening-sprint-isolation-guardrails-and-a-query-plan-that-went-from-45-million-4ce6</link>
      <guid>https://dev.to/jatinjainsaraf/a-reliability-hardening-sprint-isolation-guardrails-and-a-query-plan-that-went-from-45-million-4ce6</guid>
      <description>&lt;h1&gt;
  
  
  A Reliability-Hardening Sprint: Isolation, Guardrails, and a Query Plan That Went From 45 Million to 206
&lt;/h1&gt;

&lt;p&gt;The easiest reliability work to justify is the kind that follows an outage. It's much harder to carve out time for the kind that doesn't, fixes made because a risk is visible and worth closing, not because it already caused damage. Most of what's in this piece shipped in a single week without a single one of these changes being a response to a live incident. One of them was a response to an external report, handled the way you'd want it handled. Together they're a good picture of what proactive reliability work actually looks like, as opposed to the reactive kind that gets written up more often because it comes with a dramatic story attached.&lt;/p&gt;

&lt;h2&gt;
  
  
  Deciding which environments actually need their own failure domain
&lt;/h2&gt;

&lt;p&gt;Multiple blockchain environments were originally sharing a single default Redis instance for caching. That's a reasonable starting point when the system is small, and a real liability once one environment's traffic pattern starts affecting another's: a cache invalidation problem, a memory pressure spike, or a slow operation in one environment can degrade or contaminate the cache for every other environment sharing that same instance, even though those environments have nothing to do with each other functionally. The fix wasn't to give every environment its own instance unconditionally, it was to identify which ones actually warranted the isolation and pull just those out: mainnet and one specific microchain, the one carrying enough independent traffic and importance to justify it, each got their own dedicated Redis instance, enforced strictly enough in code that the service refuses to start if that instance's address isn't configured. Everything else still shares a common default instance. Nothing about this fix was triggered by an incident. It closed a risk that was visible in the architecture before it had the chance to become one, and it did so by isolating exactly the environments that needed it rather than paying the operational cost of isolating all of them.&lt;/p&gt;

&lt;h2&gt;
  
  
  A timeout is a blast-radius decision, not a performance tweak
&lt;/h2&gt;

&lt;p&gt;Every GraphQL resolver in the backend service, every query and every mutation, got wrapped with a wall-clock timeout, applied schema-wide rather than added resolver by resolver. It's tempting to think of a timeout as a performance optimization, something you add to make slow things feel faster. That's not really what this fix does. Its actual job is bounding how much damage a single slow or hanging query can do to everything around it: without a timeout, one resolver stuck waiting on a slow database call or a hung upstream request can hold a connection, a worker thread, or a piece of shared capacity indefinitely, degrading the whole service for requests that have nothing to do with the slow one. A timeout doesn't make the underlying slow query fast. It makes sure that whatever caused it to be slow stays contained to that one request instead of spreading.&lt;/p&gt;

&lt;h2&gt;
  
  
  The number worth quoting on its own
&lt;/h2&gt;

&lt;p&gt;A separate fix, found during a broader review of query performance, is the single most quotable result in this entire piece. A function fetching network-wide transaction statistics was filtering its query by a timestamp column instead of the table's actual partition key. That distinction sounds academic until you see what it costs: filtering on the wrong column meant Postgres couldn't use partition pruning at all, forcing a full scan across every partition, including one that alone held over 400 gigabytes of data. The query planner's own estimated cost for that query, captured at the time, was just under 45 million. Correcting the filter to use the actual partition key was the first fix, and it dropped that same estimated cost to 206, from a query the planner expected to be genuinely expensive to one it treated as nearly free from a single change of which column a filter clause referenced. The system has since moved a step further than that filter fix. The current version of this function doesn't scan the partitioned transaction table at all, even with a correct filter: it reads instead from a separate, precomputed stats snapshot table, sidestepping the scan entirely. The comment left in that code still cites the original ~45 million cost as the reason the snapshot approach exists, a permanent, visible record of the problem the current design was built to avoid.&lt;/p&gt;

&lt;p&gt;A related fix in the same family addressed a query fetching transaction lists that was missing partition-key predicates specifically inside its join clauses, which defeated the database's runtime partition pruning and forced scans across well over a hundred partitions where only a small number should have been touched. Both fixes are variations on the same underlying lesson: a partitioned table only delivers its performance benefit if every query touching it, including the parts buried inside joins and CTEs, actually filters on the partition key. Miss it in one place, and the partitioning strategy silently stops helping for that specific query while still looking correct.&lt;/p&gt;

&lt;h2&gt;
  
  
  An external report handled the right way
&lt;/h2&gt;

&lt;p&gt;Not every fix here was found internally. An unauthenticated search endpoint was found, by an external report rather than an internal audit, to accept oversized input in a way that could trigger server errors. The response was layered rather than a single patch: a hard length cap on the search input rejects oversized requests outright before they reach anything expensive, the cache key for a search result is now a hash of the input rather than the input itself, so even a maliciously oversized string can't be used to construct an arbitrarily large cache key and exhaust cache memory that way, and a dedicated GraphQL security library was added on top, enforcing limits on query depth, aliases, and token count, with batched HTTP requests disabled outright so those same limits can't be sidestepped by sending an array of operations instead of one. The one control deliberately left off was a cost-limiting layer, everything else in that same security library was turned on. The detail worth noting isn't the specific fix, it's that an externally-reported issue got a defense-in-depth response instead of the narrowest possible patch that would have made the specific report stop reproducing.&lt;/p&gt;

&lt;h2&gt;
  
  
  The small bug that's a reminder about write-path consistency
&lt;/h2&gt;

&lt;p&gt;One more fix, smaller in scope but worth including for what it represents: an address being inserted into a detail table without the padding format that the corresponding foreign-key target table always expected, causing insert failures against that constraint. It's a small, almost mundane bug, but it's a recognizable category: two different code paths writing related data that quietly disagree about a formatting convention neither one documented explicitly. It's the same underlying failure shape as the type-mismatch issues described elsewhere in this series, just at a smaller scale and caught before it caused a bigger one.&lt;/p&gt;

&lt;h2&gt;
  
  
  What this demonstrates
&lt;/h2&gt;

&lt;p&gt;None of these seven changes, cache isolation, resolver timeouts, two separate partition-pruning fixes, a security hardening response, and a small write-path consistency fix, are individually dramatic. What makes them worth writing up together is that none of them were forced by an outage severe enough to demand attention. They were found by looking, in the middle of a normal week, for risks that hadn't yet become incidents, and closing them anyway. That's a much less visible kind of engineering work than fixing something already on fire, and it's usually the reason there's less to write up after the fact, not because less happened, but because nothing broke.&lt;/p&gt;

</description>
      <category>casestudy</category>
      <category>reliability</category>
      <category>graphql</category>
    </item>
    <item>
      <title>A Type Migration Across Three Services: VARCHAR to BIGINT at Scale</title>
      <dc:creator>Jatin Jain Saraf</dc:creator>
      <pubDate>Tue, 01 Sep 2026 06:57:59 +0000</pubDate>
      <link>https://dev.to/jatinjainsaraf/a-type-migration-across-three-services-varchar-to-bigint-at-scale-4518</link>
      <guid>https://dev.to/jatinjainsaraf/a-type-migration-across-three-services-varchar-to-bigint-at-scale-4518</guid>
      <description>&lt;h1&gt;
  
  
  A Type Migration Across Three Services: VARCHAR to BIGINT at Scale
&lt;/h1&gt;

&lt;p&gt;A block height stored as text instead of a number sounds like a small, easily-forgiven decision, and for a long time it is. The chain's own RPC layer returns numeric values like block height as strings in the first place, because some of those numbers exceed what can be safely represented as a standard number in JavaScript, so storing them as text on the way in feels like the path of least resistance. The cost of that decision doesn't show up until someone runs a range query and Postgres compares those "numbers" alphabetically instead of numerically, silently returning wrong results. Fixing that meant migrating the column's actual type from text to a proper integer at its source, the indexer that originally writes it, and then getting every downstream reader of that same column to agree with the new type. The migration itself turned out to be the least risky part of the whole change.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why the column type was the easy part
&lt;/h2&gt;

&lt;p&gt;Changing a column's underlying type in Postgres is a well-understood operation. The genuine risk in a change like this isn't the &lt;code&gt;ALTER TABLE&lt;/code&gt; itself, it's every piece of application code downstream that was written against the old type's behavior without ever stating that assumption explicitly. A function that concatenates a block height into a string for logging, a comparison that relies on lexicographic ordering without realizing it, a serialization layer that formats a string differently than it formats a number, none of these show up as compile errors. They show up as runtime bugs, in production, only when that specific code path executes against the newly-typed column for the first time.&lt;/p&gt;

&lt;p&gt;That's exactly what happened here. The migration itself landed cleanly across the backend service and the socket server, both of which read from the same underlying column. Two follow-up hotfixes were needed shortly after, both in functions that hadn't been touched by the migration directly but that broke because they'd been implicitly relying on the column's old string type when computing values like the latest block height and network-wide statistics. Neither hotfix was a large fix. Both were necessary, and both were the direct, predictable cost of a type change rippling through code that had never explicitly declared its assumption about that type in the first place.&lt;/p&gt;

&lt;h2&gt;
  
  
  What coordinating across services actually requires
&lt;/h2&gt;

&lt;p&gt;The reason this qualifies as a coordination problem, not just a database migration, is that three services, the indexer writing the column, and the backend and the socket server reading it, are all deployed independently, on their own release cycles, each with its own copy of code that depends on this column's type. A schema change made once, in one place, has to be correctly consumed by every service reading that data, and there's no compiler shared across service boundaries to catch a mismatch between what the database now returns and what a given service's code still expects. The two hotfixes here are exactly what "coordination worked, but wasn't perfect on the first attempt" looks like in practice: the underlying migration itself never had to be rolled back or redone, but the code on the consuming side needed a second pass once the new type actually reached it in production.&lt;/p&gt;

&lt;h2&gt;
  
  
  What this demonstrates
&lt;/h2&gt;

&lt;p&gt;The lesson here isn't "test your migrations more," though that's true of any migration. It's that a type change at the schema level is never really contained to the schema. It's a contract change with every piece of code, across every service, that reads that column, and the services most likely to break are the ones that were never explicitly written against a documented type contract in the first place, because they were written back when the column's actual type was assumed rather than verified. Catching both breakages quickly, shipping targeted hotfixes rather than a broader rollback, and confirming the underlying migration itself held throughout is the realistic version of what "coordinating a schema change across services" looks like when it goes well: not zero surprises, but small, contained ones, caught and fixed fast.&lt;/p&gt;

</description>
      <category>casestudy</category>
      <category>postgres</category>
      <category>schemamigration</category>
    </item>
    <item>
      <title>Built for a Multi-VM Future: The RPC v4 Readiness Work</title>
      <dc:creator>Jatin Jain Saraf</dc:creator>
      <pubDate>Tue, 01 Sep 2026 06:56:59 +0000</pubDate>
      <link>https://dev.to/jatinjainsaraf/built-for-a-multi-vm-future-the-rpc-v4-readiness-work-2bc6</link>
      <guid>https://dev.to/jatinjainsaraf/built-for-a-multi-vm-future-the-rpc-v4-readiness-work-2bc6</guid>
      <description>&lt;h1&gt;
  
  
  Built for a Multi-VM Future: The RPC v4 Readiness Work
&lt;/h1&gt;

&lt;p&gt;Some engineering work happens in response to an incident. This work happened in response to a feature that hadn't fully shipped yet. Supra's chain infrastructure introduced a new RPC version explicitly designed to support multiple virtual machines processing transactions within the same block, Move and EVM to start, with more planned. At the time this readiness work began, the indexing stack was still built entirely around the older RPC version and its Move-only assumptions. The task wasn't fixing something broken. It was mapping, precisely, everything that would break once the new capability went fully live, before it did.&lt;/p&gt;

&lt;h2&gt;
  
  
  A wrapper key that touches everything downstream
&lt;/h2&gt;

&lt;p&gt;The documented change between RPC versions looks small in isolation: several top-level fields on a transaction, the sender, the payload, the output, the authenticator, gain a wrapper key identifying which virtual machine produced that data. The interesting part is that the codebase's own type definitions have already started anticipating this shape ahead of the actual rollout: the response types for these fields already model a VM-keyed wrapper alongside the currently-used one, with a placeholder branch for a second VM sitting unused in the type layer today, well before that second VM's data ever arrives over the wire. That's a reasonable way to de-risk a breaking change, let the types absorb the new shape early, in a place where being wrong costs a compile error, rather than doing it all at once under time pressure once the new format is actually live.&lt;/p&gt;

&lt;p&gt;Type-level readiness isn't the same as working end-to-end, though, and that gap is the real scope of what's left. Every place in both the indexer and the backend that reads a transaction's sender, payload, output, or authenticator still needs to actually branch on which VM produced it and unwrap accordingly, across multiple files in both repositories, not just have a type that permits doing so. Event parsing, which currently reads from one fixed location in the response, needs equivalent VM-aware branching. None of these are individually difficult changes. All of them need to happen together, correctly, and be exercised against real second-VM data before the system can claim genuine multi-VM support rather than a type layer that's merely ready for it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why this is a cross-cutting change, not a parser update
&lt;/h2&gt;

&lt;p&gt;The reason this readiness work spans far more than the RPC-consuming code is what multi-VM support actually means at the data layer. A single block will be able to contain transactions from more than one virtual machine simultaneously, and those transaction types don't share a schema. An EVM transaction's meaningful fields, sender, recipient, calldata, value, look nothing like a Move transaction's sender, function, and payload. That difference has to be handled at every layer: the indexer needs a distinct parser per virtual machine, plus a column identifying which VM produced each row, since a single flat schema can't represent both shapes cleanly. The database question is still genuinely open rather than settled: either add a dedicated table per virtual machine, one for EVM alongside the existing Move-shaped ones, and eventually a Solana or Sui equivalent, or keep one shared transaction table and add a JSONB column to hold whatever VM-specific fields don't fit the common shape. Neither is free. Separate tables keep every VM's data cleanly typed and indexed but mean every cross-VM query, "show me the last hundred transactions regardless of which VM ran them," has to union across tables that don't share a schema. A shared table with a JSONB escape hatch keeps queries simple at the cost of reintroducing exactly the kind of untyped, hard-to-index column this same series describes paying down elsewhere. That tension hasn't been resolved yet, and naming it as unresolved is more honest than picking a side before the second VM actually arrives and reveals which cost matters more in practice. Even something as basic as the wallet triggers described elsewhere in this series need updating regardless of which path gets picked, because EVM addresses use a different length and format than Move addresses, and any logic keyed on address shape needs to handle both correctly. The backend needs VM-aware logic in every resolver that returns transaction details, and the frontend needs to render two structurally different transaction types side by side in the same block view, with address formatting that doesn't assume every wallet address looks the same.&lt;/p&gt;

&lt;h2&gt;
  
  
  Doing the scoping before the deadline, not during it
&lt;/h2&gt;

&lt;p&gt;At the time of this work, a multi-VM development network already existed and was publicly reachable, with a frontend deployed against it, but the actual multi-VM parsing, schema, and UI work hadn't been built yet, meaning that live deployment likely only reflected Move transactions in practice despite technically running against multi-VM-capable infrastructure. The rollout plan follows a pattern the team had already used successfully once before, when the previous RPC version transition happened: enable the new endpoint behind a feature flag on the testnet environment first, validate the full indexing pipeline end to end against real data in that shape, and only then flip the same flag on production once the chain itself makes the new version available there.&lt;/p&gt;

&lt;h2&gt;
  
  
  What this demonstrates
&lt;/h2&gt;

&lt;p&gt;The value in this piece of work isn't a shipped feature, because at the time of writing, the full multi-VM parsing and schema support hadn't been built yet either. The value is in having already mapped, field by field and file by file, exactly what breaks and where, ahead of the deadline that would otherwise force that mapping to happen under pressure. Readiness work like this rarely gets noticed when it goes well, because going well looks like nothing happened: the feature arrives, the flag flips, and the system handles it. The alternative, discovering the breaking changes live, in production, once transactions from a second virtual machine actually start arriving, is the outcome this scoping work exists specifically to prevent.&lt;/p&gt;

</description>
      <category>casestudy</category>
      <category>blockchain</category>
      <category>multivm</category>
    </item>
  </channel>
</rss>
