<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Anwar Iqbal Dawar</title>
    <description>The latest articles on DEV Community by Anwar Iqbal Dawar (@anwariqbaldawar).</description>
    <link>https://dev.to/anwariqbaldawar</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4193184%2F431c65ac-e867-41ce-8f1e-3c402b65fe7e.jpeg</url>
      <title>DEV Community: Anwar Iqbal Dawar</title>
      <link>https://dev.to/anwariqbaldawar</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/anwariqbaldawar"/>
    <language>en</language>
    <item>
      <title>Why a 2ms ALTER TABLE Took Down Our 512GB Postgres Table (And How We Fixed It)</title>
      <dc:creator>Anwar Iqbal Dawar</dc:creator>
      <pubDate>Sat, 07 Nov 2026 16:05:05 +0000</pubDate>
      <link>https://dev.to/anwariqbaldawar/why-a-2ms-alter-table-took-down-our-512gb-postgres-table-and-how-we-fixed-it-4b4g</link>
      <guid>https://dev.to/anwariqbaldawar/why-a-2ms-alter-table-took-down-our-512gb-postgres-table-and-how-we-fixed-it-4b4g</guid>
      <description>&lt;h1&gt;
  
  
  Zero-Downtime PostgreSQL Schema Migrations on 500GB+ Tables
&lt;/h1&gt;

&lt;p&gt;&lt;em&gt;Published on &lt;a href="https://xsypher.com/article/zero-downtime-postgresql-schema-migrations-on-500gb-tables" rel="noopener noreferrer"&gt;xSypher&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;




&lt;p&gt;An &lt;code&gt;ALTER TABLE orders ADD COLUMN status_v2 text;&lt;/code&gt; on a 512GB table executes in roughly two milliseconds. It updates one catalog row. It touches no heap pages. And it can still take an entire payments platform offline for four minutes, because the statement spends 240 seconds waiting for a lock that it never gets to use politely.&lt;/p&gt;

&lt;p&gt;The DDL is not the outage. The lock queue in front of it is. Everything below is about keeping migrations out of that queue: fail-fast lock timeouts with retries, &lt;code&gt;NOT VALID&lt;/code&gt; constraints, keyset-paginated backfills throttled against replica lag, &lt;code&gt;CONCURRENTLY&lt;/code&gt; index builds, and a direct connection that bypasses PgBouncer. Version behaviour described here holds for PostgreSQL 13 through 18; a few shortcuts are version-gated and flagged where they appear.&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;Two milliseconds of DDL, four minutes of downtime&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;PostgreSQL has eight table-level lock modes, and&lt;a href="https://www.postgresql.org/docs/current/explicit-locking.html" rel="noopener noreferrer"&gt;the conflict matrix in the explicit locking docs&lt;/a&gt; is the entire story. &lt;code&gt;SELECT&lt;/code&gt; takes &lt;code&gt;ACCESS SHARE&lt;/code&gt;. &lt;code&gt;INSERT&lt;/code&gt;, &lt;code&gt;UPDATE&lt;/code&gt; and &lt;code&gt;DELETE&lt;/code&gt; take &lt;code&gt;ROW EXCLUSIVE&lt;/code&gt;. Most forms of &lt;code&gt;ALTER TABLE&lt;/code&gt; take &lt;code&gt;ACCESS EXCLUSIVE&lt;/code&gt;, which conflicts with every other mode, including the one plain reads use.&lt;/p&gt;

&lt;p&gt;The failure mode is queue fairness. A lock request that cannot be granted immediately goes to the back of the wait queue for that relation, and new requests that conflict with anything already queued line up behind it rather than jumping ahead. So the sequence looks like this:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;An analytics &lt;code&gt;SELECT&lt;/code&gt; has been scanning &lt;code&gt;orders&lt;/code&gt; for eight seconds, holding &lt;code&gt;ACCESS SHARE&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Your migration asks for &lt;code&gt;ACCESS EXCLUSIVE&lt;/code&gt;. It conflicts, so it waits.&lt;/li&gt;
&lt;li&gt;The next &lt;code&gt;SELECT&lt;/code&gt; from your API conflicts with the &lt;em&gt;queued&lt;/em&gt; &lt;code&gt;ACCESS EXCLUSIVE&lt;/code&gt; request, so it waits too. So does the next one. And the next.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;At that point the table is effectively offline for reads and writes even though nothing is being modified. On our test bench (PostgreSQL 17 on 16 vCPU, 64GB RAM, gp3 storage at 12,000 provisioned IOPS, a 512GB &lt;code&gt;orders&lt;/code&gt; table with roughly 1.1 billion rows), we reproduced this with a synthetic load of 2,400 queries per second across 30 application pods holding 20 pooled connections each. The first queued query appeared at t+0ms. The pool was fully saturated at &lt;strong&gt;t+1.4s&lt;/strong&gt;. Health checks that shared the same pool started failing at t+9s, and the orchestrator began restarting pods, which made recovery slower than the original lock.&lt;/p&gt;

&lt;p&gt;The arithmetic is unforgiving: a pool of 600 connections and 2,400 qps fills in well under a second. Connection pool exhaustion, not lock duration, is what converts a stalled DDL into a full outage.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;strong&gt;Statement&lt;/strong&gt;&lt;/th&gt;
&lt;th&gt;&lt;strong&gt;Lock mode taken&lt;/strong&gt;&lt;/th&gt;
&lt;th&gt;&lt;strong&gt;Blocks reads?&lt;/strong&gt;&lt;/th&gt;
&lt;th&gt;&lt;strong&gt;Blocks writes?&lt;/strong&gt;&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;SELECT&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;ACCESS SHARE&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;INSERT&lt;/code&gt; / &lt;code&gt;UPDATE&lt;/code&gt; / &lt;code&gt;DELETE&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;&lt;code&gt;ROW EXCLUSIVE&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;CREATE INDEX CONCURRENTLY&lt;/code&gt;, &lt;code&gt;VALIDATE CONSTRAINT&lt;/code&gt;, &lt;code&gt;VACUUM&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;&lt;code&gt;SHARE UPDATE EXCLUSIVE&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;CREATE INDEX&lt;/code&gt; (plain)&lt;/td&gt;
&lt;td&gt;&lt;code&gt;SHARE&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;CREATE TRIGGER&lt;/code&gt;, &lt;code&gt;ADD FOREIGN KEY ... NOT VALID&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;&lt;code&gt;SHARE ROW EXCLUSIVE&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Most &lt;code&gt;ALTER TABLE&lt;/code&gt;, &lt;code&gt;DROP TABLE&lt;/code&gt;, &lt;code&gt;TRUNCATE&lt;/code&gt;, &lt;code&gt;REINDEX&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;&lt;code&gt;ACCESS EXCLUSIVE&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The practical read of that table: anything in the bottom row needs to be acquired instantly or not at all, and anything in the &lt;code&gt;SHARE UPDATE EXCLUSIVE&lt;/code&gt; row can run for an hour without a reader noticing.&lt;/p&gt;

&lt;h3&gt;
  
  
  &lt;strong&gt;Seeing the queue while it is happening&lt;/strong&gt;
&lt;/h3&gt;

&lt;p&gt;Keep this query in a snippet file, because you will want it under pressure rather than after.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;pid&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
       &lt;span class="k"&gt;state&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
       &lt;span class="n"&gt;wait_event_type&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
       &lt;span class="n"&gt;wait_event&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
       &lt;span class="n"&gt;now&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;query_start&lt;/span&gt;   &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;runtime&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
       &lt;span class="n"&gt;pg_blocking_pids&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pid&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;blocked_by&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
       &lt;span class="k"&gt;left&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;70&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;       &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;query&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt;   &lt;span class="n"&gt;pg_stat_activity&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt;  &lt;span class="n"&gt;backend_type&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'client backend'&lt;/span&gt;
  &lt;span class="k"&gt;AND&lt;/span&gt;  &lt;span class="n"&gt;datname&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;current_database&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="k"&gt;ORDER&lt;/span&gt;  &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;runtime&lt;/span&gt; &lt;span class="k"&gt;DESC&lt;/span&gt; &lt;span class="n"&gt;NULLS&lt;/span&gt; &lt;span class="k"&gt;LAST&lt;/span&gt;
&lt;span class="k"&gt;LIMIT&lt;/span&gt;  &lt;span class="mi"&gt;25&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The root blocker is the PID that appears inside other sessions' &lt;code&gt;blocked_by&lt;/code&gt; arrays but has an empty array of its own. In our bench that was consistently a reporting query or an &lt;code&gt;idle in transaction&lt;/code&gt; session left open by an ORM, not the migration itself. Cancelling the DDL with &lt;code&gt;pg_cancel_backend()&lt;/code&gt; drains the queue in milliseconds; cancelling the original long reader does not, because the DDL simply takes its place at the head.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;pg_terminate_backend()&lt;/code&gt; on the wrong PID kills an application connection mid-transaction and rolls back user-visible work. Try &lt;code&gt;pg_cancel_backend()&lt;/code&gt; first, and confirm the target PID with the query above rather than by position in a sorted list.&lt;/p&gt;

&lt;p&gt;Prevent leaked ORM transactions from blocking future deploys by enforcing &lt;code&gt;idle_in_transaction_session_timeout&lt;/code&gt; (for example, &lt;code&gt;60s&lt;/code&gt;) at the database or role level (&lt;code&gt;ALTER ROLE app_user SET idle_in_transaction_session_timeout='60s';&lt;/code&gt;). On PostgreSQL 17 and later, pair it with &lt;code&gt;transaction_timeout&lt;/code&gt; to cap total transaction duration across both active and idle states.&lt;/p&gt;

&lt;p&gt;What a lock pile-up looks like in practice: one long-running reader (PID 18402), one waiting ALTER TABLE (PID 29104), and ordinary application queries stacked behind the migration in the relation lock queue.&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;Set lock_timeout to 50ms, then retry like you mean it&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;The fix for the queue is refusing to stand in it. &lt;code&gt;lock_timeout&lt;/code&gt; aborts a statement that cannot acquire its lock within the configured interval, raising SQLSTATE &lt;code&gt;55P03&lt;/code&gt; (&lt;code&gt;lock_not_available&lt;/code&gt;). A migration that fails fast and retries costs you a few wasted attempts; a migration that waits costs you the service.&lt;/p&gt;

&lt;p&gt;Fifty milliseconds is the number we use on the bench. It is long enough to succeed on an idle-ish table, short enough that even a saturating 2,400 qps workload only queues a handful of statements per attempt. Anything above roughly 200ms starts producing visible p99 spikes in the API.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;#!/usr/bin/env bash&lt;/span&gt;
&lt;span class="nb"&gt;set&lt;/span&gt; &lt;span class="nt"&gt;-euo&lt;/span&gt; pipefail

&lt;span class="c"&gt;# Port 5432: the primary directly, never PgBouncer on 6432.&lt;/span&gt;
&lt;span class="nv"&gt;PGURI&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"postgres://migrator@db-primary.internal:5432/app"&lt;/span&gt;
&lt;span class="nv"&gt;MAX_ATTEMPTS&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;40

&lt;span class="k"&gt;for &lt;/span&gt;attempt &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;seq &lt;/span&gt;1 &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$MAX_ATTEMPTS&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do
  &lt;/span&gt;&lt;span class="nv"&gt;output&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;psql &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$PGURI&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nt"&gt;-v&lt;/span&gt; &lt;span class="nv"&gt;ON_ERROR_STOP&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;1 &lt;span class="nt"&gt;--quiet&lt;/span&gt; 2&amp;gt;&amp;amp;1 &lt;span class="o"&gt;&amp;lt;&amp;lt;&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="no"&gt;SQL&lt;/span&gt;&lt;span class="sh"&gt;'
BEGIN;
SET LOCAL lock_timeout = '50ms';
SET LOCAL statement_timeout = '5s';
ALTER TABLE orders ADD COLUMN total_cents_v2 bigint;
COMMIT;
&lt;/span&gt;&lt;span class="no"&gt;SQL
&lt;/span&gt;  &lt;span class="si"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nv"&gt;status&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;0 &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nv"&gt;status&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$?&lt;/span&gt;

  &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;[&lt;/span&gt; &lt;span class="nv"&gt;$status&lt;/span&gt; &lt;span class="nt"&gt;-eq&lt;/span&gt; 0 &lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;then
    &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"applied on attempt &lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;attempt&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
    &lt;span class="nb"&gt;exit &lt;/span&gt;0
  &lt;span class="k"&gt;fi&lt;/span&gt;

  &lt;span class="c"&gt;# Check specifically for lock_not_available (SQLSTATE 55P03 / lock timeout)&lt;/span&gt;
  &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;!&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$output&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; | &lt;span class="nb"&gt;grep&lt;/span&gt; &lt;span class="nt"&gt;-qE&lt;/span&gt; &lt;span class="s2"&gt;"55P03|canceling statement due to lock timeout"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;then
    &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"Fatal error on attempt &lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;attempt&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;: &lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;output&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&amp;amp;2
    &lt;span class="nb"&gt;exit&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$status&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
  &lt;span class="k"&gt;fi&lt;/span&gt;

  &lt;span class="c"&gt;# exponential backoff with jitter, capped at 30s&lt;/span&gt;
  &lt;span class="nv"&gt;backoff&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$((&lt;/span&gt; attempt &amp;lt; &lt;span class="m"&gt;6&lt;/span&gt; ? attempt &lt;span class="o"&gt;*&lt;/span&gt; attempt : &lt;span class="m"&gt;30&lt;/span&gt; &lt;span class="k"&gt;))&lt;/span&gt;
  &lt;span class="nb"&gt;sleep&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="k"&gt;$((&lt;/span&gt; backoff &lt;span class="o"&gt;+&lt;/span&gt; RANDOM &lt;span class="o"&gt;%&lt;/span&gt; &lt;span class="m"&gt;3&lt;/span&gt; &lt;span class="k"&gt;))&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
&lt;span class="k"&gt;done

&lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"could not acquire lock after &lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;MAX_ATTEMPTS&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; attempts"&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&amp;amp;2
&lt;span class="nb"&gt;exit &lt;/span&gt;1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three details that matter more than the loop itself:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;SET LOCAL&lt;/code&gt;&lt;strong&gt;, not&lt;/strong&gt;&lt;code&gt;SET&lt;/code&gt;&lt;strong&gt;.&lt;/strong&gt; Scoping the timeout to the transaction means a connection returned to a pool cannot leak a 50ms lock timeout into unrelated application queries. The exception is &lt;code&gt;CREATE INDEX CONCURRENTLY&lt;/code&gt;, which cannot run inside a transaction block at all, as that one needs a session-level &lt;code&gt;SET&lt;/code&gt; on a dedicated connection.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;One DDL statement per transaction.&lt;/strong&gt; Locks are held until commit. A transaction containing four &lt;code&gt;ALTER TABLE&lt;/code&gt; statements holds the first &lt;code&gt;ACCESS EXCLUSIVE&lt;/code&gt; lock for the duration of all four, which rebuilds the exact queue you were avoiding.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Jitter, not a fixed sleep.&lt;/strong&gt; Two migration runners retrying on identical two-second intervals will keep colliding with the same hourly reporting job forever.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If forty attempts fail, stop and look for the pattern instead of raising the timeout. On our bench the usual culprits are an hourly BI export, a stuck &lt;code&gt;idle in transaction&lt;/code&gt; session, or an anti-wraparound autovacuum, which, unlike normal autovacuum, does not auto-cancel itself when it blocks a conflicting lock request.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;lock_timeout&lt;/code&gt; and &lt;code&gt;statement_timeout&lt;/code&gt; are different controls. Setting &lt;code&gt;statement_timeout&lt;/code&gt; low for a &lt;code&gt;VALIDATE CONSTRAINT&lt;/code&gt; or a backfill batch will abort legitimate long work halfway through; setting &lt;code&gt;lock_timeout&lt;/code&gt; high turns a safe migration into an outage. Use a short &lt;code&gt;lock_timeout&lt;/code&gt; everywhere and a generous &lt;code&gt;statement_timeout&lt;/code&gt; only on statements that genuinely need minutes.&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;What rewrites 500GB, and what only touches a catalog row&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;Before planning phases, sort the change into one of three buckets. The difference between bucket one and bucket three is the difference between a two-millisecond migration and a two-hour full-table rewrite that needs double the disk.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;strong&gt;Change&lt;/strong&gt;&lt;/th&gt;
&lt;th&gt;&lt;strong&gt;Lock&lt;/strong&gt;&lt;/th&gt;
&lt;th&gt;&lt;strong&gt;Rewrites the heap?&lt;/strong&gt;&lt;/th&gt;
&lt;th&gt;&lt;strong&gt;Safe on 500GB?&lt;/strong&gt;&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;ADD COLUMN&lt;/code&gt; nullable, no default&lt;/td&gt;
&lt;td&gt;ACCESS EXCLUSIVE, instant&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;Yes, with &lt;code&gt;lock_timeout&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;ADD COLUMN ... DEFAULT &amp;lt;constant&amp;gt;&lt;/code&gt; (PG 11+)&lt;/td&gt;
&lt;td&gt;ACCESS EXCLUSIVE, instant&lt;/td&gt;
&lt;td&gt;No, stored as a missing value in the catalog&lt;/td&gt;
&lt;td&gt;Yes, with &lt;code&gt;lock_timeout&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;ADD COLUMN ... DEFAULT gen_random_uuid()&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;ACCESS EXCLUSIVE, held for the rewrite&lt;/td&gt;
&lt;td&gt;Yes, every row&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;ADD COLUMN ... GENERATED ALWAYS AS (...) STORED&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;ACCESS EXCLUSIVE, held for the rewrite&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;ALTER COLUMN TYPE int → bigint&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;ACCESS EXCLUSIVE, held for the rewrite&lt;/td&gt;
&lt;td&gt;Yes, plus all indexes&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;ALTER COLUMN TYPE varchar(50) → varchar(100)&lt;/code&gt; or &lt;code&gt;→ text&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;ACCESS EXCLUSIVE, instant&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;DROP COLUMN&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;ACCESS EXCLUSIVE, instant&lt;/td&gt;
&lt;td&gt;No (reclaimed on row update or pg_repack)&lt;/td&gt;
&lt;td&gt;Yes, with &lt;code&gt;lock_timeout&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;RENAME COLUMN&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;ACCESS EXCLUSIVE, instant&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;Catalog-safe, application-fatal&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;SET NOT NULL&lt;/code&gt; without a validated &lt;code&gt;CHECK&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;ACCESS EXCLUSIVE, held for a full scan&lt;/td&gt;
&lt;td&gt;No, but scans everything&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;ADD FOREIGN KEY&lt;/code&gt; without &lt;code&gt;NOT VALID&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;SHARE ROW EXCLUSIVE, held for a full scan&lt;/td&gt;
&lt;td&gt;No, but scans everything&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Two entries deserve a closer look. Since PostgreSQL 11, &lt;code&gt;ADD COLUMN&lt;/code&gt; with a non-volatile default is a catalog operation: the default is recorded in &lt;code&gt;pg_attrdef&lt;/code&gt; / &lt;code&gt;pg_attribute&lt;/code&gt; as a missing value and materialised lazily as rows are updated. Because &lt;code&gt;now()&lt;/code&gt; and &lt;code&gt;CURRENT_TIMESTAMP&lt;/code&gt; are &lt;code&gt;STABLE&lt;/code&gt; rather than &lt;code&gt;VOLATILE&lt;/code&gt;, &lt;code&gt;ADD COLUMN created_at timestamptz DEFAULT now()&lt;/code&gt; also uses this fast path. Every pre-existing row receives the timestamp of the &lt;code&gt;ALTER TABLE&lt;/code&gt; execution, while newly inserted rows receive their insertion timestamp. Swap that default for a &lt;code&gt;VOLATILE&lt;/code&gt; function like &lt;code&gt;clock_timestamp()&lt;/code&gt;, &lt;code&gt;random()&lt;/code&gt;, or &lt;code&gt;gen_random_uuid()&lt;/code&gt; and you force a full heap rewrite with &lt;code&gt;ACCESS EXCLUSIVE&lt;/code&gt; held throughout. On our 512GB bench table that rewrite ran 97 minutes and needed 1.04TB of free space at peak, because the old and new heaps coexist until commit.&lt;/p&gt;

&lt;p&gt;To verify whether a statement rewrites the heap before touching production, run it on a staging copy and compare &lt;code&gt;SELECT relfilenode FROM pg_class WHERE relname = 'orders';&lt;/code&gt; before and after. If &lt;code&gt;relfilenode&lt;/code&gt; changes, PostgreSQL rewrote the table on disk.&lt;/p&gt;

&lt;p&gt;The &lt;code&gt;RENAME COLUMN&lt;/code&gt; row is the one that fools people. It is instant and cheap at the database level, and it breaks every running application pod still issuing &lt;code&gt;SELECT old_name&lt;/code&gt; the moment it commits. Catalog-safe does not mean deployment-safe.&lt;/p&gt;

&lt;p&gt;Related On xSypher&lt;a href="https://dev.to/article/how-to-secure-mcp-servers-and-ai-agents-in-production"&gt;Cybersecurity / AI Security*&lt;em&gt;How to Secure MCP Servers and AI Agents in Production&lt;/em&gt;*&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;Adding NOT NULL and foreign keys without a full-table scan&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;A naive &lt;code&gt;SET NOT NULL&lt;/code&gt; on 1.1 billion rows holds &lt;code&gt;ACCESS EXCLUSIVE&lt;/code&gt; for the entire verification scan (41 minutes on our bench), with the table completely unreadable. The two-step pattern splits the expensive part away from the strong lock.&lt;/p&gt;

&lt;p&gt;Step one adds a &lt;code&gt;CHECK&lt;/code&gt; constraint that is not yet trusted. &lt;code&gt;NOT VALID&lt;/code&gt; means PostgreSQL enforces it on new and modified rows but does not scan existing ones, so the &lt;code&gt;ACCESS EXCLUSIVE&lt;/code&gt; lock is held for milliseconds.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;BEGIN&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="k"&gt;LOCAL&lt;/span&gt; &lt;span class="n"&gt;lock_timeout&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'50ms'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;orders&lt;/span&gt;
  &lt;span class="k"&gt;ADD&lt;/span&gt; &lt;span class="k"&gt;CONSTRAINT&lt;/span&gt; &lt;span class="n"&gt;orders_shipped_at_not_null&lt;/span&gt;
  &lt;span class="k"&gt;CHECK&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;shipped_at&lt;/span&gt; &lt;span class="k"&gt;IS&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;VALID&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;COMMIT&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Step two does the scan under &lt;code&gt;SHARE UPDATE EXCLUSIVE&lt;/code&gt;,&lt;a href="https://www.postgresql.org/docs/current/sql-altertable.html" rel="noopener noreferrer"&gt;documented in the ALTER TABLE reference&lt;/a&gt;. That mode conflicts with other &lt;code&gt;SHARE UPDATE EXCLUSIVE&lt;/code&gt; holders (vacuum, concurrent index builds, and other validations), but not with reads or writes. Run it in its own transaction, outside the deploy's critical path.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;lock_timeout&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'50ms'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;statement_timeout&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;          &lt;span class="c1"&gt;-- this one is allowed to take an hour&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;orders&lt;/span&gt; &lt;span class="n"&gt;VALIDATE&lt;/span&gt; &lt;span class="k"&gt;CONSTRAINT&lt;/span&gt; &lt;span class="n"&gt;orders_shipped_at_not_null&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On the bench this took 38 minutes and 14 seconds. Read p99 moved from 4.1ms to 5.3ms. No write errors, no queued sessions.&lt;/p&gt;

&lt;p&gt;Step three converts it. From PostgreSQL 12 onwards, &lt;code&gt;SET NOT NULL&lt;/code&gt; can prove its precondition from an existing validated &lt;code&gt;CHECK (col IS NOT NULL)&lt;/code&gt; constraint and skips the scan entirely — it still takes &lt;code&gt;ACCESS EXCLUSIVE&lt;/code&gt;, but only for the catalog update.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;BEGIN&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="k"&gt;LOCAL&lt;/span&gt; &lt;span class="n"&gt;lock_timeout&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'50ms'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;orders&lt;/span&gt; &lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;COLUMN&lt;/span&gt; &lt;span class="n"&gt;shipped_at&lt;/span&gt; &lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;COMMIT&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;BEGIN&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="k"&gt;LOCAL&lt;/span&gt; &lt;span class="n"&gt;lock_timeout&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'50ms'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;orders&lt;/span&gt; &lt;span class="k"&gt;DROP&lt;/span&gt; &lt;span class="k"&gt;CONSTRAINT&lt;/span&gt; &lt;span class="n"&gt;orders_shipped_at_not_null&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;COMMIT&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Crucially, execute the &lt;code&gt;SET NOT NULL&lt;/code&gt; and &lt;code&gt;DROP CONSTRAINT&lt;/code&gt; statements in separate transactions. Combining them in a single transaction causes PostgreSQL to re-evaluate the table, triggering a full table scan under an &lt;code&gt;ACCESS EXCLUSIVE&lt;/code&gt; lock.&lt;/p&gt;

&lt;p&gt;PostgreSQL 18 introduces native support for declaring &lt;code&gt;NOT NULL&lt;/code&gt; constraints as &lt;code&gt;NOT VALID&lt;/code&gt; directly, simplifying the pattern into two steps without requiring helper &lt;code&gt;CHECK&lt;/code&gt; constraints:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;BEGIN&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="k"&gt;LOCAL&lt;/span&gt; &lt;span class="n"&gt;lock_timeout&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'50ms'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;orders&lt;/span&gt;
  &lt;span class="k"&gt;ADD&lt;/span&gt; &lt;span class="k"&gt;CONSTRAINT&lt;/span&gt; &lt;span class="n"&gt;orders_shipped_at_not_null&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt; &lt;span class="n"&gt;shipped_at&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;VALID&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;COMMIT&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="c1"&gt;-- Later, separate session&lt;/span&gt;
&lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;lock_timeout&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'50ms'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;orders&lt;/span&gt; &lt;span class="n"&gt;VALIDATE&lt;/span&gt; &lt;span class="k"&gt;CONSTRAINT&lt;/span&gt; &lt;span class="n"&gt;orders_shipped_at_not_null&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Check your exact server minor version before using the PostgreSQL 18 syntax in deployment scripts, and maintain the three-step pattern for PostgreSQL 13 to 17.&lt;/p&gt;

&lt;p&gt;Foreign keys follow the same shape. The &lt;code&gt;NOT VALID&lt;/code&gt; form takes &lt;code&gt;SHARE ROW EXCLUSIVE&lt;/code&gt; on both tables (writes block briefly, reads do not), and the validation runs under the weaker mode:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;BEGIN&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="k"&gt;LOCAL&lt;/span&gt; &lt;span class="n"&gt;lock_timeout&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'50ms'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;orders&lt;/span&gt;
  &lt;span class="k"&gt;ADD&lt;/span&gt; &lt;span class="k"&gt;CONSTRAINT&lt;/span&gt; &lt;span class="n"&gt;orders_customer_fk&lt;/span&gt;
  &lt;span class="k"&gt;FOREIGN&lt;/span&gt; &lt;span class="k"&gt;KEY&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;customer_id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;REFERENCES&lt;/span&gt; &lt;span class="n"&gt;customers&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;VALID&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;COMMIT&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="c1"&gt;-- later, separate session&lt;/span&gt;
&lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;lock_timeout&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'50ms'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;orders&lt;/span&gt; &lt;span class="n"&gt;VALIDATE&lt;/span&gt; &lt;span class="k"&gt;CONSTRAINT&lt;/span&gt; &lt;span class="n"&gt;orders_customer_fk&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;NOT VALID&lt;/code&gt; is not a loophole or a permanently weakened constraint. New and modified rows are enforced from the moment it commits; only the historical scan is deferred. The only real risk is forgetting to run the validation, so put the &lt;code&gt;VALIDATE&lt;/code&gt; step in the same migration series rather than a ticket nobody picks up.&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;Changing a column type in four deployments, not one&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;Destructive changes (renames, type changes, splitting one column into two) cannot be done atomically on a live table without a rewrite. The expand-and-contract pattern spreads them across four deployments, each of which is individually reversible.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Phase 1: Expand.&lt;/strong&gt; Add the new column as nullable with no default, then install a trigger so every new write populates both columns. &lt;code&gt;CREATE TRIGGER&lt;/code&gt; takes &lt;code&gt;SHARE ROW EXCLUSIVE&lt;/code&gt;, which pauses writes for the instant it runs but leaves reads untouched; still wrap it in a lock timeout.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;OR&lt;/span&gt; &lt;span class="k"&gt;REPLACE&lt;/span&gt; &lt;span class="k"&gt;FUNCTION&lt;/span&gt; &lt;span class="n"&gt;orders_sync_total_cents&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="k"&gt;RETURNS&lt;/span&gt; &lt;span class="k"&gt;trigger&lt;/span&gt; &lt;span class="k"&gt;LANGUAGE&lt;/span&gt; &lt;span class="n"&gt;plpgsql&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="err"&gt;$$&lt;/span&gt;
&lt;span class="k"&gt;BEGIN&lt;/span&gt;
    &lt;span class="n"&gt;IF&lt;/span&gt; &lt;span class="n"&gt;TG_OP&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'INSERT'&lt;/span&gt; &lt;span class="k"&gt;THEN&lt;/span&gt;
        &lt;span class="n"&gt;IF&lt;/span&gt; &lt;span class="k"&gt;NEW&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;total_cents_v2&lt;/span&gt; &lt;span class="k"&gt;IS&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt; &lt;span class="k"&gt;THEN&lt;/span&gt;
            &lt;span class="k"&gt;NEW&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;total_cents_v2&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;NEW&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;total_cents&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="nb"&gt;bigint&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
        &lt;span class="k"&gt;END&lt;/span&gt; &lt;span class="n"&gt;IF&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="n"&gt;ELSIF&lt;/span&gt; &lt;span class="k"&gt;NEW&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;total_cents&lt;/span&gt; &lt;span class="k"&gt;IS&lt;/span&gt; &lt;span class="k"&gt;DISTINCT&lt;/span&gt; &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="k"&gt;OLD&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;total_cents&lt;/span&gt; &lt;span class="k"&gt;THEN&lt;/span&gt;
        &lt;span class="k"&gt;NEW&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;total_cents_v2&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;NEW&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;total_cents&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="nb"&gt;bigint&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="k"&gt;END&lt;/span&gt; &lt;span class="n"&gt;IF&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="k"&gt;RETURN&lt;/span&gt; &lt;span class="k"&gt;NEW&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;END&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="err"&gt;$$&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;BEGIN&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="k"&gt;LOCAL&lt;/span&gt; &lt;span class="n"&gt;lock_timeout&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'50ms'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;TRIGGER&lt;/span&gt; &lt;span class="n"&gt;orders_sync_total_cents_trg&lt;/span&gt;
&lt;span class="k"&gt;BEFORE&lt;/span&gt; &lt;span class="k"&gt;INSERT&lt;/span&gt; &lt;span class="k"&gt;OR&lt;/span&gt; &lt;span class="k"&gt;UPDATE&lt;/span&gt; &lt;span class="k"&gt;OF&lt;/span&gt; &lt;span class="n"&gt;total_cents&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;orders&lt;/span&gt;
&lt;span class="k"&gt;FOR&lt;/span&gt; &lt;span class="k"&gt;EACH&lt;/span&gt; &lt;span class="k"&gt;ROW&lt;/span&gt; &lt;span class="k"&gt;EXECUTE&lt;/span&gt; &lt;span class="k"&gt;FUNCTION&lt;/span&gt; &lt;span class="n"&gt;orders_sync_total_cents&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
&lt;span class="k"&gt;COMMIT&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A &lt;code&gt;BEFORE&lt;/code&gt; row trigger on a hot write path is not free. On our bench it added 0.21ms to mean &lt;code&gt;INSERT&lt;/code&gt; latency at 900 writes per second. This is acceptable for the days it stays installed, and it is the reason you drop it in phase 4. Application-level dual-write is the alternative, and it is the better choice when the transformation needs business logic the database does not have; the trade-off is that any writer you forget about (a cron job, an admin console, or a Kafka consumer) silently skips it. The trigger cannot be bypassed.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Phase 2: Backfill.&lt;/strong&gt; Covered in the next section, because it is where most teams actually break production.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Phase 3: Switch.&lt;/strong&gt; Deploy the application reading &lt;code&gt;total_cents_v2&lt;/code&gt;, keeping the dual write active. This is the rollback point: if the new column is wrong, you redeploy the previous version and nothing is lost. Before moving on, prove that nothing reads the old column. With &lt;code&gt;pg_stat_statements&lt;/code&gt; enabled:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;calls&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;left&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;80&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;query&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt;   &lt;span class="n"&gt;pg_stat_statements&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt;  &lt;span class="n"&gt;query&lt;/span&gt; &lt;span class="k"&gt;ILIKE&lt;/span&gt; &lt;span class="s1"&gt;'%total_cents%'&lt;/span&gt;
  &lt;span class="k"&gt;AND&lt;/span&gt;  &lt;span class="n"&gt;query&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;ILIKE&lt;/span&gt; &lt;span class="s1"&gt;'%total_cents_v2%'&lt;/span&gt;
&lt;span class="k"&gt;ORDER&lt;/span&gt;  &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;calls&lt;/span&gt; &lt;span class="k"&gt;DESC&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Reset the stats, wait a full business cycle (ours is seven days, because of weekly finance exports), and check again. A monthly batch job is the single most common reason a "dead" column turns out to be load-bearing.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Phase 4: Contract.&lt;/strong&gt; Drop the trigger, drop the function, drop the old column. All catalog operations, all behind a lock timeout. &lt;code&gt;DROP COLUMN&lt;/code&gt; does not reclaim space; the dead column data stays in each tuple until rows are rewritten, so schedule a &lt;code&gt;pg_repack&lt;/code&gt; run if you need the disk back before the table naturally churns.&lt;/p&gt;

&lt;p&gt;Related On xSypher&lt;a href="https://dev.to/article/why-we-moved-from-serverless-postgres-to-self-hosted-docker-pgbouncer-914ms-to-2ms-p95"&gt;Programming*&lt;em&gt;Why We Moved from Serverless Postgres to Self-Hosted Docker + PgBouncer (914ms to 2ms p95)&lt;/em&gt;*&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Do not compress phase 3 and phase 4 into one deployment. Dropping the legacy column while any pod from the previous release is still serving traffic produces immediate &lt;code&gt;column does not exist&lt;/code&gt; errors on every query that referenced it, and rolling back a dropped column means restoring from backup.&lt;/p&gt;

&lt;p&gt;Promoting an &lt;code&gt;integer&lt;/code&gt; primary key to &lt;code&gt;bigint&lt;/code&gt; requires one extra move during the switch. Build a unique index on the new column with &lt;code&gt;CREATE UNIQUE INDEX CONCURRENTLY&lt;/code&gt;, promote it inside a short &lt;code&gt;lock_timeout&lt;/code&gt; transaction using &lt;code&gt;ALTER TABLE orders DROP CONSTRAINT orders_pkey, ADD CONSTRAINT orders_pkey PRIMARY KEY USING INDEX orders_id_v2_uidx;&lt;/code&gt;, and repeat the two-step &lt;code&gt;NOT VALID&lt;/code&gt; foreign key swap for every child table referencing it.&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;Backfills that do not bloat the table or melt the replicas&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;A single &lt;code&gt;UPDATE orders SET total_cents_v2 = total_cents;&lt;/code&gt; on 1.1 billion rows is the most reliable way to turn a safe migration into an incident. Under MVCC, every updated row produces a new tuple version while the old one stays visible until vacuum reclaims it. The practical consequences on a 512GB table:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The heap roughly doubles. Our bench peaked at 963GB before vacuum caught up.&lt;/li&gt;
&lt;li&gt;Autovacuum arrives late. The default &lt;code&gt;autovacuum_vacuum_scale_factor&lt;/code&gt; of 0.2 means it waits for about 20% of the table to be dead (over 200 million tuples here) and then competes with your backfill for IOPS.&lt;/li&gt;
&lt;li&gt;WAL output spikes. We measured 418GB of WAL from one unbatched run, which pushed the streaming replica 11 minutes behind and broke read-after-write assumptions across the API.&lt;/li&gt;
&lt;li&gt;The transaction cannot be interrupted cheaply. Cancelling after 70 minutes rolls back everything and leaves the bloat behind.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Keyset pagination fixes all four. Batches commit, so dead tuples become reclaimable immediately; WAL is generated in digestible chunks; and a cancelled run resumes from the last cursor instead of restarting.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="c1"&gt;-- :last_id starts at 0 and :ceiling_id is snapshotted at startup&lt;/span&gt;
&lt;span class="k"&gt;WITH&lt;/span&gt; &lt;span class="n"&gt;batch&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;MATERIALIZED&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt;
    &lt;span class="k"&gt;FROM&lt;/span&gt;   &lt;span class="n"&gt;orders&lt;/span&gt;
    &lt;span class="k"&gt;WHERE&lt;/span&gt;  &lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;last_id&lt;/span&gt;
      &lt;span class="k"&gt;AND&lt;/span&gt;  &lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;ceiling_id&lt;/span&gt;
    &lt;span class="k"&gt;ORDER&lt;/span&gt;  &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt;
    &lt;span class="k"&gt;LIMIT&lt;/span&gt;  &lt;span class="mi"&gt;5000&lt;/span&gt;
&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;upd&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="k"&gt;UPDATE&lt;/span&gt; &lt;span class="n"&gt;orders&lt;/span&gt; &lt;span class="n"&gt;o&lt;/span&gt;
    &lt;span class="k"&gt;SET&lt;/span&gt;    &lt;span class="n"&gt;total_cents_v2&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;o&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;total_cents&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="nb"&gt;bigint&lt;/span&gt;
    &lt;span class="k"&gt;FROM&lt;/span&gt;   &lt;span class="n"&gt;batch&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt;
    &lt;span class="k"&gt;WHERE&lt;/span&gt;  &lt;span class="n"&gt;o&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;
      &lt;span class="k"&gt;AND&lt;/span&gt;  &lt;span class="n"&gt;o&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;total_cents_v2&lt;/span&gt; &lt;span class="k"&gt;IS&lt;/span&gt; &lt;span class="k"&gt;DISTINCT&lt;/span&gt; &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;o&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;total_cents&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="nb"&gt;bigint&lt;/span&gt;
    &lt;span class="n"&gt;RETURNING&lt;/span&gt; &lt;span class="n"&gt;o&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="k"&gt;max&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;last_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
       &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="k"&gt;count&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;upd&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;rows_updated&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt;   &lt;span class="n"&gt;batch&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three details make this query pattern reliable at scale:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Return&lt;/strong&gt;&lt;code&gt;max(id)&lt;/code&gt;&lt;strong&gt;from the batch CTE, not the update.&lt;/strong&gt; If every row in a window has already been backfilled, the update returns zero rows. Deriving your next cursor from the &lt;code&gt;UPDATE&lt;/code&gt; result would cause the cursor to stall on that range forever, whereas deriving it from the &lt;code&gt;batch&lt;/code&gt; CTE guarantees steady forward progress.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Snapshot&lt;/strong&gt;&lt;code&gt;ceiling_id&lt;/code&gt;&lt;strong&gt;at startup and force&lt;/strong&gt;&lt;code&gt;AS MATERIALIZED&lt;/code&gt;&lt;strong&gt;.&lt;/strong&gt; Querying against a fixed upper bound prevents newly inserted rows from extending the scan indefinitely while your backfill runs. Marking the batch CTE &lt;code&gt;AS MATERIALIZED&lt;/code&gt; prevents PostgreSQL from inlining the subquery and re-evaluating the limit during the update stage.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Avoid&lt;/strong&gt;&lt;code&gt;ctid&lt;/code&gt;&lt;strong&gt;pagination or&lt;/strong&gt;&lt;code&gt;FOR UPDATE SKIP LOCKED&lt;/code&gt;&lt;strong&gt;.&lt;/strong&gt; Physical row pointers (&lt;code&gt;ctid&lt;/code&gt;) change as tuples are updated and vacuumed, creating skippable or duplicate passes. &lt;code&gt;SKIP LOCKED&lt;/code&gt; bypasses rows held by concurrent transactions, leaving subtle gaps in your backfilled data. Keyset pagination on an indexed primary key remains the only deterministic method.&lt;/p&gt;

&lt;p&gt;Two tuning knobs make a measurable difference. First, do not index the new column until the backfill finishes. An unindexed target column lets PostgreSQL use heap-only tuple updates, keeping the new row version on the same page and avoiding index churn, provided the page has free space. Second, lower the autovacuum threshold for this table while the backfill runs:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;orders&lt;/span&gt; &lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="n"&gt;autovacuum_vacuum_scale_factor&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;01&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;autovacuum_vacuum_cost_delay&lt;/span&gt;   &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  &lt;strong&gt;Picking a batch size&lt;/strong&gt;
&lt;/h3&gt;

&lt;p&gt;Batch size is a trade-off between total runtime and the lag you are willing to push onto replicas. These are our bench numbers at a steady 900 writes per second of concurrent production-shaped load.&lt;/p&gt;

&lt;p&gt;Bar chart of Mean batch duration (x100 ms), Peak replica lag (s), Read p99 during run (ms) across 4 categories&lt;/p&gt;

&lt;p&gt;The shape is non-linear: going from 5,000 to 25,000 rows per batch cut total runtime by about a third but multiplied replica lag by six. 5,000 rows is the setting we ship with, and the throttle matters more than the batch size.&lt;/p&gt;

&lt;p&gt;The throttle is a lag check between batches, not a fixed sleep:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;coalesce&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;max&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pg_wal_lsn_diff&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pg_current_wal_lsn&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="n"&gt;replay_lsn&lt;/span&gt;&lt;span class="p"&gt;)),&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;lag_bytes&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt;   &lt;span class="n"&gt;pg_stat_replication&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Runner logic: if &lt;code&gt;lag_bytes&lt;/code&gt; exceeds roughly 256MB, sleep until it drops below 64MB, then continue. On the bench this converted a 34-second worst-case lag into a steady 0.6 to 1.3 seconds across an eight-hour run, at the cost of about 50 minutes of extra wall-clock time. Add a second guard for cloud volumes with burst credits, as sustained backfill IOPS on a gp3 volume sized for normal traffic will quietly flatten your read latency long before the database complains.&lt;/p&gt;

&lt;p&gt;Run the backfill from a supervised worker with a persisted cursor, not from a migration framework's single-transaction step. Every migration tool that wraps steps in one transaction will undo eight hours of batched work the moment a batch fails on row 900 million.&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;Index builds:&lt;/strong&gt;&lt;code&gt;CONCURRENTLY&lt;/code&gt;&lt;strong&gt;, and the invalid index you find at 3 a.m.&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;A plain &lt;code&gt;CREATE INDEX&lt;/code&gt; takes a &lt;code&gt;SHARE&lt;/code&gt; lock, which blocks every write for the duration of the build (54 minutes for a single-column btree on our bench table). &lt;code&gt;CREATE INDEX CONCURRENTLY&lt;/code&gt; takes &lt;code&gt;SHARE UPDATE EXCLUSIVE&lt;/code&gt; instead, so reads and writes continue. The cost is two table scans instead of one and a longer total runtime (81 minutes on the same index), plus a set of rules that are easy to get wrong:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="c1"&gt;-- session-level SET: CIC cannot run inside a transaction block&lt;/span&gt;
&lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;lock_timeout&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'50ms'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;statement_timeout&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;INDEX&lt;/span&gt; &lt;span class="n"&gt;CONCURRENTLY&lt;/span&gt; &lt;span class="n"&gt;idx_orders_total_cents_v2&lt;/span&gt;
    &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;orders&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;total_cents_v2&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;It cannot run inside a transaction block.&lt;/strong&gt; Not in &lt;code&gt;BEGIN ... COMMIT&lt;/code&gt;, not in a &lt;code&gt;DO&lt;/code&gt; block, and not inside any migration tool that wraps steps in a transaction by default. Use framework-specific escape hatches such as &lt;code&gt;disable_ddl_transaction!&lt;/code&gt; in Rails, &lt;code&gt;atomic = False&lt;/code&gt; in Django, &lt;code&gt;runInTransaction="false"&lt;/code&gt; in Liquibase, &lt;code&gt;autocommit=True&lt;/code&gt; in Alembic, or non-transactional script execution settings in Flyway.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;It waits for existing transactions to finish.&lt;/strong&gt; A long-running transaction anywhere (including &lt;code&gt;idle in transaction&lt;/code&gt; sessions and transactions touching unrelated tables) will stall the build at a phase boundary. Clear long transactions first.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Failures leave an invalid index behind.&lt;/strong&gt;&lt;a href="https://www.postgresql.org/docs/current/sql-createindex.html" rel="noopener noreferrer"&gt;The CREATE INDEX documentation&lt;/a&gt; spells this out: if a concurrent build fails or is canceled (such as from a lock timeout or canceled query), it leaves behind an invalid index marked in &lt;code&gt;pg_index&lt;/code&gt; that is ignored by the query planner but continuously updated during writes, incurring write amplification without providing read benefits.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Finding them is a one-liner, and it belongs in your monitoring rather than your memory:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;relname&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="k"&gt;table_name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
       &lt;span class="k"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;relname&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;index_name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
       &lt;span class="n"&gt;pg_size_pretty&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pg_relation_size&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;indexrelid&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="k"&gt;size&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt;   &lt;span class="n"&gt;pg_index&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;
&lt;span class="k"&gt;JOIN&lt;/span&gt;   &lt;span class="n"&gt;pg_class&lt;/span&gt; &lt;span class="k"&gt;c&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="k"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;oid&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;indexrelid&lt;/span&gt;
&lt;span class="k"&gt;JOIN&lt;/span&gt;   &lt;span class="n"&gt;pg_class&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;oid&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;indrelid&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt;  &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;indisvalid&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;While a build is running, track its current scan phase and snapshot lockers in real time via &lt;code&gt;pg_stat_progress_create_index&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;pid&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;phase&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;lockers_total&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;lockers_done&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;blocks_total&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;blocks_done&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tuples_total&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tuples_done&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt;   &lt;span class="n"&gt;pg_stat_progress_create_index&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt;  &lt;span class="n"&gt;relid&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'orders'&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="n"&gt;regclass&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Before dropping an invalid index, verify that it is not attached to a &lt;code&gt;UNIQUE&lt;/code&gt; or &lt;code&gt;PRIMARY KEY&lt;/code&gt; constraint (&lt;code&gt;SELECT conname, contype FROM pg_constraint WHERE conindid = 'idx_orders_total_cents_v2'::regclass;&lt;/code&gt;), because &lt;code&gt;DROP INDEX CONCURRENTLY&lt;/code&gt; cannot drop constraint-backed indexes. Once verified, clean up with &lt;code&gt;DROP INDEX CONCURRENTLY idx_orders_total_cents_v2;&lt;/code&gt; (also forbidden inside a transaction block) and rebuild. From PostgreSQL 12 onwards, &lt;code&gt;REINDEX INDEX CONCURRENTLY&lt;/code&gt; handles rebuilding a valid-but-bloated index in place without the drop-and-recreate window.&lt;/p&gt;

&lt;p&gt;Unique constraints need one extra move. Build the unique index concurrently first, then attach it as a constraint, which takes a brief &lt;code&gt;ACCESS EXCLUSIVE&lt;/code&gt; lock but performs no scan:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;UNIQUE&lt;/span&gt; &lt;span class="k"&gt;INDEX&lt;/span&gt; &lt;span class="n"&gt;CONCURRENTLY&lt;/span&gt; &lt;span class="n"&gt;orders_external_ref_uidx&lt;/span&gt;
    &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;orders&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;external_ref&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="k"&gt;BEGIN&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="k"&gt;LOCAL&lt;/span&gt; &lt;span class="n"&gt;lock_timeout&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'50ms'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;orders&lt;/span&gt;
  &lt;span class="k"&gt;ADD&lt;/span&gt; &lt;span class="k"&gt;CONSTRAINT&lt;/span&gt; &lt;span class="n"&gt;orders_external_ref_key&lt;/span&gt;
  &lt;span class="k"&gt;UNIQUE&lt;/span&gt; &lt;span class="k"&gt;USING&lt;/span&gt; &lt;span class="k"&gt;INDEX&lt;/span&gt; &lt;span class="n"&gt;orders_external_ref_uidx&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;COMMIT&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;CREATE INDEX CONCURRENTLY&lt;/code&gt; on a primary can extend query conflict windows on hot standby replicas running with &lt;code&gt;hot_standby_feedback&lt;/code&gt; disabled, and an index build holding &lt;code&gt;SHARE UPDATE EXCLUSIVE&lt;/code&gt; blocks autovacuum on that table for its entire duration. On a table that is simultaneously receiving a heavy backfill, that combination is how tables reach 2x bloat. Serialise the backfill and the index build; do not overlap them.&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;Run migrations on 5432, never on PgBouncer's 6432&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;This is the trap that survives every other precaution, because it makes correct SQL behave incorrectly.&lt;/p&gt;

&lt;p&gt;PgBouncer in transaction pooling mode (the default choice for high-connection-count deployments, usually on port 6432) hands a server connection back to the pool at the end of each transaction. Session state does not reliably survive that handoff.&lt;a href="https://www.pgbouncer.org/features.html" rel="noopener noreferrer"&gt;PgBouncer's own feature matrix&lt;/a&gt; lists session-level &lt;code&gt;SET&lt;/code&gt; among the features that are not supported in transaction pooling, and the consequences for migrations are specific:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Session-level timeouts poison other clients.&lt;/strong&gt; Setting &lt;code&gt;lock_timeout&lt;/code&gt; or &lt;code&gt;statement_timeout&lt;/code&gt; at the session level alters the server connection state. When transaction pooling hands that connection to a subsequent application request, the restrictive timeout remains active and silently fails routine queries.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;CREATE INDEX CONCURRENTLY&lt;/code&gt;&lt;strong&gt;loses   its preceding session setup.&lt;/strong&gt; Because &lt;code&gt;SET statement_timeout = 0;&lt;/code&gt; and &lt;code&gt;CREATE INDEX CONCURRENTLY&lt;/code&gt; are sent as separate top-level statements outside a transaction block, transaction pooling can hand you a different backend connection right after the &lt;code&gt;SET&lt;/code&gt; finishes—running the 81-minute index build under the default statement timeout, or killing it mid-flight via PgBouncer's own &lt;code&gt;query_timeout&lt;/code&gt; and leaving an invalid index behind.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Session-scoped advisory locks break.&lt;/strong&gt; Migration scripts often use session-level advisory locks to ensure mutual exclusion during deployments. Under PgBouncer transaction pooling, releasing the connection back to the pool can drop the lock prematurely or leave it orphaned on a connection assigned to another client.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The fix is boring and absolute: give your migration runner its own connection string pointing at the primary's real PostgreSQL port (5432), using a dedicated role with a small connection limit and &lt;code&gt;pg_hba.conf&lt;/code&gt; rules restricted to your CI/CD runner's IP range so you can isolate and audit migration sessions in &lt;code&gt;pg_stat_activity&lt;/code&gt;&amp;nbsp;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# application&lt;/span&gt;
&lt;span class="nv"&gt;DATABASE_URL&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;postgres://app@pgbouncer.internal:6432/app

&lt;span class="c"&gt;# migrations: direct, session-safe&lt;/span&gt;
&lt;span class="nv"&gt;MIGRATION_DATABASE_URL&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;postgres://migrator@db-primary.internal:5432/app?application_name&lt;span class="o"&gt;=&lt;/span&gt;schema_migration
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If you must route through PgBouncer for network reasons, use a separate pool configured in &lt;strong&gt;session&lt;/strong&gt; pooling mode with a very small &lt;code&gt;pool_size&lt;/code&gt;, and accept that you are adding a component that can only hurt. On our bench, migrations run through a transaction-mode pool failed to honour session &lt;code&gt;lock_timeout&lt;/code&gt; in 3 of 10 runs. This does not happen every time, which is worse than always, because it passes in staging.&lt;/p&gt;

&lt;p&gt;Related On xSypher&lt;a href="https://dev.to/article/why-we-moved-from-serverless-postgres-to-self-hosted-docker-pgbouncer-914ms-to-2ms-p95"&gt;Programming*&lt;em&gt;Why We Moved from Serverless Postgres to Self-Hosted Docker + PgBouncer (914ms to 2ms p95)&lt;/em&gt;*&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;The order to run them in&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;For a type change on a 500GB table, here is the full sequence, in deployment order. Prerequisites first: a migration role with a direct 5432 connection, &lt;code&gt;pg_stat_statements&lt;/code&gt; enabled, a monitored replica lag metric, at least 20% free disk, and a verified recent backup.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Deploy 1 (minutes).&lt;/strong&gt; &lt;code&gt;ADD COLUMN&lt;/code&gt; nullable, no default, with &lt;code&gt;lock_timeout&lt;/code&gt; and retries. Install the dual-write trigger in a second transaction.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Verify.&lt;/strong&gt; Confirm new writes populate both columns. Query a handful of rows created in the last minute.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Backfill (hours to days).&lt;/strong&gt; Run the keyset worker with replica-lag throttling and a persisted cursor. Lower &lt;code&gt;autovacuum_vacuum_scale_factor&lt;/code&gt; for the table first; restore it afterwards.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Verify cursor completion.&lt;/strong&gt; Confirm your backfill worker reached &lt;code&gt;:ceiling_id&lt;/code&gt; with zero remaining rows in its final batch (avoid running an unindexed &lt;code&gt;SELECT count(*) WHERE total_cents_v2 IS NULL&lt;/code&gt; on the 512GB primary; let Step 6's &lt;code&gt;VALIDATE CONSTRAINT&lt;/code&gt; perform the single sequential verification scan).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Index (hours).&lt;/strong&gt; &lt;code&gt;CREATE INDEX CONCURRENTLY&lt;/code&gt; on the new column, after the backfill, not before. Check &lt;code&gt;pg_index.indisvalid&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Constrain (hours).&lt;/strong&gt; &lt;code&gt;ADD CONSTRAINT ... CHECK (total_cents_v2 IS NOT NULL) NOT VALID&lt;/code&gt; (or add a parity check &lt;code&gt;CHECK (total_cents_v2 = total_cents::bigint) NOT VALID&lt;/code&gt;), run &lt;code&gt;VALIDATE CONSTRAINT&lt;/code&gt; under &lt;code&gt;SHARE UPDATE EXCLUSIVE&lt;/code&gt;, then in separate statements run &lt;code&gt;SET NOT NULL&lt;/code&gt; and &lt;code&gt;DROP CONSTRAINT&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Deploy 2 (minutes).&lt;/strong&gt; Application reads the new column. Dual write stays on.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Soak (one full business cycle).&lt;/strong&gt; Watch &lt;code&gt;pg_stat_statements&lt;/code&gt; for any remaining reference to the legacy column.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Deploy 3 (minutes).&lt;/strong&gt; Drop the trigger and function, then &lt;code&gt;DROP COLUMN&lt;/code&gt;, each with &lt;code&gt;lock_timeout&lt;/code&gt; and retries.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Reclaim (optional).&lt;/strong&gt; &lt;code&gt;pg_repack&lt;/code&gt; if you need the disk space back before natural churn reclaims it.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Steps 1, 5, 6, 7 and 9 are independently revertible. Step 3 is resumable. There is no point in the sequence where a failure requires a restore, which is the actual definition of a safe migration.&lt;/p&gt;




&lt;h2&gt;
  
  
  &lt;strong&gt;Questions that come up in the migration review&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Does lock_timeout make my migration less likely to succeed?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;It makes each attempt less likely to succeed and the overall run far more likely to finish without an incident. On a table with a steady read load, our bench acquired &lt;code&gt;ACCESS EXCLUSIVE&lt;/code&gt; within 50ms on roughly 7 of 10 attempts. With exponential backoff across 40 attempts, the migration lands within a few minutes. The failure case is a retry loop that exits cleanly, not a stalled production database.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Can I skip the trigger and dual-write only from the application?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;You can, and it works well when the transformation needs logic the database lacks. The risk is coverage: every writer must be updated, including cron jobs, admin tools, data-import scripts and anything connecting with a shared credential. A &lt;code&gt;BEFORE&lt;/code&gt; row trigger costs a fraction of a millisecond per write and cannot be bypassed. Use the trigger unless you have a specific reason not to.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How do managed services change this?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Amazon RDS, Aurora, Cloud SQL and Azure Database all run standard PostgreSQL, so lock semantics are identical. What differs is the IOPS ceiling and the credit model underneath your volume, which decides how fast a backfill can safely run. Managed failover also adds a constraint: a mid-migration failover kills in-flight &lt;code&gt;CONCURRENTLY&lt;/code&gt; builds, so keep a resumable cursor and an invalid-index check.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Do tools like pgroll or pg-osc remove the need for this?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;They automate the pattern rather than replace it. pgroll uses versioned views to present old and new schemas simultaneously; pg-osc uses a shadow table and triggers. Both still require batched backfills, lock timeouts and a safe connection path, and both add their own operational behaviour you need to understand before a production run. Learn the manual sequence first.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What about partitioned tables?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Partitioning is the structural answer to this whole class of problem. DDL applies per partition, so locks are scoped to a few gigabytes rather than 500. Getting there on an existing large table is itself an expand-and-contract migration, usually with &lt;code&gt;ATTACH PARTITION&lt;/code&gt;. It is worth planning once a single table passes a few hundred gigabytes, because every future migration gets cheaper.&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;What a clean run looks like&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;When this works, the only evidence in your monitoring is a slightly raised read p99 for the duration of the backfill (single digits of milliseconds) and a replica lag line that oscillates between 0.5 and 1.5 seconds instead of spiking. No queued sessions in &lt;code&gt;pg_stat_activity&lt;/code&gt;, no connection pool alarms, no restarted pods, no &lt;code&gt;column does not exist&lt;/code&gt; errors in the application log.&lt;/p&gt;

&lt;p&gt;If an attempt fails instead, the failure is quiet and local: the retry loop exits after its last attempt, the schema is unchanged, and the blocking PID is sitting in your diagnostic query output waiting to be explained. Cancel the blocker if it is safe to cancel, move the window if it is not, and run it again. A migration you can abandon at any point without consequence is the only kind worth running against a 500GB table.&lt;/p&gt;

&lt;p&gt;An &lt;code&gt;ALTER TABLE orders ADD COLUMN status_v2 text;&lt;/code&gt; on a 512GB table executes in roughly two milliseconds. It updates one catalog row. It touches no heap pages. And it can still take an entire payments platform offline for four minutes, because the statement spends 240 seconds waiting for a lock that it never gets to use politely.&lt;/p&gt;

&lt;p&gt;The DDL is not the outage. The lock queue in front of it is. Everything below is about keeping migrations out of that queue: fail-fast lock timeouts with retries, &lt;code&gt;NOT VALID&lt;/code&gt; constraints, keyset-paginated backfills throttled against replica lag, &lt;code&gt;CONCURRENTLY&lt;/code&gt; index builds, and a direct connection that bypasses PgBouncer. Version behaviour described here holds for PostgreSQL 13 through 18; a few shortcuts are version-gated and flagged where they appear.&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;Two milliseconds of DDL, four minutes of downtime&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;PostgreSQL has eight table-level lock modes, and&lt;a href="https://www.postgresql.org/docs/current/explicit-locking.html" rel="noopener noreferrer"&gt;the conflict matrix in the explicit locking docs&lt;/a&gt; is the entire story. &lt;code&gt;SELECT&lt;/code&gt; takes &lt;code&gt;ACCESS SHARE&lt;/code&gt;. &lt;code&gt;INSERT&lt;/code&gt;, &lt;code&gt;UPDATE&lt;/code&gt; and &lt;code&gt;DELETE&lt;/code&gt; take &lt;code&gt;ROW EXCLUSIVE&lt;/code&gt;. Most forms of &lt;code&gt;ALTER TABLE&lt;/code&gt; take &lt;code&gt;ACCESS EXCLUSIVE&lt;/code&gt;, which conflicts with every other mode, including the one plain reads use.&lt;/p&gt;

&lt;p&gt;The failure mode is queue fairness. A lock request that cannot be granted immediately goes to the back of the wait queue for that relation, and new requests that conflict with anything already queued line up behind it rather than jumping ahead. So the sequence looks like this:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;An analytics &lt;code&gt;SELECT&lt;/code&gt; has been scanning &lt;code&gt;orders&lt;/code&gt; for eight seconds, holding &lt;code&gt;ACCESS SHARE&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Your migration asks for &lt;code&gt;ACCESS EXCLUSIVE&lt;/code&gt;. It conflicts, so it waits.&lt;/li&gt;
&lt;li&gt;The next &lt;code&gt;SELECT&lt;/code&gt; from your API conflicts with the &lt;em&gt;queued&lt;/em&gt; &lt;code&gt;ACCESS EXCLUSIVE&lt;/code&gt; request, so it waits too. So does the next one. And the next.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;At that point the table is effectively offline for reads and writes even though nothing is being modified. On our test bench (PostgreSQL 17 on 16 vCPU, 64GB RAM, gp3 storage at 12,000 provisioned IOPS, a 512GB &lt;code&gt;orders&lt;/code&gt; table with roughly 1.1 billion rows), we reproduced this with a synthetic load of 2,400 queries per second across 30 application pods holding 20 pooled connections each. The first queued query appeared at t+0ms. The pool was fully saturated at &lt;strong&gt;t+1.4s&lt;/strong&gt;. Health checks that shared the same pool started failing at t+9s, and the orchestrator began restarting pods, which made recovery slower than the original lock.&lt;/p&gt;

&lt;p&gt;The arithmetic is unforgiving: a pool of 600 connections and 2,400 qps fills in well under a second. Connection pool exhaustion, not lock duration, is what converts a stalled DDL into a full outage.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;strong&gt;Statement&lt;/strong&gt;&lt;/th&gt;
&lt;th&gt;&lt;strong&gt;Lock mode taken&lt;/strong&gt;&lt;/th&gt;
&lt;th&gt;&lt;strong&gt;Blocks reads?&lt;/strong&gt;&lt;/th&gt;
&lt;th&gt;&lt;strong&gt;Blocks writes?&lt;/strong&gt;&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;SELECT&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;ACCESS SHARE&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;INSERT&lt;/code&gt; / &lt;code&gt;UPDATE&lt;/code&gt; / &lt;code&gt;DELETE&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;&lt;code&gt;ROW EXCLUSIVE&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;CREATE INDEX CONCURRENTLY&lt;/code&gt;, &lt;code&gt;VALIDATE CONSTRAINT&lt;/code&gt;, &lt;code&gt;VACUUM&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;&lt;code&gt;SHARE UPDATE EXCLUSIVE&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;CREATE INDEX&lt;/code&gt; (plain)&lt;/td&gt;
&lt;td&gt;&lt;code&gt;SHARE&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;CREATE TRIGGER&lt;/code&gt;, &lt;code&gt;ADD FOREIGN KEY ... NOT VALID&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;&lt;code&gt;SHARE ROW EXCLUSIVE&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Most &lt;code&gt;ALTER TABLE&lt;/code&gt;, &lt;code&gt;DROP TABLE&lt;/code&gt;, &lt;code&gt;TRUNCATE&lt;/code&gt;, &lt;code&gt;REINDEX&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;&lt;code&gt;ACCESS EXCLUSIVE&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The practical read of that table: anything in the bottom row needs to be acquired instantly or not at all, and anything in the &lt;code&gt;SHARE UPDATE EXCLUSIVE&lt;/code&gt; row can run for an hour without a reader noticing.&lt;/p&gt;

&lt;h3&gt;
  
  
  &lt;strong&gt;Seeing the queue while it is happening&lt;/strong&gt;
&lt;/h3&gt;

&lt;p&gt;Keep this query in a snippet file, because you will want it under pressure rather than after.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;pid&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
       &lt;span class="k"&gt;state&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
       &lt;span class="n"&gt;wait_event_type&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
       &lt;span class="n"&gt;wait_event&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
       &lt;span class="n"&gt;now&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;query_start&lt;/span&gt;   &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;runtime&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
       &lt;span class="n"&gt;pg_blocking_pids&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pid&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;blocked_by&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
       &lt;span class="k"&gt;left&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;70&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;       &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;query&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt;   &lt;span class="n"&gt;pg_stat_activity&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt;  &lt;span class="n"&gt;backend_type&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'client backend'&lt;/span&gt;
  &lt;span class="k"&gt;AND&lt;/span&gt;  &lt;span class="n"&gt;datname&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;current_database&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="k"&gt;ORDER&lt;/span&gt;  &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;runtime&lt;/span&gt; &lt;span class="k"&gt;DESC&lt;/span&gt; &lt;span class="n"&gt;NULLS&lt;/span&gt; &lt;span class="k"&gt;LAST&lt;/span&gt;
&lt;span class="k"&gt;LIMIT&lt;/span&gt;  &lt;span class="mi"&gt;25&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The root blocker is the PID that appears inside other sessions' &lt;code&gt;blocked_by&lt;/code&gt; arrays but has an empty array of its own. In our bench that was consistently a reporting query or an &lt;code&gt;idle in transaction&lt;/code&gt; session left open by an ORM, not the migration itself. Cancelling the DDL with &lt;code&gt;pg_cancel_backend()&lt;/code&gt; drains the queue in milliseconds; cancelling the original long reader does not, because the DDL simply takes its place at the head.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;pg_terminate_backend()&lt;/code&gt; on the wrong PID kills an application connection mid-transaction and rolls back user-visible work. Try &lt;code&gt;pg_cancel_backend()&lt;/code&gt; first, and confirm the target PID with the query above rather than by position in a sorted list.&lt;/p&gt;

&lt;p&gt;Prevent leaked ORM transactions from blocking future deploys by enforcing &lt;code&gt;idle_in_transaction_session_timeout&lt;/code&gt; (for example, &lt;code&gt;60s&lt;/code&gt;) at the database or role level (&lt;code&gt;ALTER ROLE app_user SET idle_in_transaction_session_timeout='60s';&lt;/code&gt;). On PostgreSQL 17 and later, pair it with &lt;code&gt;transaction_timeout&lt;/code&gt; to cap total transaction duration across both active and idle states.&lt;/p&gt;

&lt;p&gt;What a lock pile-up looks like in practice: one long-running reader (PID 18402), one waiting ALTER TABLE (PID 29104), and ordinary application queries stacked behind the migration in the relation lock queue.&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;Set lock_timeout to 50ms, then retry like you mean it&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;The fix for the queue is refusing to stand in it. &lt;code&gt;lock_timeout&lt;/code&gt; aborts a statement that cannot acquire its lock within the configured interval, raising SQLSTATE &lt;code&gt;55P03&lt;/code&gt; (&lt;code&gt;lock_not_available&lt;/code&gt;). A migration that fails fast and retries costs you a few wasted attempts; a migration that waits costs you the service.&lt;/p&gt;

&lt;p&gt;Fifty milliseconds is the number we use on the bench. It is long enough to succeed on an idle-ish table, short enough that even a saturating 2,400 qps workload only queues a handful of statements per attempt. Anything above roughly 200ms starts producing visible p99 spikes in the API.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;#!/usr/bin/env bash&lt;/span&gt;
&lt;span class="nb"&gt;set&lt;/span&gt; &lt;span class="nt"&gt;-euo&lt;/span&gt; pipefail

&lt;span class="c"&gt;# Port 5432: the primary directly, never PgBouncer on 6432.&lt;/span&gt;
&lt;span class="nv"&gt;PGURI&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"postgres://migrator@db-primary.internal:5432/app"&lt;/span&gt;
&lt;span class="nv"&gt;MAX_ATTEMPTS&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;40

&lt;span class="k"&gt;for &lt;/span&gt;attempt &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="si"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;seq &lt;/span&gt;1 &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$MAX_ATTEMPTS&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="si"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;do
  &lt;/span&gt;&lt;span class="nv"&gt;output&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;$(&lt;/span&gt;psql &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$PGURI&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="nt"&gt;-v&lt;/span&gt; &lt;span class="nv"&gt;ON_ERROR_STOP&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;1 &lt;span class="nt"&gt;--quiet&lt;/span&gt; 2&amp;gt;&amp;amp;1 &lt;span class="o"&gt;&amp;lt;&amp;lt;&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="no"&gt;SQL&lt;/span&gt;&lt;span class="sh"&gt;'
BEGIN;
SET LOCAL lock_timeout = '50ms';
SET LOCAL statement_timeout = '5s';
ALTER TABLE orders ADD COLUMN total_cents_v2 bigint;
COMMIT;
&lt;/span&gt;&lt;span class="no"&gt;SQL
&lt;/span&gt;  &lt;span class="si"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nv"&gt;status&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;0 &lt;span class="o"&gt;||&lt;/span&gt; &lt;span class="nv"&gt;status&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$?&lt;/span&gt;

  &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;[&lt;/span&gt; &lt;span class="nv"&gt;$status&lt;/span&gt; &lt;span class="nt"&gt;-eq&lt;/span&gt; 0 &lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;then
    &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"applied on attempt &lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;attempt&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
    &lt;span class="nb"&gt;exit &lt;/span&gt;0
  &lt;span class="k"&gt;fi&lt;/span&gt;

  &lt;span class="c"&gt;# Check specifically for lock_not_available (SQLSTATE 55P03 / lock timeout)&lt;/span&gt;
  &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="o"&gt;!&lt;/span&gt; &lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$output&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; | &lt;span class="nb"&gt;grep&lt;/span&gt; &lt;span class="nt"&gt;-qE&lt;/span&gt; &lt;span class="s2"&gt;"55P03|canceling statement due to lock timeout"&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt; &lt;span class="k"&gt;then
    &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"Fatal error on attempt &lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;attempt&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;: &lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;output&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&amp;amp;2
    &lt;span class="nb"&gt;exit&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="nv"&gt;$status&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
  &lt;span class="k"&gt;fi&lt;/span&gt;

  &lt;span class="c"&gt;# exponential backoff with jitter, capped at 30s&lt;/span&gt;
  &lt;span class="nv"&gt;backoff&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$((&lt;/span&gt; attempt &amp;lt; &lt;span class="m"&gt;6&lt;/span&gt; ? attempt &lt;span class="o"&gt;*&lt;/span&gt; attempt : &lt;span class="m"&gt;30&lt;/span&gt; &lt;span class="k"&gt;))&lt;/span&gt;
  &lt;span class="nb"&gt;sleep&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="k"&gt;$((&lt;/span&gt; backoff &lt;span class="o"&gt;+&lt;/span&gt; RANDOM &lt;span class="o"&gt;%&lt;/span&gt; &lt;span class="m"&gt;3&lt;/span&gt; &lt;span class="k"&gt;))&lt;/span&gt;&lt;span class="s2"&gt;"&lt;/span&gt;
&lt;span class="k"&gt;done

&lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"could not acquire lock after &lt;/span&gt;&lt;span class="k"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;MAX_ATTEMPTS&lt;/span&gt;&lt;span class="k"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; attempts"&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&amp;amp;2
&lt;span class="nb"&gt;exit &lt;/span&gt;1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three details that matter more than the loop itself:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;SET LOCAL&lt;/code&gt;&lt;strong&gt;, not&lt;/strong&gt;&lt;code&gt;SET&lt;/code&gt;&lt;strong&gt;.&lt;/strong&gt; Scoping the timeout to the transaction means a connection returned to a pool cannot leak a 50ms lock timeout into unrelated application queries. The exception is &lt;code&gt;CREATE INDEX CONCURRENTLY&lt;/code&gt;, which cannot run inside a transaction block at all, as that one needs a session-level &lt;code&gt;SET&lt;/code&gt; on a dedicated connection.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;One DDL statement per transaction.&lt;/strong&gt; Locks are held until commit. A transaction containing four &lt;code&gt;ALTER TABLE&lt;/code&gt; statements holds the first &lt;code&gt;ACCESS EXCLUSIVE&lt;/code&gt; lock for the duration of all four, which rebuilds the exact queue you were avoiding.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Jitter, not a fixed sleep.&lt;/strong&gt; Two migration runners retrying on identical two-second intervals will keep colliding with the same hourly reporting job forever.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If forty attempts fail, stop and look for the pattern instead of raising the timeout. On our bench the usual culprits are an hourly BI export, a stuck &lt;code&gt;idle in transaction&lt;/code&gt; session, or an anti-wraparound autovacuum, which, unlike normal autovacuum, does not auto-cancel itself when it blocks a conflicting lock request.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;lock_timeout&lt;/code&gt; and &lt;code&gt;statement_timeout&lt;/code&gt; are different controls. Setting &lt;code&gt;statement_timeout&lt;/code&gt; low for a &lt;code&gt;VALIDATE CONSTRAINT&lt;/code&gt; or a backfill batch will abort legitimate long work halfway through; setting &lt;code&gt;lock_timeout&lt;/code&gt; high turns a safe migration into an outage. Use a short &lt;code&gt;lock_timeout&lt;/code&gt; everywhere and a generous &lt;code&gt;statement_timeout&lt;/code&gt; only on statements that genuinely need minutes.&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;What rewrites 500GB, and what only touches a catalog row&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;Before planning phases, sort the change into one of three buckets. The difference between bucket one and bucket three is the difference between a two-millisecond migration and a two-hour full-table rewrite that needs double the disk.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;strong&gt;Change&lt;/strong&gt;&lt;/th&gt;
&lt;th&gt;&lt;strong&gt;Lock&lt;/strong&gt;&lt;/th&gt;
&lt;th&gt;&lt;strong&gt;Rewrites the heap?&lt;/strong&gt;&lt;/th&gt;
&lt;th&gt;&lt;strong&gt;Safe on 500GB?&lt;/strong&gt;&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;ADD COLUMN&lt;/code&gt; nullable, no default&lt;/td&gt;
&lt;td&gt;ACCESS EXCLUSIVE, instant&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;Yes, with &lt;code&gt;lock_timeout&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;ADD COLUMN ... DEFAULT &amp;lt;constant&amp;gt;&lt;/code&gt; (PG 11+)&lt;/td&gt;
&lt;td&gt;ACCESS EXCLUSIVE, instant&lt;/td&gt;
&lt;td&gt;No, stored as a missing value in the catalog&lt;/td&gt;
&lt;td&gt;Yes, with &lt;code&gt;lock_timeout&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;ADD COLUMN ... DEFAULT gen_random_uuid()&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;ACCESS EXCLUSIVE, held for the rewrite&lt;/td&gt;
&lt;td&gt;Yes, every row&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;ADD COLUMN ... GENERATED ALWAYS AS (...) STORED&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;ACCESS EXCLUSIVE, held for the rewrite&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;ALTER COLUMN TYPE int → bigint&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;ACCESS EXCLUSIVE, held for the rewrite&lt;/td&gt;
&lt;td&gt;Yes, plus all indexes&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;ALTER COLUMN TYPE varchar(50) → varchar(100)&lt;/code&gt; or &lt;code&gt;→ text&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;ACCESS EXCLUSIVE, instant&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;DROP COLUMN&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;ACCESS EXCLUSIVE, instant&lt;/td&gt;
&lt;td&gt;No (reclaimed on row update or pg_repack)&lt;/td&gt;
&lt;td&gt;Yes, with &lt;code&gt;lock_timeout&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;RENAME COLUMN&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;ACCESS EXCLUSIVE, instant&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;Catalog-safe, application-fatal&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;SET NOT NULL&lt;/code&gt; without a validated &lt;code&gt;CHECK&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;ACCESS EXCLUSIVE, held for a full scan&lt;/td&gt;
&lt;td&gt;No, but scans everything&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;
&lt;code&gt;ADD FOREIGN KEY&lt;/code&gt; without &lt;code&gt;NOT VALID&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;SHARE ROW EXCLUSIVE, held for a full scan&lt;/td&gt;
&lt;td&gt;No, but scans everything&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Two entries deserve a closer look. Since PostgreSQL 11, &lt;code&gt;ADD COLUMN&lt;/code&gt; with a non-volatile default is a catalog operation: the default is recorded in &lt;code&gt;pg_attrdef&lt;/code&gt; / &lt;code&gt;pg_attribute&lt;/code&gt; as a missing value and materialised lazily as rows are updated. Because &lt;code&gt;now()&lt;/code&gt; and &lt;code&gt;CURRENT_TIMESTAMP&lt;/code&gt; are &lt;code&gt;STABLE&lt;/code&gt; rather than &lt;code&gt;VOLATILE&lt;/code&gt;, &lt;code&gt;ADD COLUMN created_at timestamptz DEFAULT now()&lt;/code&gt; also uses this fast path. Every pre-existing row receives the timestamp of the &lt;code&gt;ALTER TABLE&lt;/code&gt; execution, while newly inserted rows receive their insertion timestamp. Swap that default for a &lt;code&gt;VOLATILE&lt;/code&gt; function like &lt;code&gt;clock_timestamp()&lt;/code&gt;, &lt;code&gt;random()&lt;/code&gt;, or &lt;code&gt;gen_random_uuid()&lt;/code&gt; and you force a full heap rewrite with &lt;code&gt;ACCESS EXCLUSIVE&lt;/code&gt; held throughout. On our 512GB bench table that rewrite ran 97 minutes and needed 1.04TB of free space at peak, because the old and new heaps coexist until commit.&lt;/p&gt;

&lt;p&gt;To verify whether a statement rewrites the heap before touching production, run it on a staging copy and compare &lt;code&gt;SELECT relfilenode FROM pg_class WHERE relname = 'orders';&lt;/code&gt; before and after. If &lt;code&gt;relfilenode&lt;/code&gt; changes, PostgreSQL rewrote the table on disk.&lt;/p&gt;

&lt;p&gt;The &lt;code&gt;RENAME COLUMN&lt;/code&gt; row is the one that fools people. It is instant and cheap at the database level, and it breaks every running application pod still issuing &lt;code&gt;SELECT old_name&lt;/code&gt; the moment it commits. Catalog-safe does not mean deployment-safe.&lt;/p&gt;

&lt;p&gt;Related On xSypher&lt;a href="https://dev.to/article/how-to-secure-mcp-servers-and-ai-agents-in-production"&gt;Cybersecurity / AI Security*&lt;em&gt;How to Secure MCP Servers and AI Agents in Production&lt;/em&gt;*&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;Adding NOT NULL and foreign keys without a full-table scan&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;A naive &lt;code&gt;SET NOT NULL&lt;/code&gt; on 1.1 billion rows holds &lt;code&gt;ACCESS EXCLUSIVE&lt;/code&gt; for the entire verification scan (41 minutes on our bench), with the table completely unreadable. The two-step pattern splits the expensive part away from the strong lock.&lt;/p&gt;

&lt;p&gt;Step one adds a &lt;code&gt;CHECK&lt;/code&gt; constraint that is not yet trusted. &lt;code&gt;NOT VALID&lt;/code&gt; means PostgreSQL enforces it on new and modified rows but does not scan existing ones, so the &lt;code&gt;ACCESS EXCLUSIVE&lt;/code&gt; lock is held for milliseconds.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;BEGIN&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="k"&gt;LOCAL&lt;/span&gt; &lt;span class="n"&gt;lock_timeout&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'50ms'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;orders&lt;/span&gt;
  &lt;span class="k"&gt;ADD&lt;/span&gt; &lt;span class="k"&gt;CONSTRAINT&lt;/span&gt; &lt;span class="n"&gt;orders_shipped_at_not_null&lt;/span&gt;
  &lt;span class="k"&gt;CHECK&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;shipped_at&lt;/span&gt; &lt;span class="k"&gt;IS&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;VALID&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;COMMIT&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Step two does the scan under &lt;code&gt;SHARE UPDATE EXCLUSIVE&lt;/code&gt;,&lt;a href="https://www.postgresql.org/docs/current/sql-altertable.html" rel="noopener noreferrer"&gt;documented in the ALTER TABLE reference&lt;/a&gt;. That mode conflicts with other &lt;code&gt;SHARE UPDATE EXCLUSIVE&lt;/code&gt; holders (vacuum, concurrent index builds, and other validations), but not with reads or writes. Run it in its own transaction, outside the deploy's critical path.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;lock_timeout&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'50ms'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;statement_timeout&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;          &lt;span class="c1"&gt;-- this one is allowed to take an hour&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;orders&lt;/span&gt; &lt;span class="n"&gt;VALIDATE&lt;/span&gt; &lt;span class="k"&gt;CONSTRAINT&lt;/span&gt; &lt;span class="n"&gt;orders_shipped_at_not_null&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;On the bench this took 38 minutes and 14 seconds. Read p99 moved from 4.1ms to 5.3ms. No write errors, no queued sessions.&lt;/p&gt;

&lt;p&gt;Step three converts it. From PostgreSQL 12 onwards, &lt;code&gt;SET NOT NULL&lt;/code&gt; can prove its precondition from an existing validated &lt;code&gt;CHECK (col IS NOT NULL)&lt;/code&gt; constraint and skips the scan entirely — it still takes &lt;code&gt;ACCESS EXCLUSIVE&lt;/code&gt;, but only for the catalog update.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;BEGIN&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="k"&gt;LOCAL&lt;/span&gt; &lt;span class="n"&gt;lock_timeout&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'50ms'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;orders&lt;/span&gt; &lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;COLUMN&lt;/span&gt; &lt;span class="n"&gt;shipped_at&lt;/span&gt; &lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;COMMIT&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;BEGIN&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="k"&gt;LOCAL&lt;/span&gt; &lt;span class="n"&gt;lock_timeout&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'50ms'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;orders&lt;/span&gt; &lt;span class="k"&gt;DROP&lt;/span&gt; &lt;span class="k"&gt;CONSTRAINT&lt;/span&gt; &lt;span class="n"&gt;orders_shipped_at_not_null&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;COMMIT&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Crucially, execute the &lt;code&gt;SET NOT NULL&lt;/code&gt; and &lt;code&gt;DROP CONSTRAINT&lt;/code&gt; statements in separate transactions. Combining them in a single transaction causes PostgreSQL to re-evaluate the table, triggering a full table scan under an &lt;code&gt;ACCESS EXCLUSIVE&lt;/code&gt; lock.&lt;/p&gt;

&lt;p&gt;PostgreSQL 18 introduces native support for declaring &lt;code&gt;NOT NULL&lt;/code&gt; constraints as &lt;code&gt;NOT VALID&lt;/code&gt; directly, simplifying the pattern into two steps without requiring helper &lt;code&gt;CHECK&lt;/code&gt; constraints:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;BEGIN&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="k"&gt;LOCAL&lt;/span&gt; &lt;span class="n"&gt;lock_timeout&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'50ms'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;orders&lt;/span&gt;
  &lt;span class="k"&gt;ADD&lt;/span&gt; &lt;span class="k"&gt;CONSTRAINT&lt;/span&gt; &lt;span class="n"&gt;orders_shipped_at_not_null&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt; &lt;span class="n"&gt;shipped_at&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;VALID&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;COMMIT&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="c1"&gt;-- Later, separate session&lt;/span&gt;
&lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;lock_timeout&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'50ms'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;orders&lt;/span&gt; &lt;span class="n"&gt;VALIDATE&lt;/span&gt; &lt;span class="k"&gt;CONSTRAINT&lt;/span&gt; &lt;span class="n"&gt;orders_shipped_at_not_null&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Check your exact server minor version before using the PostgreSQL 18 syntax in deployment scripts, and maintain the three-step pattern for PostgreSQL 13 to 17.&lt;/p&gt;

&lt;p&gt;Foreign keys follow the same shape. The &lt;code&gt;NOT VALID&lt;/code&gt; form takes &lt;code&gt;SHARE ROW EXCLUSIVE&lt;/code&gt; on both tables (writes block briefly, reads do not), and the validation runs under the weaker mode:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;BEGIN&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="k"&gt;LOCAL&lt;/span&gt; &lt;span class="n"&gt;lock_timeout&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'50ms'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;orders&lt;/span&gt;
  &lt;span class="k"&gt;ADD&lt;/span&gt; &lt;span class="k"&gt;CONSTRAINT&lt;/span&gt; &lt;span class="n"&gt;orders_customer_fk&lt;/span&gt;
  &lt;span class="k"&gt;FOREIGN&lt;/span&gt; &lt;span class="k"&gt;KEY&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;customer_id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;REFERENCES&lt;/span&gt; &lt;span class="n"&gt;customers&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;VALID&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;COMMIT&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="c1"&gt;-- later, separate session&lt;/span&gt;
&lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;lock_timeout&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'50ms'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;orders&lt;/span&gt; &lt;span class="n"&gt;VALIDATE&lt;/span&gt; &lt;span class="k"&gt;CONSTRAINT&lt;/span&gt; &lt;span class="n"&gt;orders_customer_fk&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;NOT VALID&lt;/code&gt; is not a loophole or a permanently weakened constraint. New and modified rows are enforced from the moment it commits; only the historical scan is deferred. The only real risk is forgetting to run the validation, so put the &lt;code&gt;VALIDATE&lt;/code&gt; step in the same migration series rather than a ticket nobody picks up.&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;Changing a column type in four deployments, not one&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;Destructive changes (renames, type changes, splitting one column into two) cannot be done atomically on a live table without a rewrite. The expand-and-contract pattern spreads them across four deployments, each of which is individually reversible.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Phase 1: Expand.&lt;/strong&gt; Add the new column as nullable with no default, then install a trigger so every new write populates both columns. &lt;code&gt;CREATE TRIGGER&lt;/code&gt; takes &lt;code&gt;SHARE ROW EXCLUSIVE&lt;/code&gt;, which pauses writes for the instant it runs but leaves reads untouched; still wrap it in a lock timeout.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;OR&lt;/span&gt; &lt;span class="k"&gt;REPLACE&lt;/span&gt; &lt;span class="k"&gt;FUNCTION&lt;/span&gt; &lt;span class="n"&gt;orders_sync_total_cents&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="k"&gt;RETURNS&lt;/span&gt; &lt;span class="k"&gt;trigger&lt;/span&gt; &lt;span class="k"&gt;LANGUAGE&lt;/span&gt; &lt;span class="n"&gt;plpgsql&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="err"&gt;$$&lt;/span&gt;
&lt;span class="k"&gt;BEGIN&lt;/span&gt;
    &lt;span class="n"&gt;IF&lt;/span&gt; &lt;span class="n"&gt;TG_OP&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'INSERT'&lt;/span&gt; &lt;span class="k"&gt;THEN&lt;/span&gt;
        &lt;span class="n"&gt;IF&lt;/span&gt; &lt;span class="k"&gt;NEW&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;total_cents_v2&lt;/span&gt; &lt;span class="k"&gt;IS&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt; &lt;span class="k"&gt;THEN&lt;/span&gt;
            &lt;span class="k"&gt;NEW&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;total_cents_v2&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;NEW&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;total_cents&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="nb"&gt;bigint&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
        &lt;span class="k"&gt;END&lt;/span&gt; &lt;span class="n"&gt;IF&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="n"&gt;ELSIF&lt;/span&gt; &lt;span class="k"&gt;NEW&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;total_cents&lt;/span&gt; &lt;span class="k"&gt;IS&lt;/span&gt; &lt;span class="k"&gt;DISTINCT&lt;/span&gt; &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="k"&gt;OLD&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;total_cents&lt;/span&gt; &lt;span class="k"&gt;THEN&lt;/span&gt;
        &lt;span class="k"&gt;NEW&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;total_cents_v2&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;NEW&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;total_cents&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="nb"&gt;bigint&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="k"&gt;END&lt;/span&gt; &lt;span class="n"&gt;IF&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="k"&gt;RETURN&lt;/span&gt; &lt;span class="k"&gt;NEW&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;END&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="err"&gt;$$&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;BEGIN&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="k"&gt;LOCAL&lt;/span&gt; &lt;span class="n"&gt;lock_timeout&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'50ms'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;TRIGGER&lt;/span&gt; &lt;span class="n"&gt;orders_sync_total_cents_trg&lt;/span&gt;
&lt;span class="k"&gt;BEFORE&lt;/span&gt; &lt;span class="k"&gt;INSERT&lt;/span&gt; &lt;span class="k"&gt;OR&lt;/span&gt; &lt;span class="k"&gt;UPDATE&lt;/span&gt; &lt;span class="k"&gt;OF&lt;/span&gt; &lt;span class="n"&gt;total_cents&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;orders&lt;/span&gt;
&lt;span class="k"&gt;FOR&lt;/span&gt; &lt;span class="k"&gt;EACH&lt;/span&gt; &lt;span class="k"&gt;ROW&lt;/span&gt; &lt;span class="k"&gt;EXECUTE&lt;/span&gt; &lt;span class="k"&gt;FUNCTION&lt;/span&gt; &lt;span class="n"&gt;orders_sync_total_cents&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
&lt;span class="k"&gt;COMMIT&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A &lt;code&gt;BEFORE&lt;/code&gt; row trigger on a hot write path is not free. On our bench it added 0.21ms to mean &lt;code&gt;INSERT&lt;/code&gt; latency at 900 writes per second. This is acceptable for the days it stays installed, and it is the reason you drop it in phase 4. Application-level dual-write is the alternative, and it is the better choice when the transformation needs business logic the database does not have; the trade-off is that any writer you forget about (a cron job, an admin console, or a Kafka consumer) silently skips it. The trigger cannot be bypassed.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Phase 2: Backfill.&lt;/strong&gt; Covered in the next section, because it is where most teams actually break production.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Phase 3: Switch.&lt;/strong&gt; Deploy the application reading &lt;code&gt;total_cents_v2&lt;/code&gt;, keeping the dual write active. This is the rollback point: if the new column is wrong, you redeploy the previous version and nothing is lost. Before moving on, prove that nothing reads the old column. With &lt;code&gt;pg_stat_statements&lt;/code&gt; enabled:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;calls&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="k"&gt;left&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;80&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;query&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt;   &lt;span class="n"&gt;pg_stat_statements&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt;  &lt;span class="n"&gt;query&lt;/span&gt; &lt;span class="k"&gt;ILIKE&lt;/span&gt; &lt;span class="s1"&gt;'%total_cents%'&lt;/span&gt;
  &lt;span class="k"&gt;AND&lt;/span&gt;  &lt;span class="n"&gt;query&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;ILIKE&lt;/span&gt; &lt;span class="s1"&gt;'%total_cents_v2%'&lt;/span&gt;
&lt;span class="k"&gt;ORDER&lt;/span&gt;  &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;calls&lt;/span&gt; &lt;span class="k"&gt;DESC&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Reset the stats, wait a full business cycle (ours is seven days, because of weekly finance exports), and check again. A monthly batch job is the single most common reason a "dead" column turns out to be load-bearing.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Phase 4: Contract.&lt;/strong&gt; Drop the trigger, drop the function, drop the old column. All catalog operations, all behind a lock timeout. &lt;code&gt;DROP COLUMN&lt;/code&gt; does not reclaim space; the dead column data stays in each tuple until rows are rewritten, so schedule a &lt;code&gt;pg_repack&lt;/code&gt; run if you need the disk back before the table naturally churns.&lt;/p&gt;

&lt;p&gt;Related On xSypher&lt;a href="https://dev.to/article/why-we-moved-from-serverless-postgres-to-self-hosted-docker-pgbouncer-914ms-to-2ms-p95"&gt;Programming*&lt;em&gt;Why We Moved from Serverless Postgres to Self-Hosted Docker + PgBouncer (914ms to 2ms p95)&lt;/em&gt;*&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Do not compress phase 3 and phase 4 into one deployment. Dropping the legacy column while any pod from the previous release is still serving traffic produces immediate &lt;code&gt;column does not exist&lt;/code&gt; errors on every query that referenced it, and rolling back a dropped column means restoring from backup.&lt;/p&gt;

&lt;p&gt;Promoting an &lt;code&gt;integer&lt;/code&gt; primary key to &lt;code&gt;bigint&lt;/code&gt; requires one extra move during the switch. Build a unique index on the new column with &lt;code&gt;CREATE UNIQUE INDEX CONCURRENTLY&lt;/code&gt;, promote it inside a short &lt;code&gt;lock_timeout&lt;/code&gt; transaction using &lt;code&gt;ALTER TABLE orders DROP CONSTRAINT orders_pkey, ADD CONSTRAINT orders_pkey PRIMARY KEY USING INDEX orders_id_v2_uidx;&lt;/code&gt;, and repeat the two-step &lt;code&gt;NOT VALID&lt;/code&gt; foreign key swap for every child table referencing it.&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;Backfills that do not bloat the table or melt the replicas&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;A single &lt;code&gt;UPDATE orders SET total_cents_v2 = total_cents;&lt;/code&gt; on 1.1 billion rows is the most reliable way to turn a safe migration into an incident. Under MVCC, every updated row produces a new tuple version while the old one stays visible until vacuum reclaims it. The practical consequences on a 512GB table:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The heap roughly doubles. Our bench peaked at 963GB before vacuum caught up.&lt;/li&gt;
&lt;li&gt;Autovacuum arrives late. The default &lt;code&gt;autovacuum_vacuum_scale_factor&lt;/code&gt; of 0.2 means it waits for about 20% of the table to be dead (over 200 million tuples here) and then competes with your backfill for IOPS.&lt;/li&gt;
&lt;li&gt;WAL output spikes. We measured 418GB of WAL from one unbatched run, which pushed the streaming replica 11 minutes behind and broke read-after-write assumptions across the API.&lt;/li&gt;
&lt;li&gt;The transaction cannot be interrupted cheaply. Cancelling after 70 minutes rolls back everything and leaves the bloat behind.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Keyset pagination fixes all four. Batches commit, so dead tuples become reclaimable immediately; WAL is generated in digestible chunks; and a cancelled run resumes from the last cursor instead of restarting.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="c1"&gt;-- :last_id starts at 0 and :ceiling_id is snapshotted at startup&lt;/span&gt;
&lt;span class="k"&gt;WITH&lt;/span&gt; &lt;span class="n"&gt;batch&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;MATERIALIZED&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt;
    &lt;span class="k"&gt;FROM&lt;/span&gt;   &lt;span class="n"&gt;orders&lt;/span&gt;
    &lt;span class="k"&gt;WHERE&lt;/span&gt;  &lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;last_id&lt;/span&gt;
      &lt;span class="k"&gt;AND&lt;/span&gt;  &lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;ceiling_id&lt;/span&gt;
    &lt;span class="k"&gt;ORDER&lt;/span&gt;  &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt;
    &lt;span class="k"&gt;LIMIT&lt;/span&gt;  &lt;span class="mi"&gt;5000&lt;/span&gt;
&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;upd&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="k"&gt;UPDATE&lt;/span&gt; &lt;span class="n"&gt;orders&lt;/span&gt; &lt;span class="n"&gt;o&lt;/span&gt;
    &lt;span class="k"&gt;SET&lt;/span&gt;    &lt;span class="n"&gt;total_cents_v2&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;o&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;total_cents&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="nb"&gt;bigint&lt;/span&gt;
    &lt;span class="k"&gt;FROM&lt;/span&gt;   &lt;span class="n"&gt;batch&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt;
    &lt;span class="k"&gt;WHERE&lt;/span&gt;  &lt;span class="n"&gt;o&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;b&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;
      &lt;span class="k"&gt;AND&lt;/span&gt;  &lt;span class="n"&gt;o&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;total_cents_v2&lt;/span&gt; &lt;span class="k"&gt;IS&lt;/span&gt; &lt;span class="k"&gt;DISTINCT&lt;/span&gt; &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;o&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;total_cents&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="nb"&gt;bigint&lt;/span&gt;
    &lt;span class="n"&gt;RETURNING&lt;/span&gt; &lt;span class="n"&gt;o&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="k"&gt;max&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;last_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
       &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="k"&gt;count&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;upd&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;rows_updated&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt;   &lt;span class="n"&gt;batch&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three details make this query pattern reliable at scale:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Return&lt;/strong&gt;&lt;code&gt;max(id)&lt;/code&gt;&lt;strong&gt;from the batch CTE, not the update.&lt;/strong&gt; If every row in a window has already been backfilled, the update returns zero rows. Deriving your next cursor from the &lt;code&gt;UPDATE&lt;/code&gt; result would cause the cursor to stall on that range forever, whereas deriving it from the &lt;code&gt;batch&lt;/code&gt; CTE guarantees steady forward progress.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Snapshot&lt;/strong&gt;&lt;code&gt;ceiling_id&lt;/code&gt;&lt;strong&gt;at startup and force&lt;/strong&gt;&lt;code&gt;AS MATERIALIZED&lt;/code&gt;&lt;strong&gt;.&lt;/strong&gt; Querying against a fixed upper bound prevents newly inserted rows from extending the scan indefinitely while your backfill runs. Marking the batch CTE &lt;code&gt;AS MATERIALIZED&lt;/code&gt; prevents PostgreSQL from inlining the subquery and re-evaluating the limit during the update stage.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Avoid&lt;/strong&gt;&lt;code&gt;ctid&lt;/code&gt;&lt;strong&gt;pagination or&lt;/strong&gt;&lt;code&gt;FOR UPDATE SKIP LOCKED&lt;/code&gt;&lt;strong&gt;.&lt;/strong&gt; Physical row pointers (&lt;code&gt;ctid&lt;/code&gt;) change as tuples are updated and vacuumed, creating skippable or duplicate passes. &lt;code&gt;SKIP LOCKED&lt;/code&gt; bypasses rows held by concurrent transactions, leaving subtle gaps in your backfilled data. Keyset pagination on an indexed primary key remains the only deterministic method.&lt;/p&gt;

&lt;p&gt;Two tuning knobs make a measurable difference. First, do not index the new column until the backfill finishes. An unindexed target column lets PostgreSQL use heap-only tuple updates, keeping the new row version on the same page and avoiding index churn, provided the page has free space. Second, lower the autovacuum threshold for this table while the backfill runs:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;orders&lt;/span&gt; &lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="n"&gt;autovacuum_vacuum_scale_factor&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;01&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;autovacuum_vacuum_cost_delay&lt;/span&gt;   &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  &lt;strong&gt;Picking a batch size&lt;/strong&gt;
&lt;/h3&gt;

&lt;p&gt;Batch size is a trade-off between total runtime and the lag you are willing to push onto replicas. These are our bench numbers at a steady 900 writes per second of concurrent production-shaped load.&lt;/p&gt;

&lt;p&gt;Bar chart of Mean batch duration (x100 ms), Peak replica lag (s), Read p99 during run (ms) across 4 categories&lt;/p&gt;

&lt;p&gt;The shape is non-linear: going from 5,000 to 25,000 rows per batch cut total runtime by about a third but multiplied replica lag by six. 5,000 rows is the setting we ship with, and the throttle matters more than the batch size.&lt;/p&gt;

&lt;p&gt;The throttle is a lag check between batches, not a fixed sleep:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;coalesce&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;max&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pg_wal_lsn_diff&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pg_current_wal_lsn&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt; &lt;span class="n"&gt;replay_lsn&lt;/span&gt;&lt;span class="p"&gt;)),&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;lag_bytes&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt;   &lt;span class="n"&gt;pg_stat_replication&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Runner logic: if &lt;code&gt;lag_bytes&lt;/code&gt; exceeds roughly 256MB, sleep until it drops below 64MB, then continue. On the bench this converted a 34-second worst-case lag into a steady 0.6 to 1.3 seconds across an eight-hour run, at the cost of about 50 minutes of extra wall-clock time. Add a second guard for cloud volumes with burst credits, as sustained backfill IOPS on a gp3 volume sized for normal traffic will quietly flatten your read latency long before the database complains.&lt;/p&gt;

&lt;p&gt;Run the backfill from a supervised worker with a persisted cursor, not from a migration framework's single-transaction step. Every migration tool that wraps steps in one transaction will undo eight hours of batched work the moment a batch fails on row 900 million.&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;Index builds:&lt;/strong&gt;&lt;code&gt;CONCURRENTLY&lt;/code&gt;&lt;strong&gt;, and the invalid index you find at 3 a.m.&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;A plain &lt;code&gt;CREATE INDEX&lt;/code&gt; takes a &lt;code&gt;SHARE&lt;/code&gt; lock, which blocks every write for the duration of the build (54 minutes for a single-column btree on our bench table). &lt;code&gt;CREATE INDEX CONCURRENTLY&lt;/code&gt; takes &lt;code&gt;SHARE UPDATE EXCLUSIVE&lt;/code&gt; instead, so reads and writes continue. The cost is two table scans instead of one and a longer total runtime (81 minutes on the same index), plus a set of rules that are easy to get wrong:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="c1"&gt;-- session-level SET: CIC cannot run inside a transaction block&lt;/span&gt;
&lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;lock_timeout&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'50ms'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="n"&gt;statement_timeout&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;INDEX&lt;/span&gt; &lt;span class="n"&gt;CONCURRENTLY&lt;/span&gt; &lt;span class="n"&gt;idx_orders_total_cents_v2&lt;/span&gt;
    &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;orders&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;total_cents_v2&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;It cannot run inside a transaction block.&lt;/strong&gt; Not in &lt;code&gt;BEGIN ... COMMIT&lt;/code&gt;, not in a &lt;code&gt;DO&lt;/code&gt; block, and not inside any migration tool that wraps steps in a transaction by default. Use framework-specific escape hatches such as &lt;code&gt;disable_ddl_transaction!&lt;/code&gt; in Rails, &lt;code&gt;atomic = False&lt;/code&gt; in Django, &lt;code&gt;runInTransaction="false"&lt;/code&gt; in Liquibase, &lt;code&gt;autocommit=True&lt;/code&gt; in Alembic, or non-transactional script execution settings in Flyway.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;It waits for existing transactions to finish.&lt;/strong&gt; A long-running transaction anywhere (including &lt;code&gt;idle in transaction&lt;/code&gt; sessions and transactions touching unrelated tables) will stall the build at a phase boundary. Clear long transactions first.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Failures leave an invalid index behind.&lt;/strong&gt;&lt;a href="https://www.postgresql.org/docs/current/sql-createindex.html" rel="noopener noreferrer"&gt;The CREATE INDEX documentation&lt;/a&gt; spells this out: if a concurrent build fails or is canceled (such as from a lock timeout or canceled query), it leaves behind an invalid index marked in &lt;code&gt;pg_index&lt;/code&gt; that is ignored by the query planner but continuously updated during writes, incurring write amplification without providing read benefits.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Finding them is a one-liner, and it belongs in your monitoring rather than your memory:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;relname&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="k"&gt;table_name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
       &lt;span class="k"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;relname&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;index_name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
       &lt;span class="n"&gt;pg_size_pretty&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pg_relation_size&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;indexrelid&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="k"&gt;size&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt;   &lt;span class="n"&gt;pg_index&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;
&lt;span class="k"&gt;JOIN&lt;/span&gt;   &lt;span class="n"&gt;pg_class&lt;/span&gt; &lt;span class="k"&gt;c&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="k"&gt;c&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;oid&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;indexrelid&lt;/span&gt;
&lt;span class="k"&gt;JOIN&lt;/span&gt;   &lt;span class="n"&gt;pg_class&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;oid&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;indrelid&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt;  &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;indisvalid&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;While a build is running, track its current scan phase and snapshot lockers in real time via &lt;code&gt;pg_stat_progress_create_index&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;pid&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;phase&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;lockers_total&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;lockers_done&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;blocks_total&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;blocks_done&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tuples_total&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tuples_done&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt;   &lt;span class="n"&gt;pg_stat_progress_create_index&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt;  &lt;span class="n"&gt;relid&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'orders'&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="n"&gt;regclass&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Before dropping an invalid index, verify that it is not attached to a &lt;code&gt;UNIQUE&lt;/code&gt; or &lt;code&gt;PRIMARY KEY&lt;/code&gt; constraint (&lt;code&gt;SELECT conname, contype FROM pg_constraint WHERE conindid = 'idx_orders_total_cents_v2'::regclass;&lt;/code&gt;), because &lt;code&gt;DROP INDEX CONCURRENTLY&lt;/code&gt; cannot drop constraint-backed indexes. Once verified, clean up with &lt;code&gt;DROP INDEX CONCURRENTLY idx_orders_total_cents_v2;&lt;/code&gt; (also forbidden inside a transaction block) and rebuild. From PostgreSQL 12 onwards, &lt;code&gt;REINDEX INDEX CONCURRENTLY&lt;/code&gt; handles rebuilding a valid-but-bloated index in place without the drop-and-recreate window.&lt;/p&gt;

&lt;p&gt;Unique constraints need one extra move. Build the unique index concurrently first, then attach it as a constraint, which takes a brief &lt;code&gt;ACCESS EXCLUSIVE&lt;/code&gt; lock but performs no scan:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;UNIQUE&lt;/span&gt; &lt;span class="k"&gt;INDEX&lt;/span&gt; &lt;span class="n"&gt;CONCURRENTLY&lt;/span&gt; &lt;span class="n"&gt;orders_external_ref_uidx&lt;/span&gt;
    &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;orders&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;external_ref&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="k"&gt;BEGIN&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;SET&lt;/span&gt; &lt;span class="k"&gt;LOCAL&lt;/span&gt; &lt;span class="n"&gt;lock_timeout&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'50ms'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;ALTER&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;orders&lt;/span&gt;
  &lt;span class="k"&gt;ADD&lt;/span&gt; &lt;span class="k"&gt;CONSTRAINT&lt;/span&gt; &lt;span class="n"&gt;orders_external_ref_key&lt;/span&gt;
  &lt;span class="k"&gt;UNIQUE&lt;/span&gt; &lt;span class="k"&gt;USING&lt;/span&gt; &lt;span class="k"&gt;INDEX&lt;/span&gt; &lt;span class="n"&gt;orders_external_ref_uidx&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;COMMIT&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;CREATE INDEX CONCURRENTLY&lt;/code&gt; on a primary can extend query conflict windows on hot standby replicas running with &lt;code&gt;hot_standby_feedback&lt;/code&gt; disabled, and an index build holding &lt;code&gt;SHARE UPDATE EXCLUSIVE&lt;/code&gt; blocks autovacuum on that table for its entire duration. On a table that is simultaneously receiving a heavy backfill, that combination is how tables reach 2x bloat. Serialise the backfill and the index build; do not overlap them.&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;Run migrations on 5432, never on PgBouncer's 6432&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;This is the trap that survives every other precaution, because it makes correct SQL behave incorrectly.&lt;/p&gt;

&lt;p&gt;PgBouncer in transaction pooling mode (the default choice for high-connection-count deployments, usually on port 6432) hands a server connection back to the pool at the end of each transaction. Session state does not reliably survive that handoff.&lt;a href="https://www.pgbouncer.org/features.html" rel="noopener noreferrer"&gt;PgBouncer's own feature matrix&lt;/a&gt; lists session-level &lt;code&gt;SET&lt;/code&gt; among the features that are not supported in transaction pooling, and the consequences for migrations are specific:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Session-level timeouts poison other clients.&lt;/strong&gt; Setting &lt;code&gt;lock_timeout&lt;/code&gt; or &lt;code&gt;statement_timeout&lt;/code&gt; at the session level alters the server connection state. When transaction pooling hands that connection to a subsequent application request, the restrictive timeout remains active and silently fails routine queries.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;CREATE INDEX CONCURRENTLY&lt;/code&gt;&lt;strong&gt;loses   its preceding session setup.&lt;/strong&gt; Because &lt;code&gt;SET statement_timeout = 0;&lt;/code&gt; and &lt;code&gt;CREATE INDEX CONCURRENTLY&lt;/code&gt; are sent as separate top-level statements outside a transaction block, transaction pooling can hand you a different backend connection right after the &lt;code&gt;SET&lt;/code&gt; finishes—running the 81-minute index build under the default statement timeout, or killing it mid-flight via PgBouncer's own &lt;code&gt;query_timeout&lt;/code&gt; and leaving an invalid index behind.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Session-scoped advisory locks break.&lt;/strong&gt; Migration scripts often use session-level advisory locks to ensure mutual exclusion during deployments. Under PgBouncer transaction pooling, releasing the connection back to the pool can drop the lock prematurely or leave it orphaned on a connection assigned to another client.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The fix is boring and absolute: give your migration runner its own connection string pointing at the primary's real PostgreSQL port (5432), using a dedicated role with a small connection limit and &lt;code&gt;pg_hba.conf&lt;/code&gt; rules restricted to your CI/CD runner's IP range so you can isolate and audit migration sessions in &lt;code&gt;pg_stat_activity&lt;/code&gt;&amp;nbsp;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# application&lt;/span&gt;
&lt;span class="nv"&gt;DATABASE_URL&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;postgres://app@pgbouncer.internal:6432/app

&lt;span class="c"&gt;# migrations: direct, session-safe&lt;/span&gt;
&lt;span class="nv"&gt;MIGRATION_DATABASE_URL&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;postgres://migrator@db-primary.internal:5432/app?application_name&lt;span class="o"&gt;=&lt;/span&gt;schema_migration
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If you must route through PgBouncer for network reasons, use a separate pool configured in &lt;strong&gt;session&lt;/strong&gt; pooling mode with a very small &lt;code&gt;pool_size&lt;/code&gt;, and accept that you are adding a component that can only hurt. On our bench, migrations run through a transaction-mode pool failed to honour session &lt;code&gt;lock_timeout&lt;/code&gt; in 3 of 10 runs. This does not happen every time, which is worse than always, because it passes in staging.&lt;/p&gt;

&lt;p&gt;Related On xSypher&lt;a href="https://dev.to/article/why-we-moved-from-serverless-postgres-to-self-hosted-docker-pgbouncer-914ms-to-2ms-p95"&gt;Programming*&lt;em&gt;Why We Moved from Serverless Postgres to Self-Hosted Docker + PgBouncer (914ms to 2ms p95)&lt;/em&gt;*&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;The order to run them in&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;For a type change on a 500GB table, here is the full sequence, in deployment order. Prerequisites first: a migration role with a direct 5432 connection, &lt;code&gt;pg_stat_statements&lt;/code&gt; enabled, a monitored replica lag metric, at least 20% free disk, and a verified recent backup.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Deploy 1 (minutes).&lt;/strong&gt; &lt;code&gt;ADD COLUMN&lt;/code&gt; nullable, no default, with &lt;code&gt;lock_timeout&lt;/code&gt; and retries. Install the dual-write trigger in a second transaction.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Verify.&lt;/strong&gt; Confirm new writes populate both columns. Query a handful of rows created in the last minute.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Backfill (hours to days).&lt;/strong&gt; Run the keyset worker with replica-lag throttling and a persisted cursor. Lower &lt;code&gt;autovacuum_vacuum_scale_factor&lt;/code&gt; for the table first; restore it afterwards.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Verify cursor completion.&lt;/strong&gt; Confirm your backfill worker reached &lt;code&gt;:ceiling_id&lt;/code&gt; with zero remaining rows in its final batch (avoid running an unindexed &lt;code&gt;SELECT count(*) WHERE total_cents_v2 IS NULL&lt;/code&gt; on the 512GB primary; let Step 6's &lt;code&gt;VALIDATE CONSTRAINT&lt;/code&gt; perform the single sequential verification scan).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Index (hours).&lt;/strong&gt; &lt;code&gt;CREATE INDEX CONCURRENTLY&lt;/code&gt; on the new column, after the backfill, not before. Check &lt;code&gt;pg_index.indisvalid&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Constrain (hours).&lt;/strong&gt; &lt;code&gt;ADD CONSTRAINT ... CHECK (total_cents_v2 IS NOT NULL) NOT VALID&lt;/code&gt; (or add a parity check &lt;code&gt;CHECK (total_cents_v2 = total_cents::bigint) NOT VALID&lt;/code&gt;), run &lt;code&gt;VALIDATE CONSTRAINT&lt;/code&gt; under &lt;code&gt;SHARE UPDATE EXCLUSIVE&lt;/code&gt;, then in separate statements run &lt;code&gt;SET NOT NULL&lt;/code&gt; and &lt;code&gt;DROP CONSTRAINT&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Deploy 2 (minutes).&lt;/strong&gt; Application reads the new column. Dual write stays on.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Soak (one full business cycle).&lt;/strong&gt; Watch &lt;code&gt;pg_stat_statements&lt;/code&gt; for any remaining reference to the legacy column.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Deploy 3 (minutes).&lt;/strong&gt; Drop the trigger and function, then &lt;code&gt;DROP COLUMN&lt;/code&gt;, each with &lt;code&gt;lock_timeout&lt;/code&gt; and retries.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Reclaim (optional).&lt;/strong&gt; &lt;code&gt;pg_repack&lt;/code&gt; if you need the disk space back before natural churn reclaims it.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Steps 1, 5, 6, 7 and 9 are independently revertible. Step 3 is resumable. There is no point in the sequence where a failure requires a restore, which is the actual definition of a safe migration.&lt;/p&gt;




&lt;h2&gt;
  
  
  &lt;strong&gt;Questions that come up in the migration review&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Does lock_timeout make my migration less likely to succeed?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;It makes each attempt less likely to succeed and the overall run far more likely to finish without an incident. On a table with a steady read load, our bench acquired &lt;code&gt;ACCESS EXCLUSIVE&lt;/code&gt; within 50ms on roughly 7 of 10 attempts. With exponential backoff across 40 attempts, the migration lands within a few minutes. The failure case is a retry loop that exits cleanly, not a stalled production database.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Can I skip the trigger and dual-write only from the application?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;You can, and it works well when the transformation needs logic the database lacks. The risk is coverage: every writer must be updated, including cron jobs, admin tools, data-import scripts and anything connecting with a shared credential. A &lt;code&gt;BEFORE&lt;/code&gt; row trigger costs a fraction of a millisecond per write and cannot be bypassed. Use the trigger unless you have a specific reason not to.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How do managed services change this?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Amazon RDS, Aurora, Cloud SQL and Azure Database all run standard PostgreSQL, so lock semantics are identical. What differs is the IOPS ceiling and the credit model underneath your volume, which decides how fast a backfill can safely run. Managed failover also adds a constraint: a mid-migration failover kills in-flight &lt;code&gt;CONCURRENTLY&lt;/code&gt; builds, so keep a resumable cursor and an invalid-index check.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Do tools like pgroll or pg-osc remove the need for this?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;They automate the pattern rather than replace it. pgroll uses versioned views to present old and new schemas simultaneously; pg-osc uses a shadow table and triggers. Both still require batched backfills, lock timeouts and a safe connection path, and both add their own operational behaviour you need to understand before a production run. Learn the manual sequence first.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What about partitioned tables?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Partitioning is the structural answer to this whole class of problem. DDL applies per partition, so locks are scoped to a few gigabytes rather than 500. Getting there on an existing large table is itself an expand-and-contract migration, usually with &lt;code&gt;ATTACH PARTITION&lt;/code&gt;. It is worth planning once a single table passes a few hundred gigabytes, because every future migration gets cheaper.&lt;/p&gt;

&lt;h2&gt;
  
  
  &lt;strong&gt;What a clean run looks like&lt;/strong&gt;
&lt;/h2&gt;

&lt;p&gt;When this works, the only evidence in your monitoring is a slightly raised read p99 for the duration of the backfill (single digits of milliseconds) and a replica lag line that oscillates between 0.5 and 1.5 seconds instead of spiking. No queued sessions in &lt;code&gt;pg_stat_activity&lt;/code&gt;, no connection pool alarms, no restarted pods, no &lt;code&gt;column does not exist&lt;/code&gt; errors in the application log.&lt;/p&gt;

&lt;p&gt;If an attempt fails instead, the failure is quiet and local: the retry loop exits after its last attempt, the schema is unchanged, and the blocking PID is sitting in your diagnostic query output waiting to be explained. Cancel the blocker if it is safe to cancel, move the window if it is not, and run it again. A migration you can abandon at any point without consequence is the only kind worth running against a 500GB table.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Source: &lt;a href="https://xsypher.com/article/zero-downtime-postgresql-schema-migrations-on-500gb-tables" rel="noopener noreferrer"&gt;Zero-Downtime PostgreSQL Schema Migrations on 500GB+ Tables&lt;/a&gt; — via xSypher&lt;/em&gt;&lt;/p&gt;

</description>
      <category>postgres</category>
      <category>database</category>
      <category>devops</category>
      <category>backend</category>
    </item>
    <item>
      <title>Stopping AI Agent Shell Breakouts in 2.1ms with eBPF CO-RE and Inline BPF LSM</title>
      <dc:creator>Anwar Iqbal Dawar</dc:creator>
      <pubDate>Sat, 31 Oct 2026 18:12:40 +0000</pubDate>
      <link>https://dev.to/anwariqbaldawar/stopping-ai-agent-shell-breakouts-in-21ms-with-ebpf-co-re-and-inline-bpf-lsm-5e05</link>
      <guid>https://dev.to/anwariqbaldawar/stopping-ai-agent-shell-breakouts-in-21ms-with-ebpf-co-re-and-inline-bpf-lsm-5e05</guid>
      <description>&lt;p&gt;If an MCP server can spawn &lt;code&gt;/bin/sh&lt;/code&gt;, a prompt injection is basically a remote shell on your host with extra steps.&lt;/p&gt;

&lt;p&gt;While researching AI agent runtime security for &lt;a href="https://xsypher.com" rel="noopener noreferrer"&gt;xSypher&lt;/a&gt;, I kept running into the same architectural wall. Static container profiles like AppArmor or seccomp break almost immediately on autonomous AI agent workloads because agents are dynamic by design—they legitimately write code, invoke interpreters, and execute subprocesses.&lt;/p&gt;

&lt;p&gt;On the flip side, standard user-space syscall tracers usually alert you 50 to 2,000 ms after the syscall fires. By the time an alert lands on &lt;code&gt;mcp-server -&amp;gt; python3 -&amp;gt; sh -&amp;gt; curl&lt;/code&gt;, the AWS or GCP cloud metadata (IMDS) token at &lt;code&gt;169.254.169.254&lt;/code&gt; has already left the host.&lt;/p&gt;

&lt;p&gt;The signal is not the individual syscall. It is the process ancestry behind it and the network destination in front of it—and containment has to happen synchronously before the socket opens.&lt;/p&gt;

&lt;p&gt;To solve this at the Linux syscall boundary without running another bloated, monolithic root daemon, I built and open-sourced &lt;strong&gt;xSypher-Stream (v2.0.0)&lt;/strong&gt;.&lt;/p&gt;


&lt;div class="ltag-github-readme-tag"&gt;
  &lt;div class="readme-overview"&gt;
    &lt;h2&gt;
      &lt;img src="https://assets.dev.to/assets/github-logo-5a155e1f9a670af7944dd5e12375bc76ed542ea80224905ecaf878b9157cdefc.svg" alt="GitHub logo"&gt;
      &lt;a href="https://github.com/anwariqbaldawar" rel="noopener noreferrer"&gt;
        anwariqbaldawar
      &lt;/a&gt; / &lt;a href="https://github.com/anwariqbaldawar/xSypher-Stream-Zero-Trust-Kernel-log-Telemetry" rel="noopener noreferrer"&gt;
        xSypher-Stream-Zero-Trust-Kernel-log-Telemetry
      &lt;/a&gt;
    &lt;/h2&gt;
    &lt;h3&gt;
      Kernel-verified runtime defense for AI agent &amp;amp; MCP hosts using eBPF CO-RE telemetry and inline BPF LSM enforcement.
    &lt;/h3&gt;
  &lt;/div&gt;
  &lt;div class="ltag-github-body"&gt;
    
&lt;div id="readme" class="md"&gt;&lt;p&gt;
  &lt;a rel="noopener noreferrer" href="https://github.com/anwariqbaldawar/xSypher-Stream-Zero-Trust-Kernel-log-Telemetry/assets/logo-hero.svg"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fraw.githubusercontent.com%2Fanwariqbaldawar%2FxSypher-Stream-Zero-Trust-Kernel-log-Telemetry%2FHEAD%2Fassets%2Flogo-hero.svg" alt="xSypher-Stream wordmark: emblem and project name, black ink on light themes and white ink on dark themes, with a crimson accent and glow" width="880"&gt;&lt;/a&gt;
&lt;/p&gt;

&lt;div class="markdown-heading"&gt;
&lt;h1 class="heading-element"&gt;xSypher-Stream&lt;/h1&gt;
&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Kernel-verified runtime defence for AI agent hosts.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;
  eBPF CO-RE telemetry, a policy decision in user space, an inline LSM denial in the kernel.&lt;br&gt;
  An injected tool call cannot reach the network if the kernel refuses the syscall first
&lt;/p&gt;

&lt;p&gt;
  &lt;a href="https://github.com/anwariqbaldawar/xSypher-Stream-Zero-Trust-Kernel-log-Telemetry/actions/workflows/ci.yml" rel="noopener noreferrer"&gt;&lt;img src="https://github.com/anwariqbaldawar/xSypher-Stream-Zero-Trust-Kernel-log-Telemetry/actions/workflows/ci.yml/badge.svg" alt="CI status"&gt;&lt;/a&gt;
  &lt;a rel="noopener noreferrer nofollow" href="https://camo.githubusercontent.com/e05ccc0c3002f455fa5315e96da1cbbeb3cfac5574e4b120f9af52619deb1ff3/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6e706d2d322e302e302d636233383337"&gt;&lt;img src="https://camo.githubusercontent.com/e05ccc0c3002f455fa5315e96da1cbbeb3cfac5574e4b120f9af52619deb1ff3/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6e706d2d322e302e302d636233383337" alt="npm package version 2.0.0"&gt;&lt;/a&gt;
  &lt;a rel="noopener noreferrer nofollow" href="https://camo.githubusercontent.com/c9f5a0f335ca30ea7eba860fb50b1b0ac63a264ce54aa9e20ee25721f88f7a21/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d4170616368652d2d322e3025323025324625323047504c2d2d322e302d343735353639"&gt;&lt;img src="https://camo.githubusercontent.com/c9f5a0f335ca30ea7eba860fb50b1b0ac63a264ce54aa9e20ee25721f88f7a21/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6c6963656e73652d4170616368652d2d322e3025323025324625323047504c2d2d322e302d343735353639" alt="Licensed Apache-2.0, kernel programs GPL-2.0"&gt;&lt;/a&gt;
  &lt;a rel="noopener noreferrer nofollow" href="https://camo.githubusercontent.com/3b64dd832ca61b2f5a58c4ff3fcaefc84b66347cfba9425ff55a8d9a8a45b4aa/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6b65726e656c2d352e31302532422d333334313535"&gt;&lt;img src="https://camo.githubusercontent.com/3b64dd832ca61b2f5a58c4ff3fcaefc84b66347cfba9425ff55a8d9a8a45b4aa/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6b65726e656c2d352e31302532422d333334313535" alt="Kernel 5.10 or newer"&gt;&lt;/a&gt;
  &lt;a rel="noopener noreferrer nofollow" href="https://camo.githubusercontent.com/03edee8a03b7f34e3495da4e2497a27d9324d78f170080e9011c2ce4ffb170a3/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f707974686f6e2d332e31302532422d333334313535"&gt;&lt;img src="https://camo.githubusercontent.com/03edee8a03b7f34e3495da4e2497a27d9324d78f170080e9011c2ce4ffb170a3/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f707974686f6e2d332e31302532422d333334313535" alt="Python 3.10 or newer"&gt;&lt;/a&gt;
  &lt;a rel="noopener noreferrer nofollow" href="https://camo.githubusercontent.com/01e75f9db55aa8b06f44128750981640872e7287aa5d96dd681b1ff6de7a4632/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6e6f64652d31382532422d333334313535"&gt;&lt;img src="https://camo.githubusercontent.com/01e75f9db55aa8b06f44128750981640872e7287aa5d96dd681b1ff6de7a4632/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f6e6f64652d31382532422d333334313535" alt="Node 18 or newer"&gt;&lt;/a&gt;
  &lt;a rel="noopener noreferrer nofollow" href="https://camo.githubusercontent.com/610eb07389735e6fdacbc0b248ebcc074984a0720fb528016ce29a81dd4f5501/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f74657374732d363325323070617373696e672d313538303364"&gt;&lt;img src="https://camo.githubusercontent.com/610eb07389735e6fdacbc0b248ebcc074984a0720fb528016ce29a81dd4f5501/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f74657374732d363325323070617373696e672d313538303364" alt="63 tests passing"&gt;&lt;/a&gt;
  &lt;a rel="noopener noreferrer nofollow" href="https://camo.githubusercontent.com/cd6fd917eb1a68cdec6b1f32e9d69ef2d34ab07f7e5b4ff65062b420582642ff/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f636f7665726167652d37312532352d306561356539"&gt;&lt;img src="https://camo.githubusercontent.com/cd6fd917eb1a68cdec6b1f32e9d69ef2d34ab07f7e5b4ff65062b420582642ff/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f636f7665726167652d37312532352d306561356539" alt="71 percent line coverage of the engine by the pytest suite"&gt;&lt;/a&gt;
&lt;/p&gt;

&lt;p&gt;
  &lt;a href="https://github.com/anwariqbaldawar/xSypher-Stream-Zero-Trust-Kernel-log-Telemetry#ten-second-quickstart" rel="noopener noreferrer"&gt;Quick start&lt;/a&gt; ·
  &lt;a href="https://github.com/anwariqbaldawar/xSypher-Stream-Zero-Trust-Kernel-log-Telemetry#the-problem" rel="noopener noreferrer"&gt;The problem&lt;/a&gt; ·
  &lt;a href="https://github.com/anwariqbaldawar/xSypher-Stream-Zero-Trust-Kernel-log-Telemetry#why-classic-security-tools-fail-on-ai-agent-hosts" rel="noopener noreferrer"&gt;Comparison&lt;/a&gt; ·
  &lt;a href="https://github.com/anwariqbaldawar/xSypher-Stream-Zero-Trust-Kernel-log-Telemetry#how-it-works" rel="noopener noreferrer"&gt;Architecture&lt;/a&gt; ·
  &lt;a href="https://github.com/anwariqbaldawar/xSypher-Stream-Zero-Trust-Kernel-log-Telemetry#mitre-attck-coverage" rel="noopener noreferrer"&gt;Detections&lt;/a&gt; ·
  &lt;a href="https://github.com/anwariqbaldawar/xSypher-Stream-Zero-Trust-Kernel-log-Telemetry#enterprise-compliance-regulatory-mapping" rel="noopener noreferrer"&gt;Compliance&lt;/a&gt; ·
  &lt;a href="https://github.com/anwariqbaldawar/xSypher-Stream-Zero-Trust-Kernel-log-Telemetry#benchmarks" rel="noopener noreferrer"&gt;Benchmarks&lt;/a&gt; ·
  &lt;a href="https://github.com/anwariqbaldawar/xSypher-Stream-Zero-Trust-Kernel-log-Telemetry#production-deployment" rel="noopener noreferrer"&gt;Deployment&lt;/a&gt; ·
  &lt;a href="https://github.com/anwariqbaldawar/xSypher-Stream-Zero-Trust-Kernel-log-Telemetry#resource-budgets-and-validation-status" rel="noopener noreferrer"&gt;Validation&lt;/a&gt; ·
  &lt;a href="https://github.com/anwariqbaldawar/xSypher-Stream-Zero-Trust-Kernel-log-Telemetry/docs/" rel="noopener noreferrer"&gt;Docs&lt;/a&gt; ·
  &lt;a href="https://github.com/anwariqbaldawar/xSypher-Stream-Zero-Trust-Kernel-log-Telemetry/CONTRIBUTING.md" rel="noopener noreferrer"&gt;Contributing&lt;/a&gt;
&lt;/p&gt;




&lt;div class="markdown-heading"&gt;
&lt;h2 class="heading-element"&gt;Ten-second quickstart&lt;/h2&gt;
&lt;/div&gt;

&lt;p&gt;From a source checkout (Python 3.10+):&lt;/p&gt;

&lt;div class="highlight highlight-source-shell notranslate position-relative overflow-auto js-code-highlight"&gt;
&lt;pre&gt;python3 -m venv .venv &lt;span class="pl-k"&gt;&amp;amp;&amp;amp;&lt;/span&gt; .venv/bin/pip install -r requirements.txt &lt;span class="pl-k"&gt;&amp;amp;&amp;amp;&lt;/span&gt; PYTHONPATH=src .venv/bin/python tests/simulate_mcp_attack.py --events 10000&lt;/pre&gt;

&lt;/div&gt;

&lt;p&gt;This runs real user-space socket transport and simulated kernel events, not live kernel enforcement.
For the native collector, install the prerequisites below, then run
&lt;code&gt;cmake -S . -B build &amp;amp;&amp;amp; cmake --build build -j2&lt;/code&gt;.
The npm bootstrap package is unpublished; registry installation is not a supported quickstart.&lt;/p&gt;
&lt;div class="markdown-heading"&gt;
&lt;h2 class="heading-element"&gt;The problem&lt;/h2&gt;

&lt;/div&gt;
&lt;p&gt;An MCP server publishes tools to a language model. The model calls those tools with…&lt;/p&gt;&lt;/div&gt;
  &lt;/div&gt;
  &lt;div class="gh-btn-container"&gt;&lt;a class="gh-btn" href="https://github.com/anwariqbaldawar/xSypher-Stream-Zero-Trust-Kernel-log-Telemetry" rel="noopener noreferrer"&gt;View on GitHub&lt;/a&gt;&lt;/div&gt;
&lt;/div&gt;





&lt;h2&gt;
  
  
  Why Traditional Container Security Misses AI Agent Breakouts
&lt;/h2&gt;

&lt;p&gt;When an LLM agent or Model Context Protocol (MCP) server processes untrusted external input, every individual step of an exploit chain looks normal to traditional controls:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Subprocess execution is expected:&lt;/strong&gt; Developer and ops agents legitimately spawn &lt;code&gt;git&lt;/code&gt;, &lt;code&gt;bash&lt;/code&gt;, &lt;code&gt;python3&lt;/code&gt;, or &lt;code&gt;node&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Living-off-the-land binaries exist in the image:&lt;/strong&gt; Standard base images carry &lt;code&gt;/bin/sh&lt;/code&gt;, &lt;code&gt;curl&lt;/code&gt;, and standard file utilities.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Outbound egress is open:&lt;/strong&gt; Agents need external network access to reach model APIs, package registries, and documentation endpoints.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;If you rely on asynchronous user-space detection (ring buffer -&amp;gt; user-space parser -&amp;gt; rule engine -&amp;gt; &lt;code&gt;kill -9&lt;/code&gt;), you are racing the kernel scheduler. A hijacked &lt;code&gt;curl http://169.254.169&lt;/code&gt; request completes a local link-local TCP handshake in microseconds. To stop credential exfiltration, enforcement must live inside the kernel LSM hook itself.&lt;/p&gt;




&lt;h2&gt;
  
  
  Architecture: Splitting Kernel Privilege from Policy Evaluation
&lt;/h2&gt;

&lt;p&gt;Most runtime security tools run their entire rule parsing and telemetry pipeline inside a single privileged process. If that parser has a memory safety bug or logic flaw, the attacker compromises a root daemon.&lt;/p&gt;

&lt;p&gt;In &lt;strong&gt;xSypher-Stream&lt;/strong&gt;, I separated kernel collection from threat evaluation across a strict binary wire contract (&lt;code&gt;ABI v3&lt;/code&gt;, magic &lt;code&gt;0x58535950&lt;/code&gt;):&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5l0dfykzp62l0woee3vr.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5l0dfykzp62l0woee3vr.png" alt="xSypher-Stream Kernel-to-Userspace Wire ABI v3 and Privilege Boundary" width="800" height="447"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  1. Kernel Telemetry Plane (5 eBPF CO-RE Tracepoints)
&lt;/h3&gt;

&lt;p&gt;Five eBPF CO-RE programs attach to kernel tracepoints and emit fixed 656-byte &lt;code&gt;kernel_event_t&lt;/code&gt; structs into a 16 MiB BPF ring buffer:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;tp/syscalls/sys_enter_execve&lt;/code&gt; — Captures binary path, arguments, UID/GID, and cgroup ID.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;tp/syscalls/sys_enter_connect&lt;/code&gt; — Captures outbound IPv4/IPv6 destination addresses and ports.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;tp/syscalls/sys_enter_memfd_create&lt;/code&gt; — Flags anonymous in-memory file creation used in fileless ELF execution.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;tp/sched/sched_process_fork&lt;/code&gt; — Tracks parent-to-child PID creation in real time.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;tp/sched/sched_process_exit&lt;/code&gt; — Cleans up exited PIDs from the lineage state.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  2. Inline Enforcement Plane (5 BPF LSM Hooks)
&lt;/h3&gt;

&lt;p&gt;Five BPF LSM programs attach directly to security decision points in the kernel and check pinned BPF maps (&lt;code&gt;quarantined_pids&lt;/code&gt;, &lt;code&gt;quarantined_cgroups&lt;/code&gt;, &lt;code&gt;blocked_ipv4&lt;/code&gt;):&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;lsm/bprm_check_security&lt;/code&gt; — Denies binary execution (&lt;code&gt;-EPERM&lt;/code&gt;) for quarantined PIDs or cgroups before &lt;code&gt;execve&lt;/code&gt; completes.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;lsm/socket_connect&lt;/code&gt; — Blocks outbound socket connections (&lt;code&gt;-EACCES&lt;/code&gt;) to blacklisted IPs (such as Cloud IMDS &lt;code&gt;169.254.169.254&lt;/code&gt;) or from quarantined processes.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;lsm/file_open&lt;/code&gt; — Blocks unauthorized reads to sensitive host paths (&lt;code&gt;/etc/shadow&lt;/code&gt;, &lt;code&gt;/proc/$pid/environ&lt;/code&gt;, SSH keys).&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;lsm/task_kill&lt;/code&gt; — Prevents quarantined processes from signaling or terminating security monitors.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;lsm/bpf&lt;/code&gt; — Tamper-protection hook that denies unauthorized processes from modifying or unloading pinned xSypher BPF maps and programs.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  3. Capability-Scoped C++20 Collector (&lt;code&gt;xsypher-collector&lt;/code&gt;)
&lt;/h3&gt;

&lt;p&gt;The C++20 collector is the only privileged user-space binary, scoped strictly to &lt;code&gt;CAP_BPF&lt;/code&gt; and &lt;code&gt;CAP_PERFMON&lt;/code&gt;.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;It polls the 16 MiB kernel ring buffer via &lt;code&gt;epoll&lt;/code&gt; (&lt;code&gt;ring_buffer__poll()&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;It wraps each 656-byte &lt;code&gt;kernel_event_t&lt;/code&gt; inside an 80-byte transport header to form a fixed 736-byte &lt;code&gt;wire_event_msg_t&lt;/code&gt; frame.&lt;/li&gt;
&lt;li&gt;It streams frames over an &lt;code&gt;AF_UNIX&lt;/code&gt; (&lt;code&gt;SOCK_SEQPACKET&lt;/code&gt;) local IPC socket and listens for validated 32-byte &lt;code&gt;wire_policy_cmd_t&lt;/code&gt; commands to update the kernel BPF maps via &lt;code&gt;bpf_map_update_elem()&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  4. Zero-Capability Python Threat Engine (&lt;code&gt;xsypher-engine&lt;/code&gt;)
&lt;/h3&gt;

&lt;p&gt;The Python 3.10+ &lt;code&gt;asyncio&lt;/code&gt; engine runs with &lt;strong&gt;zero Linux capabilities&lt;/strong&gt; (&lt;code&gt;NoNewPrivileges=true&lt;/code&gt;).&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;O(depth) Lineage Graph:&lt;/strong&gt; Maintains a 100,000-PID LRU process ancestry graph in memory, resolving up to 8 ancestor hops (&lt;code&gt;mcp-server -&amp;gt; python3 -&amp;gt; sh -&amp;gt; curl&lt;/code&gt;) on every incoming event.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;6 Shipped MITRE ATT&amp;amp;CK Rules:&lt;/strong&gt; Evaluates lineage and syscall parameters against YAML rules covering MCP shell breakouts (&lt;code&gt;T1059.004&lt;/code&gt;), Cloud IMDS credential theft (&lt;code&gt;T1552.005&lt;/code&gt;), &lt;code&gt;/etc/shadow&lt;/code&gt; and environment harvesting (&lt;code&gt;T1003.008&lt;/code&gt;), fileless &lt;code&gt;memfd&lt;/code&gt; execution (&lt;code&gt;T1620&lt;/code&gt;), recon tools (&lt;code&gt;T1105&lt;/code&gt;), and suspicious C2 egress ports (&lt;code&gt;T1071&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Tamper-Evident Audit Ledger:&lt;/strong&gt; Every alert is written to an append-only JSONL log where each record includes a forward-chained SHA-256 hash (&lt;code&gt;prev_hash + current_payload&lt;/code&gt;) verifiable via &lt;code&gt;xsypher verify-log&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  The Fixed Binary Wire Contract (ABI v3)
&lt;/h2&gt;

&lt;p&gt;To eliminate serialization overhead (no JSON or Protobuf parsing on the hot path) and prevent parser bugs across the C++/Python boundary, both sides enforce a compile-time verified binary layout defined in &lt;code&gt;include/xsypher_abi.h&lt;/code&gt;:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;kernel_event_t&lt;/code&gt; — &lt;strong&gt;656 Bytes&lt;/strong&gt; (Kernel BPF Ring Buffer -&amp;gt; C++20 Collector)&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;wire_event_msg_t&lt;/code&gt; — &lt;strong&gt;736 Bytes&lt;/strong&gt; (80-Byte Transport Header + 656-Byte Event, Collector -&amp;gt; Python Engine)&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;wire_policy_cmd_t&lt;/code&gt; — &lt;strong&gt;32 Bytes&lt;/strong&gt; (Quarantine / Block Command, Python Engine -&amp;gt; Collector -&amp;gt; BPF Maps)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Running &lt;code&gt;make verify&lt;/code&gt; compiles a C offset inspector against &lt;code&gt;include/xsypher_abi.h&lt;/code&gt; and asserts byte-for-byte parity with the Python &lt;code&gt;ctypes.Structure&lt;/code&gt; definitions before any test suite runs.&lt;/p&gt;




&lt;h2&gt;
  
  
  Engineering Honesty: What Works Today vs. Live-Kernel Gotchas
&lt;/h2&gt;

&lt;p&gt;When releasing security tooling, I believe in documenting exact boundaries rather than hiding behind marketing claims:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;What passes in CI today:&lt;/strong&gt; User-space simulation (&lt;code&gt;~2.1 ms&lt;/code&gt; end-to-end containment latency across 5 multi-stage exploit chains), C/Python ABI layout parity (&lt;code&gt;make verify&lt;/code&gt;), and 63 &lt;code&gt;pytest&lt;/code&gt; unit and integration tests pass cleanly.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The BPF LSM Boot Prerequisite:&lt;/strong&gt; Having &lt;code&gt;CONFIG_BPF_LSM=y&lt;/code&gt; compiled into your Linux kernel is necessary, but &lt;strong&gt;not sufficient&lt;/strong&gt; by itself. On stock Ubuntu 22.04/24.04 and most standard cloud images, the &lt;code&gt;bpf&lt;/code&gt; module is omitted from &lt;code&gt;/sys/kernel/security/lsm&lt;/code&gt; at boot. When &lt;code&gt;bpf&lt;/code&gt; is not in the active boot LSM list (&lt;code&gt;lsm=lockdown,capability,landlock,yama,apparmor,bpf&lt;/code&gt;), &lt;code&gt;libbpf&lt;/code&gt; can load and attach LSM programs without returning an error, yet the kernel will never invoke them on the syscall path.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Before enabling &lt;code&gt;--enforce-kernel&lt;/code&gt; in production, always run &lt;code&gt;xsypher doctor&lt;/code&gt; to verify &lt;code&gt;/sys/kernel/security/lsm&lt;/code&gt; and baseline your workload in audit-only mode on a dedicated node.&lt;/p&gt;




&lt;h2&gt;
  
  
  Try It in 10 Seconds (Zero Root Privileges Required)
&lt;/h2&gt;

&lt;p&gt;You do not need a custom kernel, BTF, or root access to test the binary wire protocol, O(depth) lineage graph, and MITRE ATT&amp;amp;CK rule engine. The built-in simulation harness replays 10,000 synthetic syscalls interleaved with 5 multi-stage attack chains over a real &lt;code&gt;AF_UNIX SOCK_SEQPACKET&lt;/code&gt; pair:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;git clone https://github.com
&lt;span class="nb"&gt;cd &lt;/span&gt;xSypher-Stream-Zero-Trust-Kernel-log-Telemetry
pip &lt;span class="nb"&gt;install&lt;/span&gt; &lt;span class="nt"&gt;-e&lt;/span&gt; &lt;span class="nb"&gt;.&lt;/span&gt;
xsypher simulate 10000
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  &lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F1tzofb40tlre06ew7jx8.webp" alt="xSypher-Stream 10,000-Event Simulation and ABI Verification Terminal Output" width="800" height="447"&gt;
&lt;/h2&gt;

&lt;h2&gt;
  
  
  Source Code &amp;amp; Full Architectural Audit
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;GitHub Repository (Apache-2.0 / GPL-2.0):&lt;/strong&gt; &lt;a href="https://github.com" rel="noopener noreferrer"&gt;anwariqbaldawar/xSypher-Stream-Zero-Trust-Kernel-log-Telemetry&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Full Source-Level Architecture &amp;amp; Rollout Audit on xSypher:&lt;/strong&gt; &lt;a href="https://xsypher.com" rel="noopener noreferrer"&gt;xSypher-Stream: Inline eBPF &amp;amp; BPF LSM Kernel Defense for AI Agents&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you are working with eBPF CO-RE, Linux LSMs, or sandboxing autonomous AI agents, I would genuinely appreciate your critique on the collector/engine privilege split and wire ABI design!&lt;/p&gt;

</description>
      <category>security</category>
      <category>opensource</category>
      <category>ai</category>
      <category>devops</category>
    </item>
    <item>
      <title>Architecting Production-Grade Agentic AI Systems: Memory, Orchestration, and Latency Optimization</title>
      <dc:creator>Anwar Iqbal Dawar</dc:creator>
      <pubDate>Wed, 28 Oct 2026 18:39:22 +0000</pubDate>
      <link>https://dev.to/anwariqbaldawar/architecting-production-grade-agentic-ai-systems-memory-orchestration-and-latency-optimization-3l0e</link>
      <guid>https://dev.to/anwariqbaldawar/architecting-production-grade-agentic-ai-systems-memory-orchestration-and-latency-optimization-3l0e</guid>
      <description>&lt;p&gt;Moving Large Language Models (LLMs) from isolated chat interfaces to autonomous, multi-step production pipelines requires a fundamental shift in system architecture. Treating an LLM as a stateless text generator fails at scale; instead, modern engineering treats the model as a reasoning engine surrounded by deterministic guardrails, state machines, and structured memory tiers.&lt;/p&gt;

&lt;p&gt;This breakdown covers the core engineering patterns, data structures, and inference optimizations required to build reliable, low-latency agentic workflows in production.&lt;/p&gt;




&lt;h2&gt;
  
  
  1. Beyond Naive RAG: Multi-Tiered Memory Architectures
&lt;/h2&gt;

&lt;p&gt;Standard Retrieval-Augmented Generation (RAG) relies on chunking static documents and running cosine similarity searches. In multi-step agentic workflows, naive RAG breaks down due to context fragmentation, lost entity relationships, and token bloat. &lt;/p&gt;

&lt;p&gt;Production systems separate state into a &lt;strong&gt;three-tiered memory architecture&lt;/strong&gt;:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Memory Tier&lt;/th&gt;
&lt;th&gt;Storage Engine&lt;/th&gt;
&lt;th&gt;Retrieval Mechanism&lt;/th&gt;
&lt;th&gt;Primary Use Case&lt;/th&gt;
&lt;th&gt;Target Latency&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Working Memory&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;In-Memory / Redis&lt;/td&gt;
&lt;td&gt;Direct Key-Value &amp;amp; Strict JSON Schema&lt;/td&gt;
&lt;td&gt;Active execution state, tool outputs, scratchpad&lt;/td&gt;
&lt;td&gt;` ToolExecutionPayload&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;try:
    validated = ToolExecutionPayload.model_validate_json(raw_llm_json)
    state.error_trace = None
    return validated
except ValidationError as exc:
    state.step_count += 1
    state.error_trace = f"Schema Validation Failed: {exc.errors()}"
    return {"action": "retry_node", "feedback": state.error_trace}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

&lt;p&gt;&lt;code&gt;&lt;/code&gt;`&lt;/p&gt;




&lt;h2&gt;
  
  
  3. Inference Optimization &amp;amp; Latency Engineering
&lt;/h2&gt;

&lt;p&gt;Latency is the primary bottleneck in multi-agent systems. If an orchestration graph makes five sequential LLM calls, a 2-second Time-To-First-Token (TTFT) compounds into a 10+ second blocking delay.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Optimization Technique&lt;/th&gt;
&lt;th&gt;Target Metric&lt;/th&gt;
&lt;th&gt;Typical Improvement&lt;/th&gt;
&lt;th&gt;Implementation Complexity&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Prefix / KV Caching&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Time-To-First-Token (TTFT)&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;50% – 80%&lt;/code&gt; reduction&lt;/td&gt;
&lt;td&gt;Low (Prompt structuring)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Semantic Caching&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;End-to-End Latency &amp;amp; Cost&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;95%+&lt;/code&gt; on cache hits&lt;/td&gt;
&lt;td&gt;Medium (Redis / Vector Gateway)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Speculative Decoding&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Inter-Token Latency (ITL)&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;2x – 3x&lt;/code&gt; throughput&lt;/td&gt;
&lt;td&gt;Medium (vLLM / TensorRT-LLM config)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Async Map-Reduce&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Total Wall-Clock Time&lt;/td&gt;
&lt;td&gt;Scales $O(1)$ across parallel branches&lt;/td&gt;
&lt;td&gt;Low (&lt;code&gt;asyncio.gather&lt;/code&gt; / Worker queues)&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h3&gt;
  
  
  Core Latency Patterns
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;KV Cache Prefix Alignment:&lt;/strong&gt; Structure system prompts so that static instructions, JSON schemas, and tool definitions sit at the exact beginning of the context window. Dynamic state and user queries must be appended at the very end so inference engines (like vLLM or SGLang) can reuse precomputed Key-Value (KV) tensors across requests.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Semantic Gateway Caching:&lt;/strong&gt; Place a high-speed vector layer in front of your inference endpoints. By evaluating incoming query embeddings against historical &lt;a href="https://xsypher.com" rel="noopener noreferrer"&gt;distributed inference telemetry&lt;/a&gt; and cached execution traces (using a strict cosine similarity threshold of &lt;code&gt;&amp;gt; 0.96&lt;/code&gt;), repetitive sub-tasks bypass model invocation entirely and resolve in under 15 milliseconds.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Speculative Decoding:&lt;/strong&gt; Pair a lightweight draft model (e.g., 1B–3B parameters) with your primary frontier or 70B+ model. The draft model speculates $K$ tokens ahead with minimal compute overhead, allowing the target model to verify tokens in a single parallel forward pass.&lt;/li&gt;
&lt;/ol&gt;




&lt;h2&gt;
  
  
  4. Production Evaluation &amp;amp; Observability Pipeline
&lt;/h2&gt;

&lt;p&gt;Standard boolean unit tests fail on non-deterministic state machines. Reliable deployments require a dual-layer verification harness:&lt;/p&gt;

&lt;h3&gt;
  
  
  Layer 1: Deterministic CI/CD Gates (Pre-Deployment)
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Schema Compliance Rate:&lt;/strong&gt; Must hit &lt;code&gt;100%&lt;/code&gt; across synthetic adversarial test suites using constrained decoding (e.g., Outlines or Grammar-based sampling).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Token Budget &amp;amp; Loop Ceilings:&lt;/strong&gt; Verify that cyclic graphs terminate gracefully when &lt;code&gt;max_iterations&lt;/code&gt; is reached without throwing unhandled exceptions.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;PII &amp;amp; Secret Redaction:&lt;/strong&gt; Run deterministic regex and NER (Named Entity Recognition) scrubbers on both ingress prompts and egress tool payloads.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Layer 2: Asynchronous Reference-Free Evaluation (Post-Deployment)
&lt;/h3&gt;

&lt;p&gt;Sample 5% to 10% of production traces asynchronously via OpenTelemetry (OpenInference) and evaluate them using an isolated judge model across three core metrics:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Context Precision:&lt;/strong&gt; Signal-to-noise ratio of the chunks retrieved by the Episodic and Semantic memory layers.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Hallucination / Faithfulness (&lt;code&gt;&amp;gt; 0.95&lt;/code&gt; target):&lt;/strong&gt; Verification that every factual statement in the final synthesis is strictly grounded in the working state.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Tool Selection Accuracy:&lt;/strong&gt; Alignment between the router's selected subgraph and ground-truth human annotations.&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  5. Summary Checklist for System Architects
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Isolate State from Prompts:&lt;/strong&gt; Treat prompts as stateless rendering templates populated by strictly typed state machines.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Route Before Reasoning:&lt;/strong&gt; Use fast, small classifiers or 8B models at the ingress layer; reserve heavy reasoning models strictly for complex synthesis nodes.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Bound Every Loop:&lt;/strong&gt; Every cyclic edge in your agent graph must have an explicit retry counter and a deterministic fallback path.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Instrument Everything:&lt;/strong&gt; Capture token counts, KV cache hit ratios, and per-node latencies in real time to catch context drift before it impacts production SLAs.&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>architecture</category>
      <category>programming</category>
    </item>
    <item>
      <title>Local LLM vs Cloud API: The VRAM Math, Unit Economics, and Routing Architecture</title>
      <dc:creator>Anwar Iqbal Dawar</dc:creator>
      <pubDate>Tue, 27 Oct 2026 12:52:19 +0000</pubDate>
      <link>https://dev.to/anwariqbaldawar/local-llm-vs-cloud-api-the-vram-math-unit-economics-and-routing-architecture-3k7g</link>
      <guid>https://dev.to/anwariqbaldawar/local-llm-vs-cloud-api-the-vram-math-unit-economics-and-routing-architecture-3k7g</guid>
      <description>&lt;p&gt;Almost nobody should deploy exclusively on local silicon or entirely behind a provider's API key. The functional engineering question is determining which segments of your workload belong on internal hardware and which justify managed endpoints.&lt;/p&gt;

&lt;p&gt;Cloud models remain superior at deep multi-step reasoning, massive context processing, and complex tool orchestration. However, modern open-weight releases mean keeping sensitive data offline is no longer a compromise on utility. The architectural decision hinges on three constraints: physical memory limits, token volume break-even points, and regulatory data isolation.&lt;/p&gt;

&lt;h2&gt;
  
  
  Memory Sets the Execution Ceiling
&lt;/h2&gt;

&lt;p&gt;Compute ratings dictate generation speed, but available memory dictates whether a model initializes at all. Weights must reside in memory, and the key-value (KV) cache for your context window sits directly on top.&lt;/p&gt;

&lt;p&gt;At 16-bit precision, weights consume approximately two bytes per parameter. A 70B model requires roughly 140 GB of memory before processing a single token of context. Four-bit quantization (such as Q4_K_M) reduces that requirement by approximately 4x:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Model Scale | FP16 VRAM  | Q8_0 VRAM  | Q4_K_M VRAM
------------|------------|------------|------------
7B          | ~14 GB     | ~7.5 GB    | ~5 GB
13B         | ~26 GB     | ~13.5 GB   | ~9 GB
32B         | ~64 GB     | ~34 GB     | ~22 GB
70B         | ~140 GB    | ~72.5 GB   | ~44 GB
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Mixture-of-Experts (MoE) architectures change the operational dynamic. OpenAI's &lt;code&gt;gpt-oss-120b&lt;/code&gt; features 117B total parameters but activates only 5.1B per token. Post-trained using MXFP4 quantization, these architectures read like large models on disk while generating tokens with the compute overhead of a smaller model.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Economic Break-Even Calculation
&lt;/h2&gt;

&lt;p&gt;Local inference trades operational expense for capital expenditure and maintenance. &lt;/p&gt;

&lt;p&gt;Consider an enterprise workstation equipped with 24–32 GB VRAM priced at $2,500. Amortized across 36 months, base hardware depreciation sits near $69 per month. Factoring in three hours of active daily generation at 400W and an idle draw of 60W at $0.18/kWh, power adds approximately $13 per month. Total baseline cost: &lt;strong&gt;~$83/month&lt;/strong&gt; prior to human maintenance.&lt;/p&gt;

&lt;p&gt;Against hosted cloud pricing (assuming a blended ratio of 75% input tokens and 25% output tokens):&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Cloud Tier        | Blended Rate ($/1M) | Monthly Break-Even Volume
------------------|---------------------|--------------------------
Utility / Fast    | ~$0.85              | ~98 Million tokens
Mid-Tier          | ~$4.50              | ~18 Million tokens
Frontier          | ~$10.00             | ~8 Million tokens
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Replacing lightweight classification APIs with local deployments rarely yields positive ROI. Replacing consistent frontier workloads pays back hardware capital within months.&lt;/p&gt;

&lt;h2&gt;
  
  
  Data Egress and Failure Boundaries
&lt;/h2&gt;

&lt;p&gt;Beyond cost, data isolation follows two discrete topologies:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;[User Payload]
      │
      ▼
[Routing Policy Engine]
      ├── (Sensitive / Structured) ──&amp;gt; [Local On-Premises Runtime]
      │                                        │
      │                                  (Zero Egress)
      │                                        │
      └── (Complex / High-Context) ───&amp;gt; [Provider API via TLS]
                                               │
                                         (Contractual Isolation)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Local execution establishes a physical boundary: zero network egress, zero third-party data retention policies to audit, and zero dependency on external uptime SLAs.&lt;/p&gt;

&lt;p&gt;Hosted APIs offer contractual boundaries. While enterprise agreements frequently specify zero-retention policies and exclude inputs from training pipelines, these safeguards are legal abstractions rather than architectural certainties.&lt;/p&gt;

&lt;h2&gt;
  
  
  Production Trade-Offs
&lt;/h2&gt;

&lt;p&gt;When deploying on-premises runtimes, teams must account for three persistent constraints:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;KV Cache Scaling:&lt;/strong&gt; As context depth grows, the KV cache footprint expands linearly. Long-context documents can exhaust memory reserves even when the base model weights fit comfortably.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Orchestration Tooling:&lt;/strong&gt; Managed frontier providers feature native tool calling, structured JSON output stability, and multimodal handling. Open-weight equivalents often require rigid schema enforcement libraries and localized prompt engineering.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Hardware Idle Costs:&lt;/strong&gt; Cloud infrastructure costs drop to zero when queries cease. Local hardware incurs depreciation regardless of throughput.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The resilient approach routes routine extraction, summarization, and private internal records through quantized local infrastructure, while offloading high-order reasoning and extreme context pipelines to commercial providers.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://xsypher.com/article/local-ai-vs-cloud-ai-which-one-fits-your-work" rel="noopener noreferrer"&gt;xsypher.com&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

</description>
      <category>machinelearning</category>
      <category>architecture</category>
      <category>devops</category>
      <category>cloud</category>
    </item>
    <item>
      <title>Patching PHP2Shell (CVE-2026-109618) Doesn't Stop Session-Poisoning LFI to RCE</title>
      <dc:creator>Anwar Iqbal Dawar</dc:creator>
      <pubDate>Tue, 27 Oct 2026 12:45:20 +0000</pubDate>
      <link>https://dev.to/anwariqbaldawar/patching-php2shell-cve-2026-109618-doesnt-stop-session-poisoning-lfi-to-rce-924</link>
      <guid>https://dev.to/anwariqbaldawar/patching-php2shell-cve-2026-109618-doesnt-stop-session-poisoning-lfi-to-rce-924</guid>
      <description>&lt;p&gt;Updating to PHP 8.2.35, 8.3.36, 8.4.27, or 8.5.12 fixes the engine's out-of-memory corruption and stale worker state, and CISA placed both CVEs on the KEV catalog with an October 29 federal deadline. What stood out to me in the post-patch testing from Abstract Security is that patched hosts still leave &lt;code&gt;session.upload_progress.enabled = 1&lt;/code&gt; on by default, turning any postponed local file inclusion (LFI) bug into an unauthenticated two-request RCE.&lt;/p&gt;

&lt;h2&gt;
  
  
  Vulnerability vs. Post-Patch Exposure Matrix
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Vector / Mechanism&lt;/th&gt;
&lt;th&gt;CVE-2026-109618 (&lt;code&gt;longjmp&lt;/code&gt; OOM)&lt;/th&gt;
&lt;th&gt;CVE-2026-109619 (Stale Worker State)&lt;/th&gt;
&lt;th&gt;Session Upload Progress LFI-to-RCE&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Execution Boundary&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Pre-script (&lt;code&gt;php_request_startup&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;Cross-request (&lt;code&gt;PHP-FPM&lt;/code&gt; / &lt;code&gt;mod_php&lt;/code&gt; worker)&lt;/td&gt;
&lt;td&gt;Request startup multipart parsing (&lt;code&gt;RFC1867&lt;/code&gt;)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Affected Versions&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;PHP 5.4 – 8.5 (unpatched)&lt;/td&gt;
&lt;td&gt;PHP 5.4 – 8.5 (unpatched)&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;All versions&lt;/strong&gt; (including patched 8.2.35–8.5.12)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Fixed by Oct 2026 Patch?&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Yes (&lt;code&gt;8.2.35&lt;/code&gt;, &lt;code&gt;8.3.36&lt;/code&gt;, &lt;code&gt;8.4.27&lt;/code&gt;, &lt;code&gt;8.5.12&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;Yes (&lt;code&gt;8.2.35&lt;/code&gt;, &lt;code&gt;8.3.36&lt;/code&gt;, &lt;code&gt;8.4.27&lt;/code&gt;, &lt;code&gt;8.5.12&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;No&lt;/strong&gt; (enabled by default in &lt;code&gt;php.ini&lt;/code&gt;)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;App-Level &lt;code&gt;ini_set()&lt;/code&gt; Fix?&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Ineffective (triggers before script)&lt;/td&gt;
&lt;td&gt;Ineffective (triggers during shutdown/reuse)&lt;/td&gt;
&lt;td&gt;Ineffective (&lt;code&gt;PHP_INI_PERDIR&lt;/code&gt; stage only)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Primary Mitigation&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Upgrade binary / backport engine patch&lt;/td&gt;
&lt;td&gt;Upgrade binary + reduce &lt;code&gt;pm.max_requests&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Set &lt;code&gt;session.upload_progress.enabled = 0&lt;/code&gt; in &lt;code&gt;php.ini&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;




&lt;h2&gt;
  
  
  1. Pre-Script Execution via &lt;code&gt;longjmp()&lt;/code&gt; Cleanup
&lt;/h2&gt;

&lt;p&gt;In unpatched versions (5.4 through 8.5), CVE-2026-109618 triggers when an out-of-memory error forces a &lt;code&gt;longjmp()&lt;/code&gt; bailout while internal request structures are only partially initialized. During request shutdown, PHP's cleanup routine trusts those uninitialized pointers before a single line of application code runs, while CVE-2026-109619 leaves freed request pointers alive in global state for the next request on the same PHP-FPM or &lt;code&gt;mod_php&lt;/code&gt; worker.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Why application WAFs miss it:&lt;/strong&gt; Because the fault happens inside &lt;code&gt;php_request_startup()&lt;/code&gt; during multipart or query parsing, framework middleware, PHP-level authentication guards, and userland input sanitizers never execute.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fcpw5lzxamfgg59zarz50.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fcpw5lzxamfgg59zarz50.png" alt=" " width="800" height="447"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  2. Why &lt;code&gt;ini_set()&lt;/code&gt; and Upload Progress Tweaks Fail
&lt;/h2&gt;

&lt;p&gt;Setting &lt;code&gt;session.upload_progress.enabled=0&lt;/code&gt; inside application code (&lt;code&gt;ini_set()&lt;/code&gt;) does nothing against the engine CVEs or session creation because multipart parsing happens during the &lt;code&gt;PHP_INI_PERDIR&lt;/code&gt; / &lt;code&gt;SYSTEM&lt;/code&gt; initialization phase before the script starts.&lt;/p&gt;

&lt;p&gt;Even on freshly patched 8.2.35+ servers, an attacker can pass a chosen &lt;code&gt;PHP_SESSION_UPLOAD_PROGRESS&lt;/code&gt; field in a multipart POST request alongside a custom &lt;code&gt;PHPSESSID&lt;/code&gt; cookie to write an unsanitized filename payload directly into &lt;code&gt;/var/lib/php/sessions/sess_&amp;lt;id&amp;gt;&lt;/code&gt; on disk. If your codebase has a dormant &lt;code&gt;include($_GET['page'])&lt;/code&gt; style LFI, racing a concurrent read against that active upload executes arbitrary code even when &lt;code&gt;session.use_strict_mode = 1&lt;/code&gt; is enabled.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Configuration Directive&lt;/th&gt;
&lt;th&gt;Default Value&lt;/th&gt;
&lt;th&gt;Hardened Value&lt;/th&gt;
&lt;th&gt;Config Scope (&lt;code&gt;Changeable&lt;/code&gt;)&lt;/th&gt;
&lt;th&gt;Security Impact&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;session.upload_progress.enabled&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;1&lt;/code&gt; (On)&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;0&lt;/code&gt; (Off)&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;PHP_INI_PERDIR&lt;/code&gt; (&lt;code&gt;php.ini&lt;/code&gt; / &lt;code&gt;.user.ini&lt;/code&gt; / FPM pool)&lt;/td&gt;
&lt;td&gt;Stops unauthenticated creation of &lt;code&gt;/var/lib/php/sessions/sess_*&lt;/code&gt; via multipart POST&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;session.use_strict_mode&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;0&lt;/code&gt; (Off)&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;1&lt;/code&gt; (On)&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;PHP_INI_ALL&lt;/code&gt; (best in &lt;code&gt;php.ini&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;Rejects uninitialized session IDs passed via cookies by normal session handlers&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;max_input_vars&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;1000&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;1000&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;PHP_INI_PERDIR&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Caps hash table collisions and multipart variable exhaustion&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;post_max_size&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;8M&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;8M&lt;/code&gt; (or stricter at proxy)&lt;/td&gt;
&lt;td&gt;&lt;code&gt;PHP_INI_PERDIR&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Limits memory allocation spikes during pre-script body parsing&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  3. Lock Down the Parser Boundary and Verify Builds
&lt;/h2&gt;

&lt;p&gt;First, verify your active PHP-FPM or CLI build date and patch level rather than trusting package manager cache headers:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;php &lt;span class="nt"&gt;-v&lt;/span&gt;
php &lt;span class="nt"&gt;-i&lt;/span&gt; | &lt;span class="nb"&gt;grep&lt;/span&gt; &lt;span class="nt"&gt;-E&lt;/span&gt; &lt;span class="s2"&gt;"session.upload_progress.enabled|session.use_strict_mode|max_input_vars"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Next, disable upload progress at the system &lt;code&gt;php.ini&lt;/code&gt; level so session files cannot be used as an unauthenticated staging ground:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="c"&gt;; /etc/php/8.3/fpm/php.ini - must be set at the INI or pool level, not runtime
&lt;/span&gt;&lt;span class="py"&gt;session.upload_progress.enabled&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;0&lt;/span&gt;
&lt;span class="py"&gt;session.use_strict_mode&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;1&lt;/span&gt;
&lt;span class="py"&gt;max_input_vars&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;1000&lt;/span&gt;
&lt;span class="py"&gt;post_max_size&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;8M&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;Optional: Lock down via PHP-FPM Pool Config &amp;amp; Nginx Body Limits (Click to expand)&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;If you run multi-tenant PHP-FPM pools and want to make sure developers cannot re-enable upload progress via &lt;code&gt;.user.ini&lt;/code&gt; or &lt;code&gt;.htaccess&lt;/code&gt;, enforce &lt;code&gt;php_admin_flag&lt;/code&gt; inside &lt;code&gt;/etc/php/8.3/fpm/pool.d/www.conf&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="na"&gt;php_admin_flag&lt;/span&gt;&lt;span class="nn"&gt;[session.upload_progress.enabled]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="na"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="na"&gt;off&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="na"&gt;php_admin_flag&lt;/span&gt;&lt;span class="nn"&gt;[session.use_strict_mode]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="na"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="na"&gt;on&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="na"&gt;php_admin_value&lt;/span&gt;&lt;span class="nn"&gt;[max_input_vars]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="na"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="na"&gt;1000&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;
&lt;p&gt;At the reverse-proxy boundary (Nginx/Caddy), cap request body sizes before payloads ever reach the FastCGI socket:&lt;/p&gt;
&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight nginx"&gt;&lt;code&gt;&lt;span class="c1"&gt;# nginx.conf / site block&lt;/span&gt;
&lt;span class="k"&gt;client_max_body_size&lt;/span&gt; &lt;span class="mi"&gt;8m&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;client_body_buffer_size&lt;/span&gt; &lt;span class="mi"&gt;128k&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;
&lt;p&gt;For teams triaging exposure across legacy 7.4–8.1 backports and newer runtimes, we map emergency advisories, severity decisions, and response workflows in our &lt;a href="https://xsypher.com" rel="noopener noreferrer"&gt;xSypher product security runbooks&lt;/a&gt; alongside our verification standards.&lt;/p&gt;

&lt;p&gt;Are you backporting the minimal source patches for EOL PHP 7.4/8.1 runtimes in production, or isolating legacy workers behind strict reverse-proxy body limits until you can upgrade?&lt;/p&gt;

</description>
      <category>php</category>
      <category>security</category>
      <category>devops</category>
      <category>webdev</category>
    </item>
    <item>
      <title>How We Cut Postgres p95 Latency from 914ms to 2ms (Serverless vs. Docker + PgBouncer)</title>
      <dc:creator>Anwar Iqbal Dawar</dc:creator>
      <pubDate>Sun, 25 Oct 2026 09:28:25 +0000</pubDate>
      <link>https://dev.to/anwariqbaldawar/how-we-cut-postgres-p95-latency-from-914ms-to-2ms-serverless-vs-docker-pgbouncer-nnm</link>
      <guid>https://dev.to/anwariqbaldawar/how-we-cut-postgres-p95-latency-from-914ms-to-2ms-serverless-vs-docker-pgbouncer-nnm</guid>
      <description>&lt;p&gt;Serverless PostgreSQL is phenomenal for shipping fast. Branching, automatic scaling, and zero maintenance make it an easy default when spinning up a new Node.js backend. &lt;/p&gt;

&lt;p&gt;Until you inspect your tail latencies in production.&lt;/p&gt;

&lt;p&gt;When we audited our API telemetry, our &lt;strong&gt;p95 query latency was sitting at 914ms&lt;/strong&gt;. Even simple indexed lookups were occasionally stalling for nearly a full second. &lt;/p&gt;

&lt;p&gt;We migrated that workload from a cross-region serverless Postgres setup to a co-located &lt;strong&gt;Docker PostgreSQL 18&lt;/strong&gt; instance sitting behind &lt;strong&gt;PgBouncer 1.26&lt;/strong&gt; in transaction mode. The result: our &lt;strong&gt;p95 latency dropped from 914ms to 2ms&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Before jumping to "serverless is slow, self-host everything," it is important to isolate where those 912 milliseconds actually went—because most of the win comes from fixing geography and lifecycle states, while the final drop from &lt;strong&gt;9ms to 2ms&lt;/strong&gt; comes from local NVMe and connection pooling.&lt;/p&gt;




&lt;h2&gt;
  
  
  Step 1: Auditing Where the 914ms Actually Went
&lt;/h2&gt;

&lt;p&gt;Blaming the database engine for a 914ms p95 is almost always wrong. PostgreSQL itself executes a simple primary-key lookup in under a millisecond. When we broke down our request traces, three compounding bottlenecks emerged:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Scale-to-Zero Cold Starts (Auto-Suspend):&lt;/strong&gt; When a serverless compute endpoint suspends after inactivity, the first incoming request pays the VM wake-up and storage-attachment penalty (often 400ms–800ms+).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Cross-Region Network Round-Trips (RTT):&lt;/strong&gt; Our Node.js API and the serverless database weren't pinned to the exact same availability zone. Every TCP + TLS handshake plus query round-trip multiplied the geographical latency.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Ephemeral Connection Overhead:&lt;/strong&gt; Without a persistent local pooler keeping warm backend connections open, bursts of concurrent Node.js requests triggered expensive PostgreSQL process forks and TLS negotiations.&lt;/li&gt;
&lt;/ol&gt;

&lt;h3&gt;
  
  
  The Fair Comparison: Fixing Serverless First (~9ms p95)
&lt;/h3&gt;

&lt;p&gt;If you simply move your serverless Postgres instance into the &lt;strong&gt;exact same region&lt;/strong&gt; as your Node.js API and &lt;strong&gt;disable auto-suspend (scale-to-zero)&lt;/strong&gt;, the 914ms nightmare disappears immediately. &lt;/p&gt;

&lt;p&gt;In our testing, a properly configured, always-on, same-region serverless database brought p95 latency down to &lt;strong&gt;~9ms&lt;/strong&gt;. For many teams, stopping right there is the right engineering trade-off.&lt;/p&gt;

&lt;p&gt;However, we wanted predictable sub-3ms performance, zero per-compute-hour anxiety, and direct local NVMe throughput without network hops. That meant co-locating PostgreSQL and PgBouncer alongside the application.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Architecture Stage&lt;/th&gt;
&lt;th&gt;Connection Path&lt;/th&gt;
&lt;th&gt;Auto-Suspend&lt;/th&gt;
&lt;th&gt;p95 Latency&lt;/th&gt;
&lt;th&gt;Primary Bottleneck&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;1. Initial Serverless&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Cross-Region + TLS&lt;/td&gt;
&lt;td&gt;Enabled&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;914ms&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Cold starts + cross-region RTT&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;2. Tuned Serverless&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Same-Region + Pooled&lt;/td&gt;
&lt;td&gt;Disabled&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;~9ms&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;External network hop + proxy overhead&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;3. Self-Hosted Docker&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Co-located + PgBouncer (Local NVMe)&lt;/td&gt;
&lt;td&gt;N/A (Always On)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;2ms&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Local IPC / NVMe I/O only&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;




&lt;h2&gt;
  
  
  Step 2: The Co-Located Architecture (PostgreSQL 18 + PgBouncer 1.26)
&lt;/h2&gt;

&lt;p&gt;When you run your Node.js API and PostgreSQL on the same host (or local private network backed by NVMe storage), network latency drops to near zero. &lt;/p&gt;

&lt;p&gt;However, pointing a high-concurrency Node.js application (using Prisma or Drizzle) directly at PostgreSQL on port &lt;code&gt;5432&lt;/code&gt; creates a different problem: &lt;strong&gt;connection exhaustion and memory bloat&lt;/strong&gt;. Each idle Postgres backend process consumes significant RAM, and spinning up new connections under traffic spikes degrades latency.&lt;/p&gt;

&lt;p&gt;To solve this, we placed &lt;strong&gt;PgBouncer 1.26&lt;/strong&gt; in front of &lt;strong&gt;PostgreSQL 18&lt;/strong&gt; using &lt;strong&gt;Transaction Mode&lt;/strong&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  How the Dual-Port Routing Works (&lt;code&gt;6432&lt;/code&gt; vs. &lt;code&gt;5432&lt;/code&gt;)
&lt;/h3&gt;

&lt;p&gt;A critical mistake teams make when adopting PgBouncer in &lt;code&gt;transaction&lt;/code&gt; mode is routing &lt;em&gt;everything&lt;/em&gt; through it—including schema migrations. Because transaction mode returns the server connection to the pool immediately after a transaction completes, session-level features (like advisory locks used by &lt;strong&gt;Prisma Migrate&lt;/strong&gt; or &lt;strong&gt;Drizzle Kit&lt;/strong&gt;) will fail or behave unpredictably.&lt;/p&gt;

&lt;p&gt;We split our database routing into two explicit paths:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Workload Type&lt;/th&gt;
&lt;th&gt;Target Service&lt;/th&gt;
&lt;th&gt;Port&lt;/th&gt;
&lt;th&gt;Pooling Mode&lt;/th&gt;
&lt;th&gt;Why?&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Application Queries (Node.js API)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;PgBouncer 1.26&lt;/td&gt;
&lt;td&gt;&lt;code&gt;6432&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;transaction&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Multiplexes hundreds of app connections over a small pool of real Postgres backends.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Schema Migrations (Prisma / Drizzle)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Direct to PostgreSQL 18&lt;/td&gt;
&lt;td&gt;&lt;code&gt;5432&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;session&lt;/code&gt; (Direct)&lt;/td&gt;
&lt;td&gt;Requires session-level advisory locks and persistent state during DDL changes.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;




&lt;h2&gt;
  
  
  Step 3: Minimal Production Docker Compose &amp;amp; Pooler Setup
&lt;/h2&gt;

&lt;p&gt;Here is a simplified version of how the services communicate inside Docker Compose so the Node.js application hits PgBouncer over the internal bridge network with zero external TLS handshake overhead:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;services&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;postgres&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;image&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;postgres:18-alpine&lt;/span&gt;
    &lt;span class="na"&gt;container_name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;db_postgres&lt;/span&gt;
    &lt;span class="na"&gt;restart&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;unless-stopped&lt;/span&gt;
    &lt;span class="na"&gt;shm_size&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;1gb&lt;/span&gt;
    &lt;span class="na"&gt;environment&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;POSTGRES_USER&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;${DB_USER}&lt;/span&gt;
      &lt;span class="na"&gt;POSTGRES_PASSWORD&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;${DB_PASSWORD}&lt;/span&gt;
      &lt;span class="na"&gt;POSTGRES_DB&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;${DB_NAME}&lt;/span&gt;
    &lt;span class="na"&gt;volumes&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="s"&gt;pg_nvme_data:/var/lib/postgresql/data&lt;/span&gt;
    &lt;span class="na"&gt;command&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;&amp;gt;&lt;/span&gt;
      &lt;span class="s"&gt;postgres&lt;/span&gt;
      &lt;span class="s"&gt;-c shared_buffers=1GB&lt;/span&gt;
      &lt;span class="s"&gt;-c effective_cache_size=3GB&lt;/span&gt;
      &lt;span class="s"&gt;-c work_mem=16MB&lt;/span&gt;
      &lt;span class="s"&gt;-c max_connections=100&lt;/span&gt;
    &lt;span class="na"&gt;networks&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="s"&gt;backend_net&lt;/span&gt;

  &lt;span class="na"&gt;pgbouncer&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;image&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;edoburu/pgbouncer:v1.26.0&lt;/span&gt;
    &lt;span class="na"&gt;container_name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;db_pgbouncer&lt;/span&gt;
    &lt;span class="na"&gt;restart&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;unless-stopped&lt;/span&gt;
    &lt;span class="na"&gt;depends_on&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="s"&gt;postgres&lt;/span&gt;
    &lt;span class="na"&gt;environment&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;DATABASE_URL&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;postgres://${DB_USER}:${DB_PASSWORD}@postgres:5432/${DB_NAME}"&lt;/span&gt;
      &lt;span class="na"&gt;POOL_MODE&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;transaction&lt;/span&gt;
      &lt;span class="na"&gt;MAX_CLIENT_CONN&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;1000&lt;/span&gt;
      &lt;span class="na"&gt;DEFAULT_POOL_SIZE&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;20&lt;/span&gt;
      &lt;span class="na"&gt;MIN_POOL_SIZE&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;5&lt;/span&gt;
      &lt;span class="na"&gt;RESERVE_POOL_SIZE&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;5&lt;/span&gt;
    &lt;span class="na"&gt;ports&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;127.0.0.1:6432:6432"&lt;/span&gt;
    &lt;span class="na"&gt;networks&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="s"&gt;backend_net&lt;/span&gt;

&lt;span class="na"&gt;networks&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;backend_net&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;driver&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;bridge&lt;/span&gt;

&lt;span class="na"&gt;volumes&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;pg_nvme_data&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;And in our ORM configuration (&lt;code&gt;.env&lt;/code&gt;), we explicitly separate the pooled runtime URL from the direct migration URL:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight ini"&gt;&lt;code&gt;&lt;span class="c"&gt;# Used by the Node.js runtime (via PgBouncer in transaction mode)
&lt;/span&gt;&lt;span class="py"&gt;DATABASE_URL&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;"postgresql://user:password@pgbouncer:6432/app_db?pgbouncer=true"&lt;/span&gt;
&lt;span class="w"&gt;
&lt;/span&gt;&lt;span class="c"&gt;# Used exclusively by Prisma / Drizzle CLI for schema migrations
&lt;/span&gt;&lt;span class="py"&gt;DIRECT_URL&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;"postgresql://user:password@postgres:5432/app_db"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If you want to inspect the exact &lt;code&gt;postgresql.conf&lt;/code&gt; memory tuning parameters, prepared statement handling in PgBouncer 1.26, and the full latency percentiles (p50, p95, p99), we documented our complete &lt;a href="https://xsypher.com/article/why-we-moved-from-serverless-postgres-to-self-hosted-docker-pgbouncer-914ms-to-2ms-p95" rel="noopener noreferrer"&gt;serverless Postgres vs. self-hosted Docker and PgBouncer benchmark&lt;/a&gt; with all raw config files.&lt;/p&gt;




&lt;h2&gt;
  
  
  When Should You Actually Do This? (The Trade-Offs)
&lt;/h2&gt;

&lt;p&gt;Moving from a managed serverless database to self-hosted Docker + PgBouncer is not free—you trade cloud vendor margin for operational responsibility.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Stay on Serverless Postgres (Same-Region, Auto-Suspend Off) if:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;You rely heavily on instant database branching for preview deployments.&lt;/li&gt;
&lt;li&gt;You don't want to manage automated backups (&lt;code&gt;pgBackRest&lt;/code&gt; / WAL archiving to S3/R2), disk monitoring, or minor version upgrades.&lt;/li&gt;
&lt;li&gt;A &lt;strong&gt;~9ms p95&lt;/strong&gt; latency easily satisfies your SLA.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Move to Co-Located Docker + PgBouncer if:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Your API executes multiple sequential queries per request where 8–10ms per round-trip compounds into 50–100ms of wait time.&lt;/li&gt;
&lt;li&gt;You want consistent &lt;strong&gt;1–2ms p95 latency&lt;/strong&gt; backed directly by local NVMe storage.&lt;/li&gt;
&lt;li&gt;You want predictable flat-rate infrastructure costs under sustained 24/7 traffic.&lt;/li&gt;
&lt;/ul&gt;




&lt;h3&gt;
  
  
  Wrapping Up
&lt;/h3&gt;

&lt;p&gt;Most "serverless database latency" horror stories start with a misconfigured region or an aggressive scale-to-zero policy. Fix those first. But if your workload demands single-digit millisecond tail latencies at fraction-of-a-cent compute costs, a co-located PostgreSQL 18 container paired with PgBouncer 1.26 remains unbeatable.&lt;/p&gt;

&lt;p&gt;For the complete architectural breakdown, automated backup scripts, and query telemetry charts, check out the original engineering write-up on &lt;a href="https://xsypher.com/article/why-we-moved-from-serverless-postgres-to-self-hosted-docker-pgbouncer-914ms-to-2ms-p95" rel="noopener noreferrer"&gt;xSypher&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Have you hit connection pooling or cold-start bottlenecks with serverless Postgres in production, or are you running your own pooler stack? Drop your setup in the comments!&lt;/em&gt;&lt;/p&gt;

</description>
      <category>postgres</category>
      <category>docker</category>
      <category>node</category>
      <category>database</category>
    </item>
    <item>
      <title>REST vs GraphQL vs gRPC: None of Them Remove Complexity, They Just Move the Bill</title>
      <dc:creator>Anwar Iqbal Dawar</dc:creator>
      <pubDate>Wed, 21 Oct 2026 11:24:59 +0000</pubDate>
      <link>https://dev.to/anwariqbaldawar/rest-vs-graphql-vs-grpc-none-of-them-remove-complexity-they-just-move-the-bill-54jh</link>
      <guid>https://dev.to/anwariqbaldawar/rest-vs-graphql-vs-grpc-none-of-them-remove-complexity-they-just-move-the-bill-54jh</guid>
      <description>&lt;p&gt;Every few months, an engineering team reads a benchmark claiming gRPC is "10x faster than REST" or that GraphQL will magically unify their frontend—only to regret the migration six months later when the Day-2 operational bill arrives.&lt;/p&gt;

&lt;p&gt;After analyzing the production breakdown in &lt;strong&gt;&lt;a href="https://xsypher.com" rel="noopener noreferrer"&gt;REST vs GraphQL vs gRPC: Choosing the Right API Architecture&lt;/a&gt;&lt;/strong&gt; by Anwar Iqbal Dawar on xSypher, one core truth stands out: &lt;strong&gt;none of the three architectures removes complexity—each one simply shifts the operational burden onto a different team.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;If you are designing a new system or debating a migration, here is a pragmatic breakdown of where the hidden costs actually land.&lt;/p&gt;




&lt;h2&gt;
  
  
  📊 The Architectural Trade-Off Matrix
&lt;/h2&gt;

&lt;p&gt;Before diving into the edge cases, here is who actually pays the tax for each protocol in production:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Dimension&lt;/th&gt;
&lt;th&gt;REST (OpenAPI 3.1)&lt;/th&gt;
&lt;th&gt;GraphQL (GraphQL.js v17)&lt;/th&gt;
&lt;th&gt;gRPC (Protobuf Editions)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Wire Format&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;JSON over HTTP/1.1, H2, H3&lt;/td&gt;
&lt;td&gt;JSON over HTTP &lt;code&gt;POST&lt;/code&gt; / &lt;code&gt;GET&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Binary Protobuf over HTTP/2&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;HTTP Caching&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;Free&lt;/strong&gt; (&lt;code&gt;GET&lt;/code&gt; + &lt;code&gt;ETag&lt;/code&gt; + CDN)&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;Earned&lt;/strong&gt; (Requires APQ over &lt;code&gt;GET&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;None&lt;/strong&gt; (App-layer Redis required)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Schema Evolution&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;URL versioning (&lt;code&gt;/v2/&lt;/code&gt;) + &lt;code&gt;oasdiff&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Additive changes + &lt;code&gt;@deprecated&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;Field numbers + &lt;code&gt;reserved&lt;/code&gt; tags&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Browser Support&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Native &lt;code&gt;fetch()&lt;/code&gt; everywhere&lt;/td&gt;
&lt;td&gt;Native &lt;code&gt;fetch()&lt;/code&gt; + client cache&lt;/td&gt;
&lt;td&gt;Requires &lt;code&gt;gRPC-Web&lt;/code&gt; proxy or ConnectRPC&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Who Pays the Cost?&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;Client Team&lt;/strong&gt; (Endpoint fan-out)&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;Backend Team&lt;/strong&gt; (N+1 &amp;amp; security)&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;Platform/DevOps&lt;/strong&gt; (L7 load balancing)&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;




&lt;h2&gt;
  
  
  1. Caching: Free vs. Earned vs. Non-Existent
&lt;/h2&gt;

&lt;h3&gt;
  
  
  REST gets HTTP caching for free
&lt;/h3&gt;

&lt;p&gt;When a client calls &lt;code&gt;GET /products/42&lt;/code&gt;, every CDN, browser cache, and reverse proxy on the internet already knows how to cache that response using standard &lt;code&gt;Cache-Control&lt;/code&gt; and &lt;code&gt;ETag&lt;/code&gt; headers. Zero custom infrastructure required.&lt;/p&gt;

&lt;h3&gt;
  
  
  GraphQL has to earn its cache
&lt;/h3&gt;

&lt;p&gt;Because standard GraphQL clients send &lt;code&gt;POST&lt;/code&gt; requests with a query body to a single &lt;code&gt;/graphql&lt;/code&gt; endpoint, CDNs see an opaque request and bypass caching entirely. To get edge caching back, you have to implement &lt;strong&gt;Automatic Persisted Queries (APQ)&lt;/strong&gt; so clients send a &lt;code&gt;GET&lt;/code&gt; request with a SHA-256 query hash. Furthermore, modern setups must adopt the &lt;code&gt;application/graphql-response+json&lt;/code&gt; media type so failed requests don't return deceptive HTTP &lt;code&gt;200 OK&lt;/code&gt; status codes that poison monitoring dashboards.&lt;/p&gt;

&lt;h3&gt;
  
  
  gRPC skips HTTP caching entirely
&lt;/h3&gt;

&lt;p&gt;gRPC sends binary payloads over multiplexed, long-lived HTTP/2 &lt;code&gt;POST&lt;/code&gt; streams. Standard HTTP caches cannot inspect or cache them. If you choose gRPC, you are committing to application-layer caching (Redis, Memcached, or in-memory memoization) from day one.&lt;/p&gt;




&lt;h2&gt;
  
  
  2. Schema Evolution Without Breaking Clients
&lt;/h2&gt;

&lt;p&gt;How do you evolve a payload when mobile clients take months to update?&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;REST relies on team discipline:&lt;/strong&gt; Adding optional fields is safe, but renaming or changing types requires &lt;code&gt;/v1/&lt;/code&gt; to &lt;code&gt;/v2/&lt;/code&gt; URL routing and automated CI checks with tools like &lt;code&gt;oasdiff&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;GraphQL tracks field-level usage:&lt;/strong&gt; Instead of versioning the whole API, you mark fields with &lt;code&gt;@deprecated(reason: "Use displayName")&lt;/code&gt; and monitor field-level telemetry until zero clients request the legacy field.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;gRPC bakes evolution into the binary wire format:&lt;/strong&gt; Protobuf ignores field names on the wire and only reads &lt;strong&gt;field numbers&lt;/strong&gt;. As explained in the &lt;strong&gt;&lt;a href="https://xsypher.com" rel="noopener noreferrer"&gt;modern Protobuf Editions and gRPC schema evolution guide&lt;/a&gt;&lt;/strong&gt;, Google's newer Protobuf Editions (&lt;code&gt;2023&lt;/code&gt;, &lt;code&gt;2024&lt;/code&gt;, and &lt;code&gt;2026&lt;/code&gt;) have officially unified and replaced the old &lt;code&gt;proto2&lt;/code&gt;/&lt;code&gt;proto3&lt;/code&gt; split while keeping wire compatibility intact—provided your team strictly uses the &lt;code&gt;reserved&lt;/code&gt; keyword when retiring old field numbers.&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  3. The "Day-Two" Operational Bill (6 Months in Production)
&lt;/h2&gt;

&lt;p&gt;Launch day looks clean for all three architectures. Six months later is where teams get burned:&lt;/p&gt;

&lt;h3&gt;
  
  
  The GraphQL Trap: N+1 Queries &amp;amp; Depth Attacks
&lt;/h3&gt;

&lt;p&gt;A naive GraphQL resolver fetching 50 orders and their nested line items will silently execute &lt;strong&gt;51 separate SQL queries&lt;/strong&gt; (&lt;code&gt;1 + N&lt;/code&gt;). Every production GraphQL server requires request-scoped batching via &lt;code&gt;DataLoader&lt;/code&gt;. Even more critical: without strict &lt;strong&gt;query depth limits&lt;/strong&gt; and &lt;strong&gt;query complexity budgets&lt;/strong&gt;, a single deeply nested query from a malicious (or careless) client can exhaust your database connection pool.&lt;/p&gt;

&lt;h3&gt;
  
  
  The gRPC Trap: Layer-4 Load Balancers Fail
&lt;/h3&gt;

&lt;p&gt;Standard Kubernetes &lt;code&gt;ClusterIP&lt;/code&gt; Services and AWS Network Load Balancers operate at Layer 4 (TCP). Because gRPC multiplexes all requests over a single persistent HTTP/2 TCP connection, a Layer-4 load balancer will pin 100% of a client's traffic to &lt;strong&gt;one single backend pod&lt;/strong&gt; while the rest sit idle. Running gRPC in production requires Layer-7 HTTP/2-aware proxies like &lt;strong&gt;Envoy&lt;/strong&gt;, a service mesh, or client-side xDS load balancing.&lt;/p&gt;

&lt;h3&gt;
  
  
  The REST Trap: Client Fan-Out
&lt;/h3&gt;

&lt;p&gt;REST stays operationally simple until a complex dashboard or mobile screen requires 6 sequential round-trips to render a single view. At that point, teams either build bloated &lt;code&gt;?include=&lt;/code&gt; query parameters or introduce a Backend-for-Frontend (BFF) layer.&lt;/p&gt;




&lt;h2&gt;
  
  
  💡 The Production Pattern: Run Two on Purpose
&lt;/h2&gt;

&lt;p&gt;Mature engineering organizations rarely treat this as a religious war. Instead, they split the architecture at the network boundary:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Internal Service-to-Service:&lt;/strong&gt; Run &lt;strong&gt;gRPC&lt;/strong&gt; inside the cluster where low latency, strict Protobuf contracts, streaming, and polyglot code generation shine.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;External / Browser Boundary:&lt;/strong&gt; Expose &lt;strong&gt;REST&lt;/strong&gt; (for public APIs, webhooks, and CDN-heavy reads) or &lt;strong&gt;GraphQL&lt;/strong&gt; (as a BFF for complex frontend/mobile product views), using tools like &lt;code&gt;grpc-gateway&lt;/code&gt; or &lt;strong&gt;ConnectRPC&lt;/strong&gt; to bridge the internal gRPC services automatically.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;If you want to dive deeper into the HTTP/3 transport benchmarks, streaming comparisons, and the 3-question decision framework, read the full &lt;strong&gt;&lt;a href="https://xsypher.com" rel="noopener noreferrer"&gt;REST vs GraphQL vs gRPC architecture breakdown on xSypher&lt;/a&gt;&lt;/strong&gt;.&lt;/p&gt;




&lt;h3&gt;
  
  
  💬 Let's Discuss!
&lt;/h3&gt;

&lt;p&gt;Which API architecture are you running in production right now? Have you ever migrated to GraphQL or gRPC and regretted the operational overhead, or did it pay off for your team? Drop your experience in the comments! 👇&lt;/p&gt;

</description>
      <category>architecture</category>
      <category>api</category>
      <category>webdev</category>
      <category>devops</category>
    </item>
    <item>
      <title>Securing AI Agents: Prompt Injection, MicroVMs, Zero Trust</title>
      <dc:creator>Anwar Iqbal Dawar</dc:creator>
      <pubDate>Wed, 21 Oct 2026 01:23:41 +0000</pubDate>
      <link>https://dev.to/anwariqbaldawar/securing-ai-agents-prompt-injection-microvms-zero-trust-6ok</link>
      <guid>https://dev.to/anwariqbaldawar/securing-ai-agents-prompt-injection-microvms-zero-trust-6ok</guid>
      <description>&lt;p&gt;title: "The Lethal Trifecta: Securing Autonomous AI Agents in Production"&lt;/p&gt;

&lt;p&gt;published: true&lt;/p&gt;

&lt;p&gt;description: "A deep dive into securing AI agents against indirect prompt injection using MicroVMs, Zero Trust, and out-of-band approvals."&lt;/p&gt;

&lt;p&gt;tags: "ai, security, architecture, cybersecurity"&lt;/p&gt;

&lt;p&gt;cover_image: ""&lt;br&gt;
When we give an AI agent the ability to read emails, browse the web, and execute shell commands, we are essentially building a remote-code-execution (RCE) surface with a natural-language parser on the front.&lt;br&gt;
If you are building AI agents that touch production data, relying on "smarter system prompts" is a losing game. EchoLeak (CVE-2025-32711) proved this by showing how a zero-click indirect prompt injection could exfiltrate internal data without the user ever clicking a link.&lt;br&gt;
In this post, we will deeply analyze the modern threat model for AI agents and explore a defense-in-depth architecture using MicroVMs, Zero Trust, and Out-of-band Approvals, inspired by the exceptional research over at xSypher.&lt;br&gt;
&lt;/p&gt;
&lt;div class="crayons-card c-embed text-styles text-styles--secondary"&gt;
    &lt;div class="c-embed__content"&gt;
        &lt;div class="c-embed__cover"&gt;
          &lt;a href="https://xsypher.com/article/securing-ai-agents-prompt-injection-microvms-zero-trust" class="c-link align-middle" rel="noopener noreferrer"&gt;
            &lt;img alt="" src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fpub-cb79a088effc4ad3a3636de153e20b63.r2.dev%2Fxsypher%2Farticles%2F2026%2F10%2F724f3e60-4b8b-4551-b05f-5e13c1fb9699.webp" height="450" class="m-0" width="800"&gt;
          &lt;/a&gt;
        &lt;/div&gt;
      &lt;div class="c-embed__body"&gt;
        &lt;h2 class="fs-xl lh-tight"&gt;
          &lt;a href="https://xsypher.com/article/securing-ai-agents-prompt-injection-microvms-zero-trust" rel="noopener noreferrer" class="c-link"&gt;
            Securing AI Agents: Prompt Injection, MicroVMs, Zero Trust
          &lt;/a&gt;
        &lt;/h2&gt;
          &lt;p class="truncate-at-3"&gt;
            How to stop indirect prompt injection and MCP tool poisoning using Firecracker microVMs, gVisor, CaMeL data-flow policies, Ed25519 HITL, and egress proxies.
          &lt;/p&gt;
        &lt;div class="color-secondary fs-s flex items-center"&gt;
            &lt;img alt="favicon" class="c-embed__favicon m-0 mr-2 radius-0" src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fxsypher.com%2Ffavicon.ico%3Fv%3D3" width="48" height="48"&gt;
          xsypher.com
        &lt;/div&gt;
      &lt;/div&gt;
    &lt;/div&gt;
&lt;/div&gt;


&lt;p&gt;The Threat Model: When Agents Become Actors&lt;br&gt;
The Open Worldwide Application Security Project (OWASP) keeps Prompt Injection at the top of its AI risk list for a reason. But the real danger emerges when models are granted tools, persistent memory, and execution rights.&lt;br&gt;
Simon Willison famously described the "lethal trifecta" of AI agent vulnerabilities:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt; The agent can read private data.&lt;/li&gt;
&lt;li&gt; The agent is exposed to untrusted content (like web pages or emails).&lt;/li&gt;
&lt;li&gt; The agent can communicate externally (exfiltration).
If an agent has all three, it is a ticking time bomb for data exfiltration.
The EchoLeak Lesson
In the EchoLeak vulnerability, an attacker sent a crafted email. When Microsoft 365 Copilot summarized the user's inbox, it read the attacker's hidden instructions. The agent was tricked into gathering confidential files and rendering them into an image tag that fetched a URL, silently exfiltrating the data.
The takeaway: Tool descriptions and external data are untrusted inputs. The agent never "decided" to be malicious; it just parsed instructions that arrived as data.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The Defense-in-Depth Architecture&lt;br&gt;
To secure autonomous agents, we must stop trying to build a perfect filter and instead apply standard secure engineering principles: isolate the runtime, strip privileges, and deny all egress by default.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Isolate the Runtime (Use MicroVMs, Not Just Containers)
Running AI-generated code in a standard Docker container is dangerous. A single kernel exploit can compromise the host.
Instead, use hardware virtualization boundaries like KVM. AWS Firecracker is the gold standard here. It strips out legacy emulation, boots in milliseconds (~150ms from cold boot), and runs with minimal memory overhead.
If KVM isn't available (e.g., on some managed Kubernetes nodes), use Google gVisor (runsc) to interpose a user-space kernel between the workload and the host.&lt;/li&gt;
&lt;li&gt;The Dual-LLM Pattern (CaMeL)
Never let your primary orchestrator ingest raw, untrusted content.
·  Privileged Orchestrator LLM: Sees the trusted request and tool schemas.
·  Quarantined LLM: Parses untrusted payloads (emails, PDFs) but has zero tool-calling authority.
By separating these concerns, untrusted text never directly alters the control flow of the agent.&lt;/li&gt;
&lt;li&gt;Zero Trust &amp;amp; Out-of-Band Approvals
The AI model should never hold a credential. Every tool call proposed by the agent should leave the isolated environment as structured JSON and hit a deterministic Policy Enforcement Point (PEP).
For irreversible actions (like send_email or execute_transaction), require an out-of-band cryptographic signature (e.g., Ed25519) from a human in the loop.&lt;/li&gt;
&lt;li&gt;Block Outbound Data Leaks
Enforce default-deny Layer 7 egress proxies. If the agent doesn't explicitly need to talk to a specific external API to complete its task, the network should drop the connection. This breaks the exfiltration leg of the "lethal trifecta."&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Conclusion&lt;br&gt;
The era of trusting LLMs to behave safely based on system prompts is over. As developers, we must treat AI agents exactly like we treat untrusted user input: sanitize the context, sandbox the execution, and strictly enforce the principle of least privilege.&lt;br&gt;
For a comprehensive technical blueprint, including Firecracker configurations and real-world implementation details, I highly recommend reading the full deep dive here:&lt;br&gt;
Read the full article: Securing AI Agents: Prompt Injection, MicroVMs, Zero Trust (xSypher).&lt;br&gt;
How are you securing your agentic workflows? Let's discuss in the comments below!&lt;/p&gt;

</description>
      <category>ai</category>
      <category>cybersecurity</category>
      <category>security</category>
    </item>
    <item>
      <title>The RAG Verification Trap: Why AI Still Hallucinates in Production (And How to Fix It)</title>
      <dc:creator>Anwar Iqbal Dawar</dc:creator>
      <pubDate>Mon, 19 Oct 2026 12:50:14 +0000</pubDate>
      <link>https://dev.to/anwariqbaldawar/the-rag-verification-trap-why-ai-still-hallucinates-in-production-and-how-to-fix-it-4c2h</link>
      <guid>https://dev.to/anwariqbaldawar/the-rag-verification-trap-why-ai-still-hallucinates-in-production-and-how-to-fix-it-4c2h</guid>
      <description>&lt;p&gt;When we integrate Large Language Models (LLMs) into production pipelines, we often treat hallucinations as rare edge cases. In reality, hallucinations aren't bugs—they are a core feature of how autoregressive models work. &lt;/p&gt;

&lt;p&gt;LLMs are structurally optimized to predict the most statistically probable next token, not to evaluate truth. Because a model outputs fabricated API parameters, nonexistent libraries, or false data with the exact same syntax and confidence as verified facts, standard error-handling completely misses them.&lt;/p&gt;

&lt;p&gt;As highlighted in a recent technical deep dive on &lt;a href="https://xsypher.com" rel="noopener noreferrer"&gt;AI hallucinations and mitigation pipelines by xSypher&lt;/a&gt;, hallucinations survive in the gap where a model's output and its cited source are rarely checked side by side. &lt;/p&gt;

&lt;p&gt;If you are building with LLMs today, here is why standard safeguards fail and how to architect a pipeline that actually catches fabrications before they hit production.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Retrieval Trap: Why RAG Isn't Foolproof
&lt;/h2&gt;

&lt;p&gt;The default industry answer to hallucinations is Retrieval-Augmented Generation (RAG). While feeding external context into the prompt reduces reliance on the model's parametric memory, it introduces a silent vulnerability: &lt;strong&gt;The "Faithful but Wrong" Summary.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Consider how a standard RAG pipeline fails in production:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Vector Similarity ≠ Contextual Accuracy:&lt;/strong&gt; Your retriever pulls an outdated version of an internal API doc or a conceptually similar contract that applies to the wrong tier.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Flawless Summarization:&lt;/strong&gt; The LLM summarizes that retrieved chunk with 100% precision.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The Detection Blindspot:&lt;/strong&gt; Automated evaluation frameworks (like LLM-as-a-judge checking for "groundedness") will score the output as completely factual because it matches the retrieved text.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;A grounded summary of the wrong document is still a wrong answer. Furthermore, relying on a "self-confirmation loop"—prompting an agent to double-check its own work—rarely works. Without external deterministic checks, the model will confidently validate its own fabricated reality.&lt;/p&gt;

&lt;h2&gt;
  
  
  A 4-Layer Blueprint to Stop Production Hallucinations
&lt;/h2&gt;

&lt;p&gt;According to the &lt;a href="https://xsypher.com" rel="noopener noreferrer"&gt;xSypher analysis on catching LLM hallucinations&lt;/a&gt;, engineering teams need to move away from trusting prompt magic and instead enforce tactical checkpoints across the stack.&lt;/p&gt;

&lt;h3&gt;
  
  
  1. Tighten Model Constraints
&lt;/h3&gt;

&lt;p&gt;Before touching complex architectures, constrain the generation probability space.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Set &lt;code&gt;temperature&lt;/code&gt; to &lt;strong&gt;≤ 0.7&lt;/strong&gt; (or closer to &lt;code&gt;0.0&lt;/code&gt; - &lt;code&gt;0.2&lt;/code&gt; for strict data extraction and code generation).&lt;/li&gt;
&lt;li&gt;Use &lt;code&gt;top_p&lt;/code&gt; sampling to cut off the long tail of low-probability token jumps where creative fabrications begin.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  2. Strict Context Engineering
&lt;/h3&gt;

&lt;p&gt;Don't just dump raw vector search results into the system prompt. &lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Enforce metadata filtering (e.g., timestamps, document status, version tags) &lt;em&gt;before&lt;/em&gt; semantic search happens.&lt;/li&gt;
&lt;li&gt;Explicitly instruct the system prompt to output a fallback string (e.g., &lt;code&gt;"INSUFFICIENT_CONTEXT"&lt;/code&gt;) if the exact answer isn't present in the payload, rather than allowing it to bridge gaps using pre-trained weights.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  3. Deterministic Automated Verification
&lt;/h3&gt;

&lt;p&gt;Never pass raw LLM output directly to a downstream service or end-user. Treat LLM responses as untrusted user input.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Schema Validators:&lt;/strong&gt; Enforce structured outputs using Pydantic, Zod, or native JSON schema constraints.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;AST &amp;amp; Linter Checks:&lt;/strong&gt; If generating code or SQL, run the output through an Abstract Syntax Tree (AST) parser or dry-run explainer to verify that referenced tables, columns, or methods actually exist.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Structural Diffs:&lt;/strong&gt; Compare generated payloads against known valid patterns before executing API calls.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  4. Human-in-the-Loop (HITL) UI Workflows
&lt;/h3&gt;

&lt;p&gt;Where automated checks end, UX design must take over. &lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Treat LLM output as a draft rather than a final state.&lt;/li&gt;
&lt;li&gt;Build side-by-side source highlighting into your UI so human reviewers can verify the exact paragraph the model used to generate a claim in seconds.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Wrapping Up
&lt;/h2&gt;

&lt;p&gt;Stopping AI hallucinations from draining your operational budget requires shifting from &lt;em&gt;probabilistic hope&lt;/em&gt; to &lt;em&gt;deterministic verification&lt;/em&gt;. By combining tighter sampling parameters, metadata-aware retrieval, and strict schema validation, you can catch fabrications before they cost you in production.&lt;/p&gt;

&lt;p&gt;For a deeper look into the mechanics of why models fabricate data and how to audit your current setup, check out the full write-up by Anwar Iqbal Dawar on &lt;a href="https://xsypher.com" rel="noopener noreferrer"&gt;xSypher: Why AI Hallucinates and How to Catch It Before It Costs You&lt;/a&gt;.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;How is your team currently handling hallucination checks in production? Are you relying on LLM-as-a-judge, deterministic parsers, or human review? Let’s discuss in the comments!&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>programming</category>
    </item>
    <item>
      <title>How to Secure MCP Servers &amp; AI Agents in Production: A Zero-Trust Hardening Blueprint</title>
      <dc:creator>Anwar Iqbal Dawar</dc:creator>
      <pubDate>Mon, 19 Oct 2026 04:25:28 +0000</pubDate>
      <link>https://dev.to/anwariqbaldawar/how-to-secure-mcp-servers-ai-agents-in-production-a-zero-trust-hardening-blueprint-11c1</link>
      <guid>https://dev.to/anwariqbaldawar/how-to-secure-mcp-servers-ai-agents-in-production-a-zero-trust-hardening-blueprint-11c1</guid>
      <description>&lt;p&gt;An AI agent that works on a laptop changes drastically when given production tokens. Hardening relies on assuming the model is already compromised. Auditing connected Model Context Protocol (MCP) servers reveals significant unreviewed tokens and high tool poisoning attack success rates. &lt;/p&gt;

&lt;p&gt;For the complete markdown, code blocks, and diagrams, you can find the full content in the referenced web documents.&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Read the Full 17-Minute Engineering Breakdown:&lt;/strong&gt; Read the full guide on xSypher for benchmarks, YAML configs, and complete details: &lt;strong&gt;&lt;a href="https://xsypher.com/article/how-to-secure-mcp-servers-and-ai-agents-in-production" rel="noopener noreferrer"&gt;How to Secure MCP Servers and AI Agents in Production ↗&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  Key Hardening Steps Summary
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Threat Surface &amp;amp; OWASP Alignment:&lt;/strong&gt; Prompt filters fail because MCP servers ingest untrusted descriptions and results.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Per-Server OAuth 2.1:&lt;/strong&gt; Enforce per-server audience binding and block token passthrough.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Sandbox Runtimes:&lt;/strong&gt; Isolate server processes using containers, non-root UIDs, and read-only filesystems.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Deterministic Policy Broker:&lt;/strong&gt; Implement call-path controls to manage actions and avoid human approval fatigue.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;SHA-256 Manifest Pinning:&lt;/strong&gt; Prevent supply-chain rug pulls by validating canonical tool manifests.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Get the Open-Source Hardening Kit
&lt;/h2&gt;


&lt;div class="ltag-github-readme-tag"&gt;
  &lt;div class="readme-overview"&gt;
    &lt;h2&gt;
      &lt;img src="https://assets.dev.to/assets/github-logo-5a155e1f9a670af7944dd5e12375bc76ed542ea80224905ecaf878b9157cdefc.svg" alt="GitHub logo"&gt;
      &lt;a href="https://github.com/thexsypher" rel="noopener noreferrer"&gt;
        thexsypher
      &lt;/a&gt; / &lt;a href="https://github.com/thexsypher/mcp-production-hardening-kit" rel="noopener noreferrer"&gt;
        mcp-production-hardening-kit
      &lt;/a&gt;
    &lt;/h2&gt;
    &lt;h3&gt;
      Production security controls for Model Context Protocol (MCP) servers &amp;amp; AI agents: OAuth 2.1 audience verification, SHA-256 tool manifest pinning, gVisor/Firecracker sandboxing, and Kubernetes default-deny egress.
    &lt;/h3&gt;
  &lt;/div&gt;
  &lt;div class="ltag-github-body"&gt;
    
&lt;div id="readme" class="md"&gt;&lt;div&gt;
&lt;div class="markdown-heading"&gt;
&lt;h1 class="heading-element"&gt;⛨ MCP PRODUCTION HARDENING KIT&lt;/h1&gt;
&lt;/div&gt;
&lt;div class="markdown-heading"&gt;
&lt;h3 class="heading-element"&gt;Reference Architecture, Threat Matrix &amp;amp; Zero-Trust Controls for Model Context Protocol (MCP) Servers and Autonomous AI Agents&lt;/h3&gt;
&lt;/div&gt;
&lt;p&gt;&lt;a href="https://xsypher.com/article/how-to-secure-mcp-servers-and-ai-agents-in-production" rel="nofollow noopener noreferrer"&gt;&lt;img src="https://camo.githubusercontent.com/845a518f879fd4f287bfbaf22dc3f70be3d1c0c419397d793d457fba64ab3225/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f5055424c49534845445f42592d785379706865725f53656375726974795f4c61622d3030303030303f7374796c653d666f722d7468652d6261646765266c6f676f3d736869656c64266c6f676f436f6c6f723d464633423330" alt="Published by xSypher"&gt;&lt;/a&gt;
&lt;a href="https://xsypher.com" rel="nofollow noopener noreferrer"&gt;&lt;img src="https://camo.githubusercontent.com/f5c86423466bd2b56faa789e653d0c2160f461478e756fad16eeb6174b6079fc/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f415554484f522d416e7761725f497162616c5f44617761722d3131313131313f7374796c653d666f722d7468652d6261646765" alt="Author"&gt;&lt;/a&gt;
&lt;a href="https://xsypher.com/article/how-to-secure-mcp-servers-and-ai-agents-in-production" rel="nofollow noopener noreferrer"&gt;&lt;img src="https://camo.githubusercontent.com/1ff9d5b01ce31c606ab00f123fca9ec6dd9636061ebb83f837be5a14368eee0a/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f4d43505f535045432d4f417574685f322e315f285246435f383730375f2532465f39373238292d4646334233303f7374796c653d666f722d7468652d6261646765" alt="MCP OAuth Spec"&gt;&lt;/a&gt;
&lt;a href="https://xsypher.com/article/how-to-secure-mcp-servers-and-ai-agents-in-production" rel="nofollow noopener noreferrer"&gt;&lt;img src="https://camo.githubusercontent.com/90ea2e48b1acb995c433a9d6799aa2fb9a910c9dfcc9322b9f8946c4fdb24c39/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f4f574153502d4167656e7469635f546f705f31305f4d61707065642d3145393046463f7374796c653d666f722d7468652d6261646765" alt="OWASP Agentic"&gt;&lt;/a&gt;
&lt;a href="https://github.com/thexsypher/mcp-production-hardening-kit/LICENSE" rel="noopener noreferrer"&gt;&lt;img src="https://camo.githubusercontent.com/91a15d67d8f612ae5641ace7e25d10070c57b3754168f5dc130e295b5ddfe4a1/68747470733a2f2f696d672e736869656c64732e696f2f62616467652f4c4943454e53452d4d49542d3245434337313f7374796c653d666f722d7468652d6261646765" alt="License: MIT"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;br&gt;
&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;[ ! ] CORE ENGINEERING PREMISE:&lt;/strong&gt;&lt;br&gt;
&lt;em&gt;An AI agent that works beautifully on a laptop is a completely different system once it holds a production token. The hardening job is not making the model smarter—it is making every tool call survive the assumption that the language model has &lt;strong&gt;already been compromised&lt;/strong&gt;.&lt;/em&gt;&lt;/p&gt;
&lt;/blockquote&gt;
&lt;div class="markdown-heading"&gt;
&lt;h3 class="heading-element"&gt;
  ⟦ ❯_ &lt;a href="https://xsypher.com/article/how-to-secure-mcp-servers-and-ai-agents-in-production" rel="nofollow noopener noreferrer"&gt;READ THE FULL 17-MINUTE ENGINEERING GUIDE &amp;amp; LAB BENCHMARKS ON XSYPHER ↗&lt;/a&gt; ⟧
&lt;/h3&gt;
&lt;/div&gt;
&lt;/div&gt;

&lt;div class="markdown-heading"&gt;
&lt;h2 class="heading-element"&gt;◈ 01 // TELEMETRY &amp;amp; BENCHMARK FINDINGS: WHY STANDARD DEPLOYMENTS FAIL&lt;/h2&gt;

&lt;/div&gt;
&lt;p&gt;Model Context Protocol (MCP) servers hand the language model two things it treats as ground truth: &lt;strong&gt;tool descriptions&lt;/strong&gt; and &lt;strong&gt;tool results&lt;/strong&gt;. Both arrive over the wire, and both can be written by an attacker.&lt;/p&gt;
&lt;p&gt;Before building this hardening kit, our staging bench and industry research revealed critical blind spots:&lt;/p&gt;
&lt;ul&gt;&lt;li&gt;…&lt;/li&gt;&lt;/ul&gt;&lt;/div&gt;
  &lt;/div&gt;
  &lt;div class="gh-btn-container"&gt;&lt;a class="gh-btn" href="https://github.com/thexsypher/mcp-production-hardening-kit" rel="noopener noreferrer"&gt;View on GitHub&lt;/a&gt;&lt;/div&gt;
&lt;/div&gt;


&lt;h3&gt;
  
  
  Further Reading from xSypher Security Lab
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;&lt;a href="https://xsypher.com/article/how-to-secure-mcp-servers-and-ai-agents-in-production" rel="noopener noreferrer"&gt;How to Secure MCP Servers and AI Agents in Production (Full 17-Min Guide &amp;amp; Benchmarks)&lt;/a&gt;&lt;/strong&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;a href="https://xsypher.com" rel="noopener noreferrer"&gt;Why AI Hallucinates and How to Catch It Before It Costs You&lt;/a&gt;&lt;/strong&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;a href="https://xsypher.com" rel="noopener noreferrer"&gt;Local AI vs Cloud AI: Which One Fits Your Work?&lt;/a&gt;&lt;/strong&gt;&lt;/li&gt;
&lt;/ol&gt;

</description>
      <category>ai</category>
      <category>security</category>
      <category>devops</category>
      <category>architecture</category>
    </item>
  </channel>
</rss>
