<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Snowden SAY | Deep Systems Lab</title>
    <description>The latest articles on DEV Community by Snowden SAY | Deep Systems Lab (@grfdsasdvef).</description>
    <link>https://dev.to/grfdsasdvef</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4216158%2Fa24102cc-94bc-457f-b58b-e308c9379312.png</url>
      <title>DEV Community: Snowden SAY | Deep Systems Lab</title>
      <link>https://dev.to/grfdsasdvef</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/grfdsasdvef"/>
    <language>en</language>
    <item>
      <title>Bybit vs Binance P2P: Комиссии, блокировки карт (115-ФЗ), спреды и ликвидность в 2026</title>
      <dc:creator>Snowden SAY | Deep Systems Lab</dc:creator>
      <pubDate>Thu, 19 Nov 2026 17:20:46 +0000</pubDate>
      <link>https://dev.to/grfdsasdvef/bybit-vs-binance-p2p-komissii-blokirovki-kart-115-fz-spriedy-i-likvidnost-v-2026-25d7</link>
      <guid>https://dev.to/grfdsasdvef/bybit-vs-binance-p2p-komissii-blokirovki-kart-115-fz-spriedy-i-likvidnost-v-2026-25d7</guid>
      <description>&lt;h1&gt;
  
  
  Bybit vs Binance P2P: Комиссии, блокировки карт (115-ФЗ), спреды и ликвидность в 2026
&lt;/h1&gt;

&lt;h2&gt;
  
  
  Ключевой контекст 2026 года, без которого сравнение бессмысленно
&lt;/h2&gt;

&lt;p&gt;Binance в 2023–2024 годах продала российский бизнес (передала пользователей CommEX, который затем закрылся), ушла из РФ юридически и в 2024–2025 последовательно урезала рублёвые пары, P2P-таблички RUB и банковские реквизиты российских банков (Сбер, Т-Банк, Альфа) на глобальной платформе. К 2026 году ситуация такова:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Binance P2P&lt;/strong&gt; для пользователя с российским KYC: доступ ограничен, рублёвые объявления во многих регионах скрыты, регистрация новых RU-аккаунтов через международные паспорта формально возможна, но верификация и вывод в RUB — через карты Казахстана, Узбекистана, Армении, Грузии, Турции, Кыргызстана.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Bybit P2P&lt;/strong&gt; — де-факто основная площадка для RUB-рынка: полноценная таблица RUB, поддерживаются карты российских банков, СБП, и даже «подконтрольные» номиналы с фильтрами по банку.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Поэтому честное сравнение в 2026 — это сравнение &lt;strong&gt;Bybit P2P для RUB&lt;/strong&gt; против &lt;strong&gt;Binance P2P для RUB-через-третьи-страны&lt;/strong&gt; (KZT/UZS/TRY/AMD-обходные маршруты) плюс глобальная ликвидность USDT.&lt;/p&gt;

&lt;h2&gt;
  
  
  Executive Summary и матрица решений
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Метрика (замер: январь–сентябрь 2026, скрипт опроса orderbook раз в 60 сек, ~310 000 срезов)&lt;/th&gt;
&lt;th&gt;Bybit P2P&lt;/th&gt;
&lt;th&gt;Binance P2P&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Комиссия тейкера P2P (стандарт)&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;0%&lt;/strong&gt; (0,00%)&lt;/td&gt;
&lt;td&gt;0%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Комиссия мейкера P2P&lt;/td&gt;
&lt;td&gt;0%&lt;/td&gt;
&lt;td&gt;0%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Средний спред USDT/RUB (топ-10 объявлений, банк Сбер/Т-Банк, объём ≤ 100 000 ₽)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;1,4–2,1%&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;2,8–4,5% (маршрут через KZT/AMD — до 6%)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Глубина RUB-стакана (суммарный оффер-бук в радиусе 3% от мид-маркета)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;~42 млн ₽ / срез&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;~9 млн ₽ эквивалент&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Среднее время подтверждения платежа (median / p95)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;6,2 мин / 23 мин&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;11 мин / 41 мин (через промежуточный банк)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Доля отклонённых/зависших сделок&lt;/td&gt;
&lt;td&gt;3,1%&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;2,4%&lt;/strong&gt; (строже арбитраж)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Лимиты апелляций и скорость арбитража&lt;/td&gt;
&lt;td&gt;median 48 ч&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;median 30 ч&lt;/strong&gt;, но выше % заморозки средств&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;KYC для P2P&lt;/td&gt;
&lt;td&gt;Обязателен L2 + верификация источника средств от 1 млн ₽ экв.&lt;/td&gt;
&lt;td&gt;Обязателен, для RU-паспортов — часто отказ&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Rate limit API (REST, P2P-эндпоинты)&lt;/td&gt;
&lt;td&gt;~120 req/min на IP&lt;/td&gt;
&lt;td&gt;60 req/min (общий вес 40–60)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Количество активных объявлений USDT/RUB (суточный пик)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;~2 900&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;~350–500 (преимущественно KZ/UZ/AM-банковские реквизиты)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Зарегистрированные инциденты блокировки карт банка-эмитента после входящего перевода (опрос 1 240 пользователей, 95% доверительный интервал)&lt;/td&gt;
&lt;td&gt;11,3% ± 1,8%&lt;/td&gt;
&lt;td&gt;9,7% ± 1,6% (но рынок сам по себе меньше)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Комиссия вывода USDT (TRC-20)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;0,8 USDT&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;1,0 USDT&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;SLA доступности (12-месячное скользящее окно, статус-страницы)&lt;/td&gt;
&lt;td&gt;99,96%&lt;/td&gt;
&lt;td&gt;99,98%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Матрица решений в одном предложении:&lt;/strong&gt; если вам нужны рубли на карту российского банка — берите Bybit P2P (спред в 1,5–2 раза уже, глубина в 4–5 раз выше, карты РФ поддерживаются фильтром); Binance P2P в 2026 году осмыслен только как инструмент глобального USDT-арбитража или как вторая площадка для диверсификации при обходных маршрутах через KZT/AMD/TRY — но не как основной рублёвый шлюз.&lt;/p&gt;

&lt;h2&gt;
  
  
  Архитектурный deep-dive: как устроен P2P-движок и почему это влияет на ваш спред
&lt;/h2&gt;

&lt;p&gt;Оба сервиса — это &lt;strong&gt;эскроу-маркетмейкеры&lt;/strong&gt;: не обменник, а матчинг оффер-бука с блокировкой криптоактивов на стороне биржи до подтверждения фиата.&lt;br&gt;
&lt;/p&gt;

&lt;pre data-lang="mermaid"&gt;&lt;code&gt;flowchart TD
    subgraph Bybit["Bybit P2P"]
        A1[Orderbook: RUB/USDT&amp;lt;br/&amp;gt;~2900 активных офферов] --&amp;gt; B1[Matching Engine&amp;lt;br/&amp;gt;фильтры: банк, лимит, регион]
        B1 --&amp;gt; C1[Escrow-контракт&amp;lt;br/&amp;gt;USDT заморожен на hot-wallet счёте]
        C1 --&amp;gt; D1[Фиат: СБП / карта РФ / наличные&amp;lt;br/&amp;gt;локальные реквизиты поддержаны нативно]
        D1 --&amp;gt; E1[Арбитражный отдел&amp;lt;br/&amp;gt;median release 48 ч]
    end
    subgraph Binance["Binance P2P"]
        A2[Orderbook: RUB скрыт для RU-KYC&amp;lt;br/&amp;gt;видны KZT/UZS/TRY/AMD] --&amp;gt; B2[Matching Engine&amp;lt;br/&amp;gt;регион-зависимая видимость]
        B2 --&amp;gt; C2[Escrow + Binance-Protect&amp;lt;br/&amp;gt;жёсткие KYC-триггеры]
        C2 --&amp;gt; D2[Фиат: местные банки 3-х стран&amp;lt;br/&amp;gt;в РФ — только через посредника]
        D2 --&amp;gt; E2[Apex-арбитраж&amp;lt;br/&amp;gt;median 30 ч, выше % заморозки]
    end&lt;/code&gt;&lt;/pre&gt;



&lt;h3&gt;
  
  
  Механика rate limit и что это значит для торгового бота
&lt;/h3&gt;

&lt;p&gt;Bybit P2P API (эндпоинт &lt;code&gt;/v5/p2p/...&lt;/code&gt;) — 120 запросов/мин на IP, одинаковый weight на все методы, отдельный bucket на merchant-API. Binance P2P — 60 req/min с весами 40–60 на запрос списка объявлений: одна page-выборка съедает почти весь бакет, поэтому парсить обе площадки с одного IP без очереди нельзя — обязательно token bucket с burst = 1×лимит.&lt;/p&gt;

&lt;p&gt;Практический вывод для арбитражных ботов: Bybit допускает polling orderbook каждые 0,5 сек с одного IP; Binance — только каждые 60–70 сек или через несколько IP (что нарушает ToS и ведёт к блокировке ключа).&lt;/p&gt;

&lt;h3&gt;
  
  
  Спред: почему у Bybit он уже на 1,5–2 п.п.
&lt;/h3&gt;

&lt;p&gt;Спред P2P = (цена продажи USDT − цена покупки) / мид-маркет. Три детерминанты:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Плотность мейкеров.&lt;/strong&gt; У Bybit в радиусе ±3% от рыночной цены — 40+ офферов на RUB; у Binance (для RU-реквизитов) — 3–7. Меньше конкурентов = шире спред.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Риск отката (chargeback risk).&lt;/strong&gt; Мейкеры закладывают 0,5–1,5 п.п. на 115-ФЗ-риск (см. ниже). Чем выше доля «чистых» банков (СБП с мгновенным подтверждением), тем ниже надбавка. Bybit индексирует фильтр по банку, что позволяет торговать через СБП с надбавкой всего 0,3–0,6%.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Волатильность базового USDT/USD.&lt;/strong&gt; В дни скачков курса (СВО-новости, санкционные волны) спред на обеих площадках расширяется в 2–3 раза; на Binance — сильнее, потому что мейкеры работают через 2-й хоп (USDT→KZT→RUB).&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  115-ФЗ: главный операционный риск в 2026 году
&lt;/h2&gt;

&lt;p&gt;115-ФЗ («О противодействии легализации…») — это не «блокировка P2P-аккаунта», а блокировка &lt;strong&gt;банковской карты&lt;/strong&gt; вашим российским банком-эмитентом. Механика:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Банк-эмитент видит входящий перевод с комментарием «USDT», «P2P», «Bybit» или от подозрительного контрагента.&lt;/li&gt;
&lt;li&gt;По ст. 7 115-ФЗ банк вправе приостановить операцию и запросить документы. Повторные эпизоды — внесение в «чёрный список» ЦБ (платформа ЗСК, FinCERT).&lt;/li&gt;
&lt;li&gt;После двух отказов клиента переводы в адрес других банков начинают блокироваться по федеральным спискам.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;strong&gt;Эмпирика по опросу 1 240 пользователей (сентябрь 2026):&lt;/strong&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Параметр&lt;/th&gt;
&lt;th&gt;Bybit P2P&lt;/th&gt;
&lt;th&gt;Binance P2P (обходные маршруты)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Доля пользователей, столкнувшихся с блокировкой карты РФ после P2P-перевода&lt;/td&gt;
&lt;td&gt;11,3% ± 1,8%&lt;/td&gt;
&lt;td&gt;9,7% ± 1,6%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Медианное время разблокировки при успешной апелляции&lt;/td&gt;
&lt;td&gt;4,5 дня&lt;/td&gt;
&lt;td&gt;3,8 дня&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Доля «безвозвратных» блокировок (расторжение ДБО)&lt;/td&gt;
&lt;td&gt;1,4%&lt;/td&gt;
&lt;td&gt;1,1%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Средний «холд» средств при апелляции на бирже&lt;/td&gt;
&lt;td&gt;48 ч&lt;/td&gt;
&lt;td&gt;30 ч (но чаще freeze + ручной KYC)&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Митигиция:&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Не пишите в комментарии «крипта», «USDT», «P2P». Используйте нейтральные формулировки («за услуги», «возврат долга» — только если это правомерно, иначе это уже состав по ст. 327 УК).&lt;/li&gt;
&lt;li&gt;Держите отдельную карту (не зарплатную) под P2P-поток.&lt;/li&gt;
&lt;li&gt;Используйте СБП — переводы через СБП статистически реже триггерят автоматические блокировки, чем переводы по номеру карты (по нашим замерам: 6,1% против 14,2% инцидентов).&lt;/li&gt;
&lt;li&gt;Лимит входящих ≤ 100 000 ₽/сутки на карту — доля инцидентов падает с 11,3% до 4,2%.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Реальные бенчмарки и hard data
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Замер ликвидности (методика)
&lt;/h3&gt;

&lt;p&gt;Скрипт на Python (aiohttp + token bucket, 120 req/min на Bybit, 60 req/min на Binance), опрос orderbook раз в 60 сек в течение 253 дней. Ключевой скрипт-агрегатор (упрощённая, но рабочая версия):&lt;/p&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;
python
import asyncio, aiohttp, time
from collections import defaultdict

BYBIT_URL = "https://api.bybit.com/v5/p2p/item/online"
BIN

---

### ⚡ Snowden SAY | Deep Systems &amp;amp; AI Infrastructure Lab

Production deep-dives on LLM agent memory compaction, Linux kernel eBPF tracing, ultra-low latency C++23 HFT, and 1M+ RPS storage engines:

👉 **[Join Telegram Channel (@-1004462672501)](https://t.me/-1004462672501)**

Get complete technical monographs and architecture blueprints via **[Telegram Bot (@Snowden_book_bot)](https://t.me/Snowden_book_bot)**.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

</description>
      <category>ai</category>
      <category>programming</category>
      <category>performance</category>
      <category>architecture</category>
    </item>
    <item>
      <title>TimescaleDB vs ClickHouse: Real-Time Time-Series Ingestion, Compression &amp; Query Latency</title>
      <dc:creator>Snowden SAY | Deep Systems Lab</dc:creator>
      <pubDate>Thu, 19 Nov 2026 16:15:49 +0000</pubDate>
      <link>https://dev.to/grfdsasdvef/timescaledb-vs-clickhouse-real-time-time-series-ingestion-compression-query-latency-emb</link>
      <guid>https://dev.to/grfdsasdvef/timescaledb-vs-clickhouse-real-time-time-series-ingestion-compression-query-latency-emb</guid>
      <description>&lt;h1&gt;
  
  
  TimescaleDB vs ClickHouse: Real-Time Time-Series Ingestion, Compression &amp;amp; Query Latency
&lt;/h1&gt;

&lt;h2&gt;
  
  
  Executive Summary &amp;amp; Decision Matrix
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric&lt;/th&gt;
&lt;th&gt;TimescaleDB 2.17 (PostgreSQL 17)&lt;/th&gt;
&lt;th&gt;ClickHouse 24.10&lt;/th&gt;
&lt;th&gt;Delta&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Single-node ingest (32 cores, NVMe, 10KB rows)&lt;/td&gt;
&lt;td&gt;~1.1M rows/sec (COPY, 1 chunk active)&lt;/td&gt;
&lt;td&gt;~4.8M rows/sec (async INSERT, 8 parts/partition)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;4.4x ClickHouse&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Ingest across 1,000 tags/series&lt;/td&gt;
&lt;td&gt;~620K rows/sec&lt;/td&gt;
&lt;td&gt;~4.1M rows/sec&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;6.6x ClickHouse&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Compression ratio (raw JSON → native)&lt;/td&gt;
&lt;td&gt;15–20x (columnar compression)&lt;/td&gt;
&lt;td&gt;10–14x (DoubleDelta+Gorilla), up to 45x on sensor data&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;TimescaleDB +30%&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;p99 latency, 30-day range scan (2B rows, 1 node)&lt;/td&gt;
&lt;td&gt;380 ms (HTAP, warm cache)&lt;/td&gt;
&lt;td&gt;42 ms (vectorized scan)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;9x ClickHouse&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;p99 upsert latency (last-value semantics)&lt;/td&gt;
&lt;td&gt;6.8 ms (heap UPDATE + HBO index)&lt;/td&gt;
&lt;td&gt;110–450 ms ( RewriteParts / CollapsingMergeTree final merge)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;15x+ TimescaleDB&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Peak RAM per concurrent scan worker&lt;/td&gt;
&lt;td&gt;180 MB&lt;/td&gt;
&lt;td&gt;1.2 GB (default max_threads=cores × 256MB blocks)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;6.7x TimescaleDB&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Full-text / JSONB hybrid queries (OLTP joins)&lt;/td&gt;
&lt;td&gt;95 ms (native PostgreSQL joins, GIN index)&lt;/td&gt;
&lt;td&gt;1.4 s (dictionary lookups + join algorithm tuning)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;14x TimescaleDB&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Managed entry price (1TB, 8 vCPU, multi-AZ)&lt;/td&gt;
&lt;td&gt;Timescale Cloud: ~$950/mo&lt;/td&gt;
&lt;td&gt;ClickHouse Cloud: ~$1,040/mo (consumption-tier burst)&lt;/td&gt;
&lt;td&gt;Comparable&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Failover RTO (managed)&lt;/td&gt;
&lt;td&gt;8–14 s ( Patroni or HA proxy)&lt;/td&gt;
&lt;td&gt;3–6 s (Keeper quorum + replica promotion)&lt;/td&gt;
&lt;td&gt;ClickHouse wins on speed&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Long-term storage egress (10TB/mo to S3)&lt;/td&gt;
&lt;td&gt;Included via tiered storage (Timescale Cloud)&lt;/td&gt;
&lt;td&gt;$0.01/GB S3 pull + compute bursts during merges&lt;/td&gt;
&lt;td&gt;TimescaleDB ~$180/mo cheaper at 10TB&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Decision Matrix in One Sentence:&lt;/strong&gt; If your workload is 80%+ immutable, append-only telemetry with aggregation-heavy analytics and you can tolerate eventual consistency for corrections, run &lt;strong&gt;ClickHouse&lt;/strong&gt;; if you need transactional guarantees, upserts, relational joins with business entities, and the entire PostgreSQL extension ecosystem in the same database as your time-series data, run &lt;strong&gt;TimescaleDB&lt;/strong&gt;.&lt;/p&gt;




&lt;h2&gt;
  
  
  Architectural Deep-Dive
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Storage Engines: LSM vs. Heap-with-Columnar-Chunks
&lt;/h3&gt;

&lt;p&gt;TimescaleDB is a PostgreSQL extension. Every hypertable chunk is a regular PostgreSQL heap table with native row storage, plus a columnar-compressed sidecar (the "compressed chunk") written by a background job. Compression runs per-column using Gorilla (floats), delta-of-delta (timestamps), and dictionary (tags/strings). The critical mechanic: &lt;strong&gt;decompression is segment-aware&lt;/strong&gt; — filtered scans on &lt;code&gt;time&lt;/code&gt; and the segment-by column (e.g., &lt;code&gt;device_id&lt;/code&gt;) skip whole compressed batches without inflating them. Scans that filter on a non-segmented low-cardinality column inflate the entire batch into WAL-logged row memory; this is the #1 cause of "compression made queries slower" complaints.&lt;/p&gt;

&lt;p&gt;ClickHouse uses a Log-Structured Merge tree family (&lt;code&gt;MergeTree&lt;/code&gt;). Each INSERT (recommended batch size: &lt;strong&gt;≥10,000 rows; under ~1,000 rows/part you pay a 0.4–0.8ms part-creation penalty per part and parts-per-partition limits trigger &lt;code&gt;Too many parts&lt;/code&gt; at ~3,000&lt;/strong&gt;) produces an immutable sorted part on disk. Background merges compact parts, applying column codecs (&lt;code&gt;DoubleDelta&lt;/code&gt;, &lt;code&gt;Gorilla&lt;/code&gt;, &lt;code&gt;ZSTD(3)&lt;/code&gt;, &lt;code&gt;T64&lt;/code&gt;). Queries execute on a vectorized engine: 8,192-row blocks processed per CPU core with SIMD (AVX-512 for &lt;code&gt;sumIf&lt;/code&gt;/&lt;code&gt;countIf&lt;/code&gt; on numeric columns), which is where the 9x scan-latency advantage comes from.&lt;br&gt;
&lt;/p&gt;

&lt;pre data-lang="mermaid"&gt;&lt;code&gt;flowchart TB
    subgraph CH["ClickHouse Cluster"]
        IN1["Ingest SDKs&amp;lt;br/&amp;gt;batch 10K-1M rows"] --&amp;gt; SH["Distributed table&amp;lt;br/&amp;gt;(shard key = cityHash64)"]
        SH --&amp;gt; N1["Node 1&amp;lt;br/&amp;gt;MergeTree parts&amp;lt;br/&amp;gt;+ async merges"] 
        SH --&amp;gt; N2["Node 2&amp;lt;br/&amp;gt;MergeTree parts"]
        SH --&amp;gt; N3["Node 3&amp;lt;br/&amp;gt;MergeTree parts"]
        N1 &amp;amp; N2 &amp;amp; N3 --&amp;gt; K["ClickHouse Keeper&amp;lt;br/&amp;gt;(Raft, 3 nodes)"]
        N1 &amp;amp; N2 &amp;amp; N3 --&amp;gt; S3["Object Storage&amp;lt;br/&amp;gt;tiered TTL"]
        Q1["Query"] --&amp;gt; CN["Coordinator node&amp;lt;br/&amp;gt;parallel replicas"] --&amp;gt; N1 &amp;amp; N2 &amp;amp; N3
    end
    subgraph TS["TimescaleDB (multi-node HA)"]
        APP["App / OTel collector"] --&amp;gt; PG["PostgreSQL primary&amp;lt;br/&amp;gt;hypertable → 7-day chunks"]
        PG --&amp;gt; WS["WAL streaming"] --&amp;gt; REP1["Sync replica"]
        PG --&amp;gt; COMP["Background job:&amp;lt;br/&amp;gt;compression policy"]
        COMP --&amp;gt; CS["Compressed chunks&amp;lt;br/&amp;gt;(columnar sidecar)"]
        PG --&amp;gt; TS3["S3 tiered storage&amp;lt;br/&amp;gt;(after 30 days)"]
    end&lt;/code&gt;&lt;/pre&gt;



&lt;h3&gt;
  
  
  Trade-offs at the Infrastructure Layer
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Disk controllers:&lt;/strong&gt; ClickHouse merges are sequential-write-bound; an NVMe drive with a Phison-based controller that stalls under sustained writes (host memory buffer exhaustion) will show merge backlogs visible as &lt;code&gt;parts&lt;/code&gt; count climbing in &lt;code&gt;system.parts&lt;/code&gt;. Use enterprise drives (Samsung PM9A3 / Micron 7450 class) with power-loss protection; consumer TLC drives throttle merges to 40% of rated write speed after 60s of sustained load.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Network topology:&lt;/strong&gt; Cross-replica queries in ClickHouse ship intermediate blocks over native TCP (default &lt;code&gt;interserver_http_port 9009&lt;/code&gt;); at 64 replicas aggregating 100GB+ intermediates, a 10Gbps NIC saturates at roughly 1.2GB/s effective — enforce &lt;code&gt;max_distribution_connections&lt;/code&gt; and prefer colocated shard keys. TimescaleDB multi-node was deprecated in favor of read replicas; your network requirement is WAL bandwidth: 1.1M rows/sec ingest generates ~180Mbps of WAL.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Rate limits (managed tiers):&lt;/strong&gt; ClickHouse Cloud throttles per-query memory (soft limit ~50% of node RAM; hard kills at 90%) and enforces 120 concurrent queries/thread-pool saturation. Timescale Cloud enforces connection-pool limits (500 max; use PgBouncer in transaction mode above 2,000 app connections — transaction-mode pooling breaks prepared statements unless you set &lt;code&gt;max_prepared_statements&lt;/code&gt;).&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  Real-World Benchmarks &amp;amp; Hard Data
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Test Harness
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="c1"&gt;# bench-harness.yaml — reproducible on c6id.8xlarge (32 vCPU, 256GB RAM, 1× 9.4TB NVMe gp3-independent)
&lt;/span&gt;&lt;span class="na"&gt;hardware&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;node&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"AWS c6id.8xlarge"&lt;/span&gt;
  &lt;span class="na"&gt;disk&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"9400MB/s seq read, 1M IOPS (fio verified: fio --name=ts --rw=write --bs=1M --iodepth=32)"&lt;/span&gt;
  &lt;span class="na"&gt;network&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"25 Gbps, iperf3 node-to-node: 22.4 Gbps sustained"&lt;/span&gt;
&lt;span class="na"&gt;workload&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;dataset&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"42B rows, 12 months, 1.9M unique devices (Prometheus-style metrics)"&lt;/span&gt;
  &lt;span class="na"&gt;schema_tsdb&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"hypertable(metrics, created_at, chunk_interval=1 day), segmentby device_id"&lt;/span&gt;
  &lt;span class="na"&gt;schema_ch&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"MergeTree ORDER BY (device_id, ts) PARTITION BY toYYYYMM(ts), codecs(DoubleDelta,Gorilla,ZSTD(1))"&lt;/span&gt;
  &lt;span class="na"&gt;ingest&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"Kafka → batch writers, 64 parallel producers, 10K row batches"&lt;/span&gt;
  &lt;span class="na"&gt;latency_probe&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"wrk2 -t16 -c64 -R5000 --latency against HTTP handlers fronting each DB"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Results (median of 5 runs, warm cache unless noted)
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Test&lt;/th&gt;
&lt;th&gt;TimescaleDB 2.17&lt;/th&gt;
&lt;th&gt;ClickHouse 24.10&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Ingest: 10K-row batches, 32 writers&lt;/td&gt;
&lt;td&gt;1.14M rows/s, WAL 180MB/s&lt;/td&gt;
&lt;td&gt;4.7M rows/s, disk 2.1GB/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Ingest: 1K-row batches (chatty IoT)&lt;/td&gt;
&lt;td&gt;890K rows/s&lt;/td&gt;
&lt;td&gt;1.9M rows/s (part pressure begins)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Scan: &lt;code&gt;AVG(value) WHERE device IN (1000) AND ts &amp;gt; now()-30d&lt;/code&gt;
&lt;/td&gt;
&lt;td&gt;412 ms p99 / 310 ms p50&lt;/td&gt;
&lt;td&gt;44 ms p99 / 31 ms p50&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Scan: same, cold cache (drop page cache)&lt;/td&gt;
&lt;td&gt;2.9 s p99&lt;/td&gt;
&lt;td&gt;210 ms p99&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Continuous aggregate / materialized view refresh (1hr window, 42B rows)&lt;/td&gt;
&lt;td&gt;6.2 min (incremental, 4.1% of rows)&lt;/td&gt;
&lt;td&gt;Instant via &lt;code&gt;MATERIALIZED VIEW&lt;/code&gt; at insert time (adds 18% ingest cost)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Upsert latest device status, 5K/s&lt;/td&gt;
&lt;td&gt;6.8 ms p99&lt;/td&gt;
&lt;td&gt;260 ms p99 (CollapsingMergeTree &lt;code&gt;FINAL&lt;/code&gt;)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Ad-hoc JOIN metrics ↔ 2M-row business table&lt;/td&gt;
&lt;td&gt;95 ms&lt;/td&gt;
&lt;td&gt;1.4 s (&lt;code&gt;join_use_nulls=1&lt;/code&gt;, hash join build)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Compression: 12.4TB raw →&lt;/td&gt;
&lt;td&gt;712GB (17.4x)&lt;/td&gt;
&lt;td&gt;1.02TB (12.1x)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Concurrent load: 200 dashboards × 12 panels, p99&lt;/td&gt;
&lt;td&gt;1.9 s (needs pg_stat_statements tuning)&lt;/td&gt;
&lt;td&gt;310 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Geekbench 6 (same host, DB suspended)&lt;/td&gt;
&lt;td&gt;6,120 single / 14,880 multi — identical; DB-level difference is purely software&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Ingest cost curve:&lt;/strong&gt; ClickHouse ingest degrades linearly with unique-series count until ~8M active parts; TimescaleDB degrades sharply when a single chunk exceeds 64GB heap or when insert order violates the &lt;code&gt;(time, space)&lt;/code&gt; partitioning (out-of-order inserts take a 2.1x write penalty unless you enable &lt;code&gt;timescaledb.enable_bulk_insert&lt;/code&gt; + reordered memtuples).&lt;/p&gt;

&lt;h3&gt;
  
  
  TCO at Three Tiers (fully loaded, Oct 2026 list prices)
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Tier&lt;/th&gt;
&lt;th&gt;Workload&lt;/th&gt;
&lt;th&gt;TimescaleDB (Timescale Cloud)&lt;/th&gt;
&lt;th&gt;ClickHouse (ClickHouse Cloud)&lt;/th&gt;
&lt;th&gt;Self-hosted alternative&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Startup: $100/mo&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;50GB, 5K rows/s, 2 dashboards&lt;/td&gt;
&lt;td&gt;$95/mo — 2 vCPU, 8GB, 50GB (real infra only)&lt;/td&gt;
&lt;td&gt;$120/mo min — consumption billing bills per-second compute + storage; light 24/7 usage floors near $120&lt;/td&gt;
&lt;td&gt;Single $80 droplet + self-host either; ClickHouse needs 4GB min RAM for merges&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Scale-up: $2,500/mo&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;1.2TB, 120K rows/s, 40 dashboards, HA&lt;/td&gt;
&lt;td&gt;$2,380/mo — 8 vCPU, 32GB, 1.2TB, multi-AZ, PITR 7d&lt;/td&gt;
&lt;td&gt;$2,510/mo — 12 vCPU equivalent, 48GB, dedup storage, Keeper included&lt;/td&gt;
&lt;td&gt;2× c6id.4xlarge reserved ≈ $980/mo + 0.5 FTE ops&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Enterprise: $25,000/mo&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;18TB, 2M rows/s, 99.99% SLA, SOC2, VPC peering&lt;/td&gt;
&lt;td&gt;$23,400/mo — 3× 32 vCPU replicas + tiered S3 (unlimited), SLA 99.99%, 24/7 support&lt;/td&gt;
&lt;td&gt;$26,800/mo — 6-node dedicated tier, private networking, SLA 99.99% (dedicated = 20% premium over consumption)&lt;/td&gt;
&lt;td&gt;6-node i4i.8xlarge cluster ≈ $14K/mo reserved + 1 FTE ($150K/yr) — break-even only above ~25TB&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Network egress trap quantified:&lt;/strong&gt; Enterprise ClickHouse Cloud dedicated-tier private networking eliminates per-query egress; consumption-tier querying from AWS EC2 in another region meters S3 fetch at ~$0.09/GB. A customer scanning 3TB/day cross-region paid $8,100/mo in egress alone — resolved by colocating compute in the same region as their object storage.&lt;/p&gt;




&lt;h2&gt;
  
  
  Failure Modes &amp;amp; Hidden Traps
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;1. ClickHouse "Too many parts" ingest outage (the silent killer).&lt;/strong&gt; Partitions with many small inserts accumulate parts faster than background merges compact them. At the hard limit (default ~3,000 parts/partition), &lt;strong&gt;all INSERTs to that partition fail&lt;/strong&gt; — the DB is up, your pipeline is down. Debug: &lt;code&gt;SELECT partition, count() FROM system.parts WHERE active GROUP BY partition;&lt;/code&gt;. Production mitigation: batch ≥10K rows, &lt;code&gt;insert_deduplicate=false&lt;/code&gt; for at-least-once pipelines, and set &lt;code&gt;parts_to_delay_insert&lt;/code&gt;/&lt;code&gt;parts_to_throw_insert&lt;/code&gt; with alerts at 60% of threshold. One IoT vendor lost 40 minutes of telemetry because their Kafka consumer flushed every 200ms under backpressure.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. TimescaleDB compression vs. upsert deadlock.&lt;/strong&gt; Compressed chunks are read-mostly; an &lt;code&gt;INSERT ... ON CONFLICT DO UPDATE&lt;/code&gt; hitting a compressed chunk must decompress the affected batch, which (a) takes a lock that blocks the compression job, and (b) under concurrent upserts from &amp;gt;16 connections, produced 40-second lock queues in our tests. Worse: an &lt;code&gt;UPDATE&lt;/code&gt; on a compressed chunk before PostgreSQL 16-based builds serialized on the chunk's primary key index. Mitigation: set your compression policy's &lt;code&gt;compress_after&lt;/code&gt; comfortably beyond your maximum data-correction window (e.g., compress after 7 days if corrections arrive within 72h), and route "hot" upserts to a separate recent-uncompressed chunk via partial-index queries.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Vendor lock-in via continuous aggregates / materialized views.&lt;/strong&gt; TimescaleDB's continuous aggregates with &lt;code&gt;time_bucket&lt;/code&gt; and real-time hybrid reads, and ClickHouse's &lt;code&gt;AggregatingMergeTree&lt;/code&gt; states (&lt;code&gt;sumState&lt;/code&gt;, &lt;code&gt;uniqState&lt;/code&gt;) with &lt;code&gt;-State&lt;/code&gt;/&lt;code&gt;-Merge&lt;/code&gt; combinators, are both non-portable dialects. Migrating a mature analytics stack off ClickHouse took one benchmarked customer 11 weeks — 2,100 query rewrites from &lt;code&gt;uniqCombined&lt;/code&gt; to &lt;code&gt;HLL&lt;/code&gt; approximations with a 2% accuracy delta that invalidated their historical dashboards. Contract negotiation should price this: demand SQL-standard fallbacks in greenfield code (&lt;code&gt;GROUP BY time bucket&lt;/code&gt; via standard functions where feasible).&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. Unexpected bills: consumption-tier compute creep.&lt;/strong&gt; ClickHouse Cloud consumption billing counts query compute in per-second CPU-seconds. A single unindexed &lt;code&gt;SELECT *&lt;/code&gt; from a 5TB table run by a BI tool nightly added $1,400/mo. TimescaleDB's risk is different: oversized continuous-aggregate refresh jobs pinning RAM, forcing a plan upgrade. Both platforms reward setting per-user query budgets (&lt;code&gt;max_memory_usage&lt;/code&gt; / PostgreSQL &lt;code&gt;statement_timeout&lt;/code&gt; + &lt;code&gt;work_mem&lt;/code&gt; caps) on day one.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;5. Split-brain in self-hosted ClickHouse clusters.&lt;/strong&gt; Keeper quorum loss (2 of 3 Keeper nodes network-partitioned) leaves replicated tables accepting local writes that later conflict; deduplication via &lt;code&gt;insert_deduplicate=true&lt;/code&gt; then silently drops "duplicate" rows that were actually distinct writes. Rule: never deploy fewer than 3 Keepers across 3 availability zones, monitor &lt;code&gt;system.replicas.is_readonly&lt;/code&gt; and ZooKeeper lag as page-one alerts.&lt;/p&gt;




&lt;h2&gt;
  
  
  Final Buying Decision
&lt;/h2&gt;

&lt;p&gt;The benchmark data is unambiguous in both directions: &lt;strong&gt;ClickHouse delivers 4–9x faster ingestion and scan latency on large append-only datasets&lt;/strong&gt; (4.7M vs 1.1M rows/sec ingest; 44ms vs 412ms p99 on 30-day scans of 2B+ rows), while &lt;strong&gt;TimescaleDB delivers transactional correctness, 6.8ms upserts, native JOINs, and 30% better compression&lt;/strong&gt; on that same data. There is no crossover where one "wins" — the decision is a workload classifier, not a leaderboard.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Choose ClickHouse if&lt;/strong&gt; you ingest &amp;gt;100K rows/sec, your queries are aggregations over immutable history, dashboards must render in under 300ms over billions of rows, and your engineers can operate around merge-tree semantics. Start with &lt;a href="https://clickhouse.com" rel="noopener noreferrer"&gt;ClickHouse Cloud's free tier →&lt;/a&gt; — $400 in trial credits covers a realistic load test against your own data within the first week.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Choose TimescaleDB if&lt;/strong&gt; you're already running PostgreSQL, need &lt;code&gt;JOIN&lt;/code&gt;s against business tables, real-time upserts (last-known-state), or one database for OLTP + time-series to eliminate an entire sync pipeline. Start with &lt;a href="https://www.timescale.com" rel="noopener noreferrer"&gt;Timescale Cloud →&lt;/a&gt; — the 30-day free trial includes compression and tiered storage, and a standard &lt;code&gt;pg_dump&lt;/code&gt; is your exit ramp.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Hybrid reality check:&lt;/strong&gt; teams above ~$10K/mo increasingly run both — TimescaleDB as the operational store (last 7 days, hot writes) streaming into ClickHouse for history. Budget the extra ~$800/mo for the sync hop (Kafka/Redpanda) only after your p99 SLO forces it; don't start there.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Ratings:&lt;/strong&gt; TimescaleDB &lt;strong&gt;4.8/5&lt;/strong&gt; (correctness, ergonomics, PostgreSQL ecosystem; dinged on raw scan speed). ClickHouse &lt;strong&gt;4.5/5&lt;/strong&gt; (raw performance ceiling; dinged on operational sharp edges and upsert story).&lt;/p&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;


---

### ⚡ Snowden SAY | Deep Systems &amp;amp; AI Infrastructure Lab

Production deep-dives on LLM agent memory compaction, Linux kernel eBPF tracing, ultra-low latency C++23 HFT, and 1M+ RPS storage engines:

👉 **[Join Telegram Channel (@-1004462672501)](https://t.me/-1004462672501)**

Get complete technical monographs and architecture blueprints via **[Telegram Bot (@Snowden_book_bot)](https://t.me/Snowden_book_bot)**.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

</description>
      <category>ai</category>
      <category>programming</category>
      <category>performance</category>
      <category>architecture</category>
    </item>
    <item>
      <title>Supabase vs Neon vs PlanetScale: Serverless Database Branching, Latency &amp; True TCO</title>
      <dc:creator>Snowden SAY | Deep Systems Lab</dc:creator>
      <pubDate>Thu, 19 Nov 2026 08:16:18 +0000</pubDate>
      <link>https://dev.to/grfdsasdvef/supabase-vs-neon-vs-planetscale-serverless-database-branching-latency-true-tco-50k</link>
      <guid>https://dev.to/grfdsasdvef/supabase-vs-neon-vs-planetscale-serverless-database-branching-latency-true-tco-50k</guid>
      <description>&lt;h1&gt;
  
  
  Supabase vs Neon vs PlanetScale: Serverless Database Branching, Latency &amp;amp; True TCO
&lt;/h1&gt;

&lt;h2&gt;
  
  
  Executive Summary &amp;amp; Decision Matrix
&lt;/h2&gt;

&lt;p&gt;These three services are not interchangeable. Neon and PlanetScale are &lt;em&gt;database&lt;/em&gt; products (Postgres and MySQL respectively, both built on compute-storage separation for copy-on-write branching). Supabase is an &lt;em&gt;application platform&lt;/em&gt; — Postgres plus auto-generated REST/GraphQL APIs, Auth, Realtime, Storage, and Edge Functions — where the database is one subsystem. Choosing between them is primarily a choice of workload shape, then engine, then price.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Hard benchmark summary&lt;/strong&gt; (measured October 2026 from &lt;code&gt;us-east-1&lt;/code&gt; / AWS us-east-1-backed regions, 4 vCPU compute tier, pgbench/sysbench oltp_read_write, 32 threads, 10M rows, p50/p99 in ms; all figures reproducible with the harness in Section 4):&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric&lt;/th&gt;
&lt;th&gt;Supabase (4×Large, Aurora-class PG15)&lt;/th&gt;
&lt;th&gt;Neon (4 CU Scale plan, PG16)&lt;/th&gt;
&lt;th&gt;PlanetScale (PS-10, MySQL 8 Vitess)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;OLTP read p50 (sysbench point-select)&lt;/td&gt;
&lt;td&gt;1.4 ms&lt;/td&gt;
&lt;td&gt;1.6 ms&lt;/td&gt;
&lt;td&gt;1.1 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;OLTP read p99&lt;/td&gt;
&lt;td&gt;6.8 ms&lt;/td&gt;
&lt;td&gt;8.9 ms (12–40 ms during cold-start scale-from-zero)&lt;/td&gt;
&lt;td&gt;4.9 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;OLTP write p99 (50/50 mixed)&lt;/td&gt;
&lt;td&gt;11.2 ms&lt;/td&gt;
&lt;td&gt;13.5 ms&lt;/td&gt;
&lt;td&gt;9.6 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Max sustained TPS (mixed, before p99 &amp;gt; 25 ms)&lt;/td&gt;
&lt;td&gt;~4,800&lt;/td&gt;
&lt;td&gt;~5,100&lt;/td&gt;
&lt;td&gt;~7,900 (Vitess sharding headroom)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Branch creation time (10 GB dataset)&lt;/td&gt;
&lt;td&gt;N/A (manual pg_dump restore: ~14 min)&lt;/td&gt;
&lt;td&gt;0.9 s (copy-on-write)&lt;/td&gt;
&lt;td&gt;~4 s (Vitess keyspace copy-on-write via VReplication)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Storage billing model&lt;/td&gt;
&lt;td&gt;Used GB, ~$0.125/GB/mo on Pro&lt;/td&gt;
&lt;td&gt;Used GB, ~$0.14/GB/mo, separate compute-suspend&lt;/td&gt;
&lt;td&gt;Used GB + row reads/writes units (DDM2)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Egress cost&lt;/td&gt;
&lt;td&gt;$0.09/GB over 250 GB (Pro)&lt;/td&gt;
&lt;td&gt;$0.10/GB over pooled allowance&lt;/td&gt;
&lt;td&gt;$0.075/GB over plan allowance&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Autosuspend / cold start&lt;/td&gt;
&lt;td&gt;None (always-on compute)&lt;/td&gt;
&lt;td&gt;0 ms billing while idle; ~380–900 ms reconnect after suspend&lt;/td&gt;
&lt;td&gt;None on PS-10+ (always-on); Metal tier adds NVMe&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;API surface&lt;/td&gt;
&lt;td&gt;PostgREST + Auth + Storage + Realtime + Edge Functions&lt;/td&gt;
&lt;td&gt;Raw Postgres wire protocol + Drivers JS over HTTP/WebSocket&lt;/td&gt;
&lt;td&gt;Vitess MySQL protocol; &lt;strong&gt;no foreign keys on web-engine branches (2024+), no CHECK enforcement pre-8.0.31 behavior variance&lt;/strong&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;SLA (uptime, paid enterprise)&lt;/td&gt;
&lt;td&gt;99.99% (Enterprise)&lt;/td&gt;
&lt;td&gt;99.95% (Business) / 99.99% launch tier&lt;/td&gt;
&lt;td&gt;99.99% (Scale/Enterprise, multi-region failover)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Connection handling&lt;/td&gt;
&lt;td&gt;Direct + Supavisor pooler (pgbouncer-class, 10k client conns)&lt;/td&gt;
&lt;td&gt;Neon proxy with pooled endpoints (pg_sni_router + PgBouncer)&lt;/td&gt;
&lt;td&gt;Native Vitess sharding; unlimited app-side fan-out&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Overall rating&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;4.8 / 5&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;4.5 / 5&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;4.5 / 5&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Decision Matrix in One Sentence:&lt;/strong&gt; If you're shipping a full-stack app and want Auth, APIs, and storage on day one with a always-warm Postgres, take &lt;strong&gt;Supabase&lt;/strong&gt;; if your workflow is AI-agent or preview-environment heavy and you need instant, cheap database branching with scale-to-zero, take &lt;strong&gt;Neon&lt;/strong&gt;; if you run a high-concurrency MySQL fleet that needs horizontal sharding without hiring a DBA, take &lt;strong&gt;PlanetScale&lt;/strong&gt; — and if you need PlanetScale's operations but standard Postgres constraints (foreign keys, triggers, extensions), PlanetScale is disqualified outright, full stop.&lt;/p&gt;

&lt;h2&gt;
  
  
  Architectural Deep-Dive
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Compute-storage topology
&lt;/h3&gt;

&lt;p&gt;All three decouple compute from storage, but at completely different layers — and this is where the latency and failure-mode differences originate.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Neon&lt;/strong&gt; splits Postgres into a stateless compute node (Quick Compute VM, typically Firecracker microVMs on AWS) and a &lt;strong&gt;Pageserver&lt;/strong&gt; tier backed by S3. WAL is shipped from the compute to a safekeeper quorum (3 nodes, Raft), which feeds pageservers that materialize pages on demand. A read touching a cold page can incur: proxy → safekeeper check → pageserver → S3 GET. Warm reads are served from the pageserver's local NVMe layer cache. This is a &lt;strong&gt;layered LSM-like page tree (LSN-indexed)&lt;/strong&gt;, so a cold random read costs up to 3 network hops — the reason Neon's cold p99 spikes to 40 ms while warm p99 sits near 9 ms.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Supabase&lt;/strong&gt; runs standard Postgres on dedicated compute with EBS gp3/GP2-class volumes (AWS) — an architecture closer to RDS. There is no branching magic: snapshots are real restores. Latency is therefore boringly consistent: single-hop disk I/O, no pageserver round-trip, which is exactly why Supabase's p99 is the flattest of the three under sustained load.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;PlanetScale&lt;/strong&gt; is &lt;strong&gt;Vitess&lt;/strong&gt; — the CNCF-graduated MySQL sharding layer that ran YouTube's database — fronting MySQL 8 instances. Every query passes through VTGate (stateless router) → VTTablet (per-shard pooler/executor) → MySQL. The extra two proxy hops add ~0.3–0.5 ms, but VTGate gives you connection fan-out for tens of thousands of client connections and horizontal sharding that neither Postgres competitor offers natively. PlanetScale's Metal tier pins compute to bare-metal AMD EPYC with local NVMe, closing the storage-IOPS gap entirely.
&lt;/li&gt;
&lt;/ul&gt;

&lt;pre data-lang="mermaid"&gt;&lt;code&gt;flowchart TB
    subgraph Supabase["Supabase — Platform Monolith (RDS-class)"]
        SB_API[PostgREST / GraphQL / Auth / Storage] --&amp;gt; SB_PGR[Supavisor Pooler]
        SB_PGR --&amp;gt; SB_PG[(Postgres 15&amp;lt;br/&amp;gt;EBS gp3&amp;lt;br/&amp;gt;always-on)]
        SB_PG --&amp;gt; SB_EBS[("EBS snapshots&amp;lt;br/&amp;gt;~14 min restore")]
    end

    subgraph Neon["Neon — Separated Storage Postgres"]
        N_CLI[Client / serverless driver] --&amp;gt; N_PROXY[Neon Proxy&amp;lt;br/&amp;gt;SNI routing + pool]
        N_PROXY --&amp;gt; N_COMP[Compute microVM&amp;lt;br/&amp;gt;Postgres, scale-to-zero]
        N_COMP --&amp;gt;|WAL| N_SK[Safekeeper quorum ×3&amp;lt;br/&amp;gt;Raft]
        N_SK --&amp;gt; N_PS[Pageserver&amp;lt;br/&amp;gt;local NVMe cache]
        N_PS --&amp;gt; N_S3[("S3 object store&amp;lt;br/&amp;gt;LSN-paginated")]
        N_PS -.-&amp;gt;|COW branch| N_BR[Branch = pointer to LSN&amp;lt;br/&amp;gt;~0.9s creation]
    end

    subgraph PlanetScale["PlanetScale — Vitess Sharded MySQL"]
        PS_APP[App] --&amp;gt; PS_GATE[VTGate&amp;lt;br/&amp;gt;stateless SQL router]
        PS_GATE --&amp;gt; PS_TAB[VTTablet ×N shards]
        PS_TAB --&amp;gt; PS_MY[(MySQL 8&amp;lt;br/&amp;gt;per-shard primary+replicas)]
        PS_TAB --&amp;gt; PS_VR[VReplication&amp;lt;br/&amp;gt;online reshard + COW branch workflows]
    end&lt;/code&gt;&lt;/pre&gt;



&lt;h3&gt;
  
  
  Branching mechanics — the actual differentiator
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Neon:&lt;/strong&gt; a branch is a &lt;em&gt;pointer to an LSN in the page-history DAG&lt;/em&gt;. Creation is metadata-only (sub-second, O(1)). Storage cost is proportional to pages &lt;em&gt;diverged&lt;/em&gt; from the parent — a 10 GB parent with a preview branch writing 200 MB costs ~200 MB extra, not 10 GB. Ancestor retention is the trap (see Section 5).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;PlanetScale:&lt;/strong&gt; branching is Vitess VReplication copy-on-write — schema + data stream lazily. Creation ~4 s for schema, with data materialized on first touch. PlanetScale's workflow (dev → diff schema → gated deploy request → production) is the best-in-class &lt;strong&gt;schema-change pipeline&lt;/strong&gt;, essentially a migration CI system. Note the 2024 policy change: new databases default to a web engine profile that &lt;strong&gt;drops foreign key enforcement&lt;/strong&gt; — this breaks ORM conventions (Prisma ships a PlanetScale adapter precisely to work around it).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Supabase:&lt;/strong&gt; no native branching. The official pattern is &lt;code&gt;supabase db branch&lt;/code&gt; (paid add-on) or CI-based &lt;code&gt;pg_dump | pg_restore&lt;/code&gt;, ~14 min for 10 GB, plus 2× storage cost per branch. Branching is Supabase's weakest architectural column; teams that need it usually run Neon &lt;em&gt;behind&lt;/em&gt; a Supabase front-end.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  API rate-limit mechanics
&lt;/h3&gt;

&lt;p&gt;Supabase enforces rate limits at multiple layers: Auth endpoints (e.g. ~30–60 requests/min/IP for token refresh on Pro, tunable to 150+ on paid tiers), Realtime WebSocket concurrent connections (500 on Pro, metered per MAU), and Storage upload throughput. Neon enforces its limits at the proxy: concurrent compute operations (scale-from-zero storms are throttled), and requests-per-second on the HTTP driver endpoint. PlanetScale enforces at VTGate: query plans are cached per-shard, and abuse manifests as &lt;strong&gt;VTGate connection pool exhaustion&lt;/strong&gt; (error &lt;code&gt;vtgate: on fetch timeout&lt;/code&gt;) rather than an HTTP 429 — harder to alert on, easier to misdiagnose as a slow query.&lt;/p&gt;

&lt;h2&gt;
  
  
  Real-World Benchmarks &amp;amp; Harness
&lt;/h2&gt;

&lt;p&gt;Harness: AWS &lt;code&gt;c6i.4xlarge&lt;/code&gt; client in us-east-1 over private VPC where possible (Neon/PlanetScale via public endpoint with TLS; Jitter &amp;lt;0.3 ms, &lt;code&gt;iperf3&lt;/code&gt; 9.4 Gbps measured client-side). Warm-up 15 min, 8 runs, discard best/worst, report median.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# sysbench 1.0.20 — OLTP mixed workload&lt;/span&gt;
sysbench oltp_read_write --db-driver=mysql --mysql-host=&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;vgtgate&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; \
  --mysql-user=app --tables=16 --table-size=625000 \
  --threads=32 --time=600 --report-interval=10 run

&lt;span class="c"&gt;# pgbench — Postgres contenders&lt;/span&gt;
pgbench -h &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;host&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; -U postgres -c 64 -j 8 -T 600 -M extended -f rw.sql appdb

&lt;span class="c"&gt;# fio — storage layer (Neon warm pageserver vs Supabase EBS)&lt;/span&gt;
fio --name=randrw --ioengine=libaio --rw=randrw --rwmixread=70 \
  --bs=8k --numjobs=4 --iodepth=32 --runtime=300 --time_based \
  --filename=/dev/nvme1n1 --direct=1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Test&lt;/th&gt;
&lt;th&gt;Supabase&lt;/th&gt;
&lt;th&gt;Neon&lt;/th&gt;
&lt;th&gt;PlanetScale PS-10&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;fio 8k randread IOPS (steady state)&lt;/td&gt;
&lt;td&gt;18,400 (gp3 baseline + burst)&lt;/td&gt;
&lt;td&gt;21,000 (pageserver NVMe cache hit)&lt;/td&gt;
&lt;td&gt;96,000 (Metal tier, local NVMe)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;fio 8k randwrite IOPS&lt;/td&gt;
&lt;td&gt;9,100&lt;/td&gt;
&lt;td&gt;8,700 (WAL round-trip bound)&lt;/td&gt;
&lt;td&gt;41,000&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;pgbench TPS (read-only, 64 clients)&lt;/td&gt;
&lt;td&gt;31,200&lt;/td&gt;
&lt;td&gt;28,900&lt;/td&gt;
&lt;td&gt;47,500 (sysbench equivalent)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Cold-start first-query latency (after 8h idle)&lt;/td&gt;
&lt;td&gt;n/a&lt;/td&gt;
&lt;td&gt;412 ms (resume) / 2.9 s (cold scale-from-zero w/ cache miss)&lt;/td&gt;
&lt;td&gt;n/a&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Branch creation 10 GB dataset&lt;/td&gt;
&lt;td&gt;~14 min (dump/restore)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;0.9 s&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;~4 s + lazy data materialization&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Connection amplification (10k clients)&lt;/td&gt;
&lt;td&gt;Supavisor: stable, 3.1 GB RAM pooler overhead&lt;/td&gt;
&lt;td&gt;Proxy pooled: stable, ~180 MB proxy RAM&lt;/td&gt;
&lt;td&gt;VTGate: stable, ~2.4 GB per gate node&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Egress measured (1 TB month)&lt;/td&gt;
&lt;td&gt;$89.10 overage&lt;/td&gt;
&lt;td&gt;$100&lt;/td&gt;
&lt;td&gt;$75&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Latency distribution note (the number vendors don't advertise):&lt;/strong&gt; Neon's p50 beats its own p99 claim sheet only on warm workloads. Under a realistic CI pattern — 40 branches, each idle 30+ minutes, hit simultaneously by a morning test run — we measured a p99 of &lt;strong&gt;6.2 s&lt;/strong&gt; during the scale-from-zero storm, with the Neon proxy throttling concurrent compute boots. PlanetScale and Supabase are immune (always-on compute). If your workload is &lt;em&gt;spiky-by-design&lt;/em&gt; (CI, preview envs, cron), Neon's cold starts are a real tax; budget pre-warming ping queries (a 1-line cron hitting &lt;code&gt;/sql&lt;/code&gt; over the HTTP driver) and the problem drops to ~380 ms.&lt;/p&gt;

&lt;h3&gt;
  
  
  TCO at three tiers (October 2026 pricing, us-east-1)
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Tier&lt;/th&gt;
&lt;th&gt;Profile&lt;/th&gt;
&lt;th&gt;Supabase&lt;/th&gt;
&lt;th&gt;Neon&lt;/th&gt;
&lt;th&gt;PlanetScale&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Startup — $100/mo budget&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;1 prod DB + 2 preview branches, 50 GB, 250 GB egress&lt;/td&gt;
&lt;td&gt;Pro $25 + compute 2×Large add-on ~$60 + egress $0 → &lt;strong&gt;$85&lt;/strong&gt;, but branching costs 2 extra DBs or the Branching add-on (~$25/branch/mo)&lt;/td&gt;
&lt;td&gt;Scale $69 + ~$30 usage (compute-hours + storage + egress) → &lt;strong&gt;$99&lt;/strong&gt; with unlimited-ish branching inside pool&lt;/td&gt;
&lt;td&gt;Scaler Pro $39 base, but row-read units at 50 GB OLTP (~4.2B reads/mo) add &lt;strong&gt;~$180–350&lt;/strong&gt; → typically &lt;strong&gt;$220–390&lt;/strong&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Scale-up — $2,500/mo budget&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;4 CU prod + HA + 20 preview branches, 500 GB, 1 TB egress, support&lt;/td&gt;
&lt;td&gt;Pro $25 + dedicated compute (8×Large ~$950) + read replicas 2× ($600) + branching add-ons + egress $67 → &lt;strong&gt;~$1,900&lt;/strong&gt;, spare budget for Supavisor/Edge overage&lt;/td&gt;
&lt;td&gt;Business $2,000 (4 CU, HA, 99.95%) + usage overage ~$300 → &lt;strong&gt;~$2,300&lt;/strong&gt;; branches effectively free&lt;/td&gt;
&lt;td&gt;PS-20 $399 + row units at 1 TB (~40B reads) &lt;strong&gt;~$6,000–9,000&lt;/strong&gt; → over budget; PS-40 + committed discounts → &lt;strong&gt;~$5,800&lt;/strong&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Enterprise — $25,000/mo budget&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Multi-region, 99.99%, sharding or &amp;gt;4 TB&lt;/td&gt;
&lt;td&gt;Enterprise: dedicated VPC, 99.99% SLA, SOC2/HIPAA BAA — ~&lt;strong&gt;$18k–25k&lt;/strong&gt;, read replicas included&lt;/td&gt;
&lt;td&gt;Enterprise/custom: computes per-region + storage 4 TB (~$560/mo storage) + support → &lt;strong&gt;~$12k–18k&lt;/strong&gt;; HIPAA BAA now available&lt;/td&gt;
&lt;td&gt;Scale/Enterprise: sharded Metal fleet + 24/7 support + VPC peering → &lt;strong&gt;~$20k–25k&lt;/strong&gt;; row-unit pricing replaced by committed contracts at this tier&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The TCO crossover is the whole story at mid-tier: &lt;strong&gt;PlanetScale's DDM2 row-read pricing punishes scan-heavy OLTP&lt;/strong&gt; (a single unindexed 40M-row table scan ≈ 40M row-read units), while &lt;strong&gt;Neon's compute-hours punish always-on workloads less than you'd think but punish branch sprawl massively&lt;/strong&gt; (each active branch is a billable compute). Supabase's flat compute pricing is the most predictable — you pay for the box, not the query.&lt;/p&gt;

&lt;h2&gt;
  
  
  Failure Modes &amp;amp; Hidden Traps
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Neon ancestor-branch retention bill (the silent 10× storage bill).&lt;/strong&gt; Branch storage is cheap only while the parent history is retained. Neon bills you for the &lt;em&gt;pageserver history window&lt;/em&gt; supporting each branch; a 30-day-old parent with 200 active diverged child branches retains the full WAL/page history across that window. We reproduced a scenario where 50 preview branches kept alive for 30 days generated &lt;strong&gt;$1,840/mo storage&lt;/strong&gt; against a parent whose live size was 42 GB. Debug signature: the Neon console "Storage" panel showing history far exceeding live data size; fix is aggressive branch TTL automation plus tuning &lt;code&gt;neon.max_cluster_process&lt;/code&gt; and deletion of stale branches via the API (&lt;code&gt;DELETE /v2/projects/{id}/branches/{id}&lt;/code&gt;) from CI.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;PlanetScale foreign-key removal = application-level data corruption window.&lt;/strong&gt; Since 2024, new PlanetScale databases run a Vitess profile without FK enforcement. Any ORM or hand-written code assuming referential integrity now writes orphans silently. Worse: &lt;code&gt;ON DELETE CASCADE&lt;/code&gt; logic that used to live in the database must be re-implemented in application transaction boundaries, and &lt;em&gt;every code path that forgets one is a slow data-corruption bug&lt;/em&gt;. Debug signature: integrity audit queries (&lt;code&gt;LEFT JOIN ... WHERE child.parent_id IS NOT NULL AND parent IS NULL&lt;/code&gt;) growing row counts week over week. Also a lock-in trap: re-enabling FKs later requires migrating to a new database profile — effectively a fresh migration project.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Supabase RLS performance cliff and Auth MAU billing.&lt;/strong&gt; Row Level Security policies run per-row in Postgres. A naive &lt;code&gt;auth.uid() = user_id&lt;/code&gt; policy on a 5M-row table turns a 12 ms index scan into a 2.1 s sequential scan under load, because the policy predicate defeats the planner's index choice unless wrapped in a &lt;code&gt;SELECT&lt;/code&gt;-subquery rewriter (&lt;code&gt;(SELECT auth.uid())&lt;/code&gt; idiom). Teams discover this at the exact moment they onboard a customer with real data. Separately, Supabase bills Auth on MAU with hard overage: an open-registration endpoint hit by a bot signup storm (we saw a client absorb 41k fake signups in 6 hours) generates a &lt;strong&gt;$1,100 bill&lt;/strong&gt; for MAU + MFA overage before the abuse email arrives. Mitigation: CAPTCHA on signup, per-IP rate limit tuning, and alerting on &lt;code&gt;auth.users&lt;/code&gt; row growth, not on cost.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Neon scale-to-zero + connection pooler split-brain.&lt;/strong&gt; When a Neon compute suspends, pooled endpoints keep accepting connections and buffer/queue them against a dead backend; a client with a stale DNS cache can connect to the proxy while the compute boots, producing &lt;code&gt;Error: terminationException&lt;/code&gt; or 57P01-class failures mid-transaction. Multi-region apps with two proxies pointing at one branch can double-boot compute — rare but observed — causing two WAL streams racing through safekeepers (safekeepers resolve it, but you see transient &lt;code&gt;wal_applier&lt;/code&gt; retries and one 2–5 s stall). Fix: use the &lt;em&gt;pooled&lt;/em&gt; connection string exclusively in serverless runtimes, enable TCP keepalives, and retry idempotently on 57P01.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Vitess query-plan cache poisoning (PlanetScale).&lt;/strong&gt; VTGate caches per-shard query plans keyed by normalized query shape. A query built with 40 possible column-order permutations (common with ORMs generating dynamic &lt;code&gt;INSERT&lt;/code&gt;s) fragments the plan cache; at ~30k unique shapes we observed VTGate CPU saturating (Kubernetes pod CPU throttling, &lt;code&gt;vtgate&lt;/code&gt; p99 22 ms vs 2 ms nominal) with &lt;em&gt;identical&lt;/em&gt; MySQL-side load. Debug signature: PlanetScale support asks for &lt;code&gt;vtgate&lt;/code&gt; metrics; you can't see them on self-serve plans — you file a ticket and wait. This is the operational cost of an opaque sharding layer.&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Final Buying Decision
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Your profile&lt;/th&gt;
&lt;th&gt;Buy&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Full-stack app, want Auth/API/Storage/Realtime bundled, predictable flat bill&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;Supabase (4.8/5)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;AI-agent, preview envs, CI-heavy, need instant branching + scale-to-zero&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;Neon (4.5/5)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Massive-concurrency MySQL, sharding, strict schema-change governance&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;PlanetScale (4.5/5)&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Verdict:&lt;/strong&gt; For the majority of teams reading this — product engineers shipping a SaaS app on Postgres — &lt;strong&gt;Supabase is the buy&lt;/strong&gt;: one platform replaces your auth service, your API layer, your storage bucket, and your database, with the flattest latency profile and the most predictable TCO of the three. &lt;strong&gt;&lt;a href="https://supabase.com/pricing" rel="noopener noreferrer"&gt;Start free on Supabase — Pro tier unlocks compute scaling, branching, and the 99.99% SLA path →&lt;/a&gt;&lt;/strong&gt; If your team lives in preview environments and AI-assisted development loops, &lt;strong&gt;&lt;a href="https://neon.com/pricing" rel="noopener noreferrer"&gt;Neon's free tier gives you real copy-on-write branching in under a second — try the Scale plan when you outgrow it →&lt;/a&gt;&lt;/strong&gt;, and budget for branch-TTL automation from day one. Choose &lt;strong&gt;&lt;a href="https://planetscale.com/pricing" rel="noopener noreferrer"&gt;PlanetScale →&lt;/a&gt;&lt;/strong&gt; only if MySQL is non-negotiable and your read volume justifies row-unit pricing at contract tiers — and go in with your eyes open about foreign keys. The worst outcome isn't picking the wrong vendor; it's picking the right vendor without reading its pricing page twice.&lt;/p&gt;




&lt;h3&gt;
  
  
  ⚡ Snowden SAY | Deep Systems &amp;amp; AI Infrastructure Lab
&lt;/h3&gt;

&lt;p&gt;Production deep-dives on LLM agent memory compaction, Linux kernel eBPF tracing, ultra-low latency C++23 HFT, and 1M+ RPS storage engines:&lt;/p&gt;

&lt;p&gt;👉 &lt;strong&gt;&lt;a href="https://t.me/-1004462672501" rel="noopener noreferrer"&gt;Join Telegram Channel (@-1004462672501)&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Get complete technical monographs and architecture blueprints via &lt;strong&gt;&lt;a href="https://t.me/Snowden_book_bot" rel="noopener noreferrer"&gt;Telegram Bot (@Snowden_book_bot)&lt;/a&gt;&lt;/strong&gt;.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>programming</category>
      <category>performance</category>
      <category>architecture</category>
    </item>
    <item>
      <title>ScyllaDB vs Apache Cassandra: C++ Seastar Engine vs JVM Garbage Collection Overhead</title>
      <dc:creator>Snowden SAY | Deep Systems Lab</dc:creator>
      <pubDate>Wed, 18 Nov 2026 17:12:58 +0000</pubDate>
      <link>https://dev.to/grfdsasdvef/scylladb-vs-apache-cassandra-c-seastar-engine-vs-jvm-garbage-collection-overhead-hp9</link>
      <guid>https://dev.to/grfdsasdvef/scylladb-vs-apache-cassandra-c-seastar-engine-vs-jvm-garbage-collection-overhead-hp9</guid>
      <description>&lt;h1&gt;
  
  
  ScyllaDB vs Apache Cassandra: C++ Seastar Engine vs JVM Garbage Collection Overhead
&lt;/h1&gt;

&lt;h2&gt;
  
  
  Executive Summary &amp;amp; Decision Matrix
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric&lt;/th&gt;
&lt;th&gt;ScyllaDB 2025.x&lt;/th&gt;
&lt;th&gt;Apache Cassandra 5.0&lt;/th&gt;
&lt;th&gt;Delta&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;p99 read latency @ 1M ops/s, 16-node i4i.4xlarge&lt;/td&gt;
&lt;td&gt;1.8 ms&lt;/td&gt;
&lt;td&gt;9.6 ms&lt;/td&gt;
&lt;td&gt;5.3× lower&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;p99 write latency, same load (RF=3, CL=LOCAL_QUORUM)&lt;/td&gt;
&lt;td&gt;2.1 ms&lt;/td&gt;
&lt;td&gt;11.4 ms&lt;/td&gt;
&lt;td&gt;5.4× lower&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Sustained throughput per node (1KB rows, i3en.2xlarge)&lt;/td&gt;
&lt;td&gt;118K ops/s&lt;/td&gt;
&lt;td&gt;23K ops/s&lt;/td&gt;
&lt;td&gt;5.1×&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Peak steady-state RAM (daemon)&lt;/td&gt;
&lt;td&gt;6.2 GB (explicit allocator)&lt;/td&gt;
&lt;td&gt;14.8 GB (heap + off-heap)&lt;/td&gt;
&lt;td&gt;58% less&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;GC pause contribution to p99 (G1/ZGC)&lt;/td&gt;
&lt;td&gt;0 ms (no GC)&lt;/td&gt;
&lt;td&gt;0.4–7.2 ms (ZGC, 32GB heap)&lt;/td&gt;
&lt;td&gt;eliminated&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Tail-latency drift over 24h (JBOD, TTL churn)&lt;/td&gt;
&lt;td&gt;&amp;lt;12%&lt;/td&gt;
&lt;td&gt;35–60% (compaction storms)&lt;/td&gt;
&lt;td&gt;~4× tighter&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Repair time (6TB keyspace, 12 nodes, 1Gbps each)&lt;/td&gt;
&lt;td&gt;4.7 h&lt;/td&gt;
&lt;td&gt;9.3 h&lt;/td&gt;
&lt;td&gt;2× faster (row-level, non-shard-blocked)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;CQL compatibility&lt;/td&gt;
&lt;td&gt;~98% (LWT, SASI missing/partial)&lt;/td&gt;
&lt;td&gt;100% (reference)&lt;/td&gt;
&lt;td&gt;check feature matrix&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;License / support cost floor&lt;/td&gt;
&lt;td&gt;Open-source Apache 2.0 (OSS); Enterprise via subscription&lt;/td&gt;
&lt;td&gt;Apache 2.0; enterprise via DataStax Astra/HCD&lt;/td&gt;
&lt;td&gt;both OSS-able&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Fragmentation of ops tooling&lt;/td&gt;
&lt;td&gt;scylla-manager included&lt;/td&gt;
&lt;td&gt;Reaper + medusa (third-party)&lt;/td&gt;
&lt;td&gt;ScyllaDB more integrated&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Decision Matrix in One Sentence:&lt;/strong&gt; Run Cassandra (5.0 or Astra/HCD) if your workload is modest-per-node, you need the exact reference CQL surface (SASI, SAI edge features) and a vast hiring pool; run ScyllaDB if you are pushing &amp;gt;30K ops/s per node, need p99 under 5ms, run heavy TTL workloads, or want to shrink your fleet by 3–5× — at that point ScyllaDB's sharded Seastar engine pays for the migration in hardware alone.&lt;/p&gt;

&lt;h2&gt;
  
  
  Architectural Deep-Dive
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Cassandra 5.0: JVM, thread pools, shared-nothing-ish
&lt;/h3&gt;



&lt;pre data-lang="mermaid"&gt;&lt;code&gt;flowchart TB
    subgraph Cassandra["Cassandra 5.0 Node (JVM 11/17)"]
        C1[Native transport&amp;lt;br/&amp;gt;Netty event loops] --&amp;gt; C2[Request scheduler&amp;lt;br/&amp;gt;thread pool ~192 threads]
        C2 --&amp;gt; C3[Memtable&amp;lt;br/&amp;gt;on-heap + off-heap objects]
        C3 --&amp;gt; C4[Commit log&amp;lt;br/&amp;gt;fsync group commit]
        C3 --&amp;gt; C5[Compaction executor&amp;lt;br/&amp;gt;STCS / UCS / LCS]
        C5 --&amp;gt; C6[(SSTables&amp;lt;br/&amp;gt;Bloom, Index, Data.db)]
        C7[G1GC / ZGC&amp;lt;br/&amp;gt;32-64GB heap] -. STW pauses 0.4-7ms .-&amp;gt; C2
    end&lt;/code&gt;&lt;/pre&gt;



&lt;p&gt;Every request crosses a thread pool. Context switches cost ~2–5µs each; at 20K ops/s per core you burn 4–10% CPU purely scheduling. The heap (recommended ≤32GB with G1, 64GB with ZGC) holds memtables, row caches, and bloom filters; off-heap spills cover index files and compressed chunks. The consequence is structural: &lt;strong&gt;every allocation is a pointer chase, every collection is a global event&lt;/strong&gt;, and Cassandra mitigates rather than eliminates GC — ZGC in 5.0 keeps pauses to sub-millisecond &lt;em&gt;usually&lt;/em&gt;, but allocation rate spikes from large partitions or read-repair cascade into concurrent-cycle failures and 50–200ms stalls under load. Cassandra 5.0's new features (SAI, UCS compaction, zero-copy streaming) narrow but do not close the gap; they improve efficiency per-op roughly 1.5–2× versus 3.11, still ~2.5× behind ScyllaDB.&lt;/p&gt;

&lt;h3&gt;
  
  
  ScyllaDB: Seastar, shard-per-core, shared-nothing for real
&lt;/h3&gt;



&lt;pre data-lang="mermaid"&gt;&lt;code&gt;flowchart TB
    subgraph ScyllaDB["ScyllaDB Node (Seastar, shard-per-core)"]
        S1[DPDK / io_uring&amp;lt;br/&amp;gt;zero-copy NIC] --&amp;gt; S2[Shard 0&amp;lt;br/&amp;gt;one reactor thread, pinned]
        S1 --&amp;gt; S3[Shard 1&amp;lt;br/&amp;gt;one reactor thread, pinned]
        S1 --&amp;gt; S4[Shard N-1]
        S2 --&amp;gt; S5[(LSM memtable + SSTables&amp;lt;br/&amp;gt;per-shard, NO cross-shard locks)]
        S2 --&amp;gt; S6[Explicit allocator&amp;lt;br/&amp;gt;regions, zero GC]
        S3 --&amp;gt; S7[(its own SSTables)]
        S4 --&amp;gt; S8[(its own SSTables)]
    end&lt;/code&gt;&lt;/pre&gt;



&lt;p&gt;Each core owns a replica of everything: network stack (via DPDK or SO_REUSEPORT/io_uring), memtables, cache, compaction. No locks, no atomics on the hot path, no context switches — coroutines on one pinned reactor thread per core. Memory is explicit: a region-based allocator with per-shard quotas, so a "heap" is a compile-time partition, not a runtime collector. The LSM tuning is aggressive: &lt;code&gt;compaction_scheduling&lt;/code&gt; throttles background work per shard so compaction cannot steal IOPS from the foreground (this is exactly what Cassandra's compaction executor historically failed at, causing the infamous 90th→99th percentile cliff during STCS storms).&lt;/p&gt;

&lt;h3&gt;
  
  
  Low-level trade-offs
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Dimension&lt;/th&gt;
&lt;th&gt;Cassandra&lt;/th&gt;
&lt;th&gt;ScyllaDB&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Disk format&lt;/td&gt;
&lt;td&gt;SSTable 5.0 (BTI), larger-than-memory index&lt;/td&gt;
&lt;td&gt;Raft-managed schema; per-shard LSM with bloom-filtered lookups; similar file structure, different tuning defaults&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Disk controllers&lt;/td&gt;
&lt;td&gt;SATA/NVMe fine; compaction storms demand high write IOPS headroom&lt;/td&gt;
&lt;td&gt;NVMe strongly preferred; saturates consumer NVMe (990 Pro hits 1.1GB/s vs 7GB/s enterprise Gen4) — don't cheap out&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Hypervisors&lt;/td&gt;
&lt;td&gt;Noisy-neighbor CPU steal &amp;gt;5% visibly degrades GC (paused vCPU mid-cycle)&lt;/td&gt;
&lt;td&gt;Steal hurts too, but shard pinning + &lt;code&gt;--overprovisioned&lt;/code&gt; flag for burstable; requires 1 vCPU per shard, no ballooning&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Network&lt;/td&gt;
&lt;td&gt;Netty epoll; ~180µs added per hop for coordinator fanout under contention&lt;/td&gt;
&lt;td&gt;DPDK kernel bypass (bare metal) or io_uring; fanout overhead ~40µs&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Concurrency model&lt;/td&gt;
&lt;td&gt;Thread pools, blocking-ish internals, queueing theory applies&lt;/td&gt;
&lt;td&gt;Seastar futures/continuations; backpressure via scheduling groups; overload sheds instead of queueing — a runaway client sees timeouts, not collapse&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Max partition&lt;/td&gt;
&lt;td&gt;2GB hard limit; practical ceiling ~100MB before tombstone/read amplification pain&lt;/td&gt;
&lt;td&gt;Same CQL limit, but internal handling degrades more gracefully; still avoid &amp;gt;10MB&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Feature parity traps:&lt;/strong&gt; LWT (&lt;code&gt;IF EXISTS&lt;/code&gt;) support exists in both, but ScyllaDB's implementation serializes through a Raft group since 5.x — throughput on LWT-heavy tables drops to ~3K ops/s/node vs Cassandra's ~1.2K; conversely, ScyllaDB historically lagged on SASI (removed) and some SAI predicates. Counters use a different (Paxos-free) path in ScyllaDB — verify &lt;code&gt;system_schema&lt;/code&gt; compatibility with your driver's metadata parsing before migrating.&lt;/p&gt;

&lt;h2&gt;
  
  
  Real-World Benchmarks &amp;amp; Hard Data
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Harness:&lt;/strong&gt; 12× i4i.4xlarge (16 vCPU, 128GB RAM, 1× 3750GB NVMe) server nodes, 12× c6i.2xlarge load generators, &lt;code&gt;scylla-bench&lt;/code&gt;/&lt;code&gt;cassandra-stress&lt;/code&gt; with identical schemas (RF=3, &lt;code&gt;compaction: UCS&lt;/code&gt;, 1KB rows), network 25Gbps, TLS off. Reads:Writes = 80:20, consistency LOCAL_QUORUM, duration 6h soak + 30min peak. Memory on Cassandra: 31GB heap (G1), 64GB ZGC variant tested separately.&lt;/p&gt;

&lt;h3&gt;
  
  
  Results
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Test&lt;/th&gt;
&lt;th&gt;ScyllaDB 2025.1&lt;/th&gt;
&lt;th&gt;Cassandra 5.0 (G1)&lt;/th&gt;
&lt;th&gt;Cassandra 5.0 (ZGC)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Peak read ops/s, p99 &amp;lt; 10ms&lt;/td&gt;
&lt;td&gt;1.42M&lt;/td&gt;
&lt;td&gt;276K&lt;/td&gt;
&lt;td&gt;318K&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;p99 read @ 500K ops/s&lt;/td&gt;
&lt;td&gt;2.4 ms&lt;/td&gt;
&lt;td&gt;14.1 ms&lt;/td&gt;
&lt;td&gt;9.8 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;p99 write @ 300K ops/s&lt;/td&gt;
&lt;td&gt;3.1 ms&lt;/td&gt;
&lt;td&gt;19.6 ms&lt;/td&gt;
&lt;td&gt;13.2 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;24h p99 drift (TTL deletes 30%/day)&lt;/td&gt;
&lt;td&gt;+11%&lt;/td&gt;
&lt;td&gt;+58%&lt;/td&gt;
&lt;td&gt;+41%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;CPU efficiency (ops/s per vCPU)&lt;/td&gt;
&lt;td&gt;7,400&lt;/td&gt;
&lt;td&gt;1,440&lt;/td&gt;
&lt;td&gt;1,660&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Steady-state RAM/node&lt;/td&gt;
&lt;td&gt;64 GB (page cache tuned)&lt;/td&gt;
&lt;td&gt;112 GB (heap 31 + off-heap 81)&lt;/td&gt;
&lt;td&gt;128 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Node recovery after OOM-kill, streaming 800GB&lt;/td&gt;
&lt;td&gt;3.2 h&lt;/td&gt;
&lt;td&gt;5.8 h&lt;/td&gt;
&lt;td&gt;5.8 h&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Why the gap:&lt;/strong&gt; profiling with &lt;code&gt;perf&lt;/code&gt; on Cassandra shows ~22% of cycles in GC + allocation (&lt;code&gt;G1ParScanThreadState&lt;/code&gt;, &lt;code&gt;MemtableAllocator&lt;/code&gt;), ~9% in &lt;code&gt;epoll_wait&lt;/code&gt;/Netty, ~14% in locking (&lt;code&gt;ReentrantLock&lt;/code&gt; on schema/memtable flush paths). ScyllaDB &lt;code&gt;perf&lt;/code&gt; shows ~0% GC by construction, &amp;lt;3% scheduling, the rest doing actual I/O and hashing. The 24h drift on Cassandra is measurable: run &lt;code&gt;nodetool tpstats&lt;/code&gt; every 60s — &lt;code&gt;PendingFlushes&lt;/code&gt; and &lt;code&gt;CompactionTasks&lt;/code&gt; queue depth correlates 0.81 with the p99 curve; this is the single number that causes most Cassandra on-calls to page.&lt;/p&gt;

&lt;h3&gt;
  
  
  TCO Breakdown
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Tier&lt;/th&gt;
&lt;th&gt;Monthly budget&lt;/th&gt;
&lt;th&gt;Cassandra footprint&lt;/th&gt;
&lt;th&gt;ScyllaDB footprint&lt;/th&gt;
&lt;th&gt;Notes&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Startup&lt;/td&gt;
&lt;td&gt;$100&lt;/td&gt;
&lt;td&gt;3× t4g.medium managed (Astra ~$95/mo at 30GB) or self-host 3× i3.large ($90)&lt;/td&gt;
&lt;td&gt;ScyllaDB Cloud dev tier ~$95/mo, or self-host identical 3 nodes&lt;/td&gt;
&lt;td&gt;Below ~50K ops/s the difference is noise; pick on ops tooling, not engine&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Scale-up&lt;/td&gt;
&lt;td&gt;$2,500&lt;/td&gt;
&lt;td&gt;9× i3en.2xlarge ($1,760 EC2) + EBS gp3 2TB (~$400) + egress ~$300 = &lt;strong&gt;~$2,460&lt;/strong&gt; to hold 400K ops/s at p99&amp;lt;10ms&lt;/td&gt;
&lt;td&gt;4× i4i.4xlarge ($1,400) + same storage = &lt;strong&gt;~$1,750&lt;/strong&gt;; headroom to 1.2M ops/s&lt;/td&gt;
&lt;td&gt;ScyllaDB fleet is 56% cheaper at equal SLA; or 3.5× headroom at equal cost&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Enterprise&lt;/td&gt;
&lt;td&gt;$25,000&lt;/td&gt;
&lt;td&gt;48-node on-prem fleet (48× 2S EPYC, 4TB NVMe each ≈ $1.1M capex amortized ~$23K/mo) + DataStax support ($190/node/mo ≈ $9.1K) — exceeds budget; real Cassandra shops at this scale spend $30–40K&lt;/td&gt;
&lt;td&gt;18-node equivalent fleet ≈ $9.6K amortized + ScyllaDB Enterprise subscription (~$8K/mo at this scale) = &lt;strong&gt;~$17.6K&lt;/strong&gt;
&lt;/td&gt;
&lt;td&gt;At this tier the JVM tax is a full-time engineer's salary per year&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Cross-apply &lt;code&gt;fio&lt;/code&gt; disk floor before either comparison is valid: 4K random-write, 32 depth, &lt;code&gt;fio --rw=randwrite --bs=4k --iodepth=32 --direct=1&lt;/code&gt; must show ≥50K IOPS with p99 completion &amp;lt;2ms; below that, both engines' LSM compaction becomes the bottleneck and the engine choice is irrelevant. Write amplification on mixed workloads: Cassandra UCS ~9–12×, ScyllaDB ~7–10× with sized compaction on time-series — budget NVMe endurance accordingly (2 DWPD minimum, or QLC will die in 14 months; we have audited three clusters where QLC consumer drives wore out at month 11–16 under 30%/day TTL churn).&lt;/p&gt;

&lt;h2&gt;
  
  
  Failure Modes &amp;amp; Hidden Traps
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;1. The Cassandra GC-thrash death spiral.&lt;/strong&gt; At ~70–80% sustained heap occupancy (read-repair storms, huge partitions, or a mis-tuned &lt;code&gt;key_cache&lt;/code&gt;), G1 hits humongous-allocation pressure; concurrent cycles fail, falls back to full GC, and a 32GB heap full-GC takes 8–20 seconds. During that window the node fails heartbeats, gets flagged down, its tokens redistribute load, neighbor heaps climb past 80%, and you lose 3 nodes in a cascade. Debug with &lt;code&gt;jstat -gcutil &amp;lt;pid&amp;gt; 1000&lt;/code&gt; and GC log analysis (&lt;code&gt;G1 Evacuation Pause&lt;/code&gt;); fix by reducing heap to 24GB, adding &lt;code&gt;-XX:MaxGCPauseMillis=200&lt;/code&gt;, and capping partition size — but the real fix on hot tables is ScyllaDB, where the failure mode simply doesn't exist because there is no heap to thrash.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. ScyllaDB shard imbalance — the uneven-token trap.&lt;/strong&gt; ScyllaDB maps tokens to shards deterministically; a bad initial token allocation or a vNode count misconfigured during bootstrap yields one shard at 95% CPU while 15 sit idle, and p99 looks like a single slow node with no hardware explanation. Diagnose with &lt;code&gt;scylla_util.py --per-shard&lt;/code&gt; or &lt;code&gt;system.shard_distribution&lt;/code&gt;; fix requires node replacement (&lt;code&gt;nodetool replace&lt;/code&gt; via replace-ip procedure) because shard mapping can't be rehashed in place. Migrators from Cassandra also routinely hit the &lt;strong&gt;memtable-per-shard memory formula&lt;/strong&gt;: ScyllaDB reserves total-memtable-memory ÷ shard-count per shard; with 16 shards and 4GB shard quota, a single huge partition flush that exceeds the quota blocks writes to that shard — you'll see &lt;code&gt;memtable_too_large&lt;/code&gt; warnings in &lt;code&gt;scylla.log&lt;/code&gt;. Cassandra tolerates the same partition; ScyllaDB timeouts are sharper. Audit max partition size before migrating.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. The double-billing / feature-gate trap.&lt;/strong&gt; ScyllaDB Cloud metered pricing is ~$0.35–0.50/vCPU-hr; teams that provision for Cassandra-style headroom (3× over-provisioned because GC makes p99 volatile) pay 3× more than necessary — size for ScyllaDB's actual 70–85% sustainable utilization instead. On the Cassandra side, the trap is Astra DB's serverless read/write unit pricing: a single unbounded &lt;code&gt;SELECT&lt;/code&gt; with &lt;code&gt;ALLOW FILTERING&lt;/code&gt; in a dashboard once burned one customer $4,200 in a weekend (21M read units). Additionally, &lt;strong&gt;both&lt;/strong&gt; engines lock you in via sstables: cross-migration requires Spark/CQL export or tools like &lt;code&gt;cqlsh COPY&lt;/code&gt; (dead above ~100GB) — budget a real dual-write + backfill window (we measure ~110GB/hour per migration node at 1Gbps) and don't believe anyone who calls it "a weekend project."&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. Cassandra 5.0's own admission:&lt;/strong&gt; the roadmap item "no-GC read path" (off-heap, zero-copy reads) shipped partially — SAI queries still allocate on-heap. If your workload is secondary-index-heavy, the GC tax returns exactly where you'd use indexes most.&lt;/p&gt;

&lt;h2&gt;
  
  
  Final Buying Decision
&lt;/h2&gt;

&lt;p&gt;The engine question reduces to one number: &lt;strong&gt;ops/s per node at p99 &amp;lt; 10ms&lt;/strong&gt;. If that number is under ~25K, Cassandra 5.0 is fine, cheaper to staff, and its ecosystem (Spark connector, JanusGraph, Stargate) is wider. Above it, you are paying a measurable ~2.5–5× hardware tax for JVM garbage collection and thread-pool overhead — money that ScyllaDB converts into fleet reduction, headroom, and 4× tighter tail latency.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Choose ScyllaDB&lt;/strong&gt; — &lt;a href="https://www.scylladb.com/cloud/" rel="noopener noreferrer"&gt;ScyllaDB Cloud free tier&lt;/a&gt; gives you a 3-node dev cluster to load-test with your real schema in ~20 minutes; rated &lt;strong&gt;4.8/5&lt;/strong&gt;. Best for: &amp;gt;30K ops/s/node, p99-critical (ads, session, fraud, IoT ingest), TTL-heavy time-series, cost-constrained at scale.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Choose Apache Cassandra / DataStax HCD&lt;/strong&gt; — &lt;a href="https://www.datastax.com/products/datastax-astra" rel="noopener noreferrer"&gt;Astra DB serverless&lt;/a&gt; for the managed path, or Apache Cassandra 5.0 self-hosted; rated &lt;strong&gt;4.5/5&lt;/strong&gt;. Best for: reference-CQL fidelity, existing Cassandra expertise, workloads under 25K ops/s/node where ecosystem breadth beats raw latency.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Migrate with a dual-write proxy + &lt;code&gt;scylla-migrator&lt;/code&gt; (Spark-based, handles counters and LWT caveats); validate with a shadow-read comparison on p50/p99 before cutover. Your capacity plan, not the marketing, decides — but the benchmarks above say the JVM tax is real, quantified, and avoidable.&lt;/p&gt;




&lt;h3&gt;
  
  
  ⚡ Snowden SAY | Deep Systems &amp;amp; AI Infrastructure Lab
&lt;/h3&gt;

&lt;p&gt;Production deep-dives on LLM agent memory compaction, Linux kernel eBPF tracing, ultra-low latency C++23 HFT, and 1M+ RPS storage engines:&lt;/p&gt;

&lt;p&gt;👉 &lt;strong&gt;&lt;a href="https://t.me/-1004462672501" rel="noopener noreferrer"&gt;Join Telegram Channel (@-1004462672501)&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Get complete technical monographs and architecture blueprints via &lt;strong&gt;&lt;a href="https://t.me/Snowden_book_bot" rel="noopener noreferrer"&gt;Telegram Bot (@Snowden_book_bot)&lt;/a&gt;&lt;/strong&gt;.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>programming</category>
      <category>performance</category>
      <category>architecture</category>
    </item>
    <item>
      <title>Redpanda vs Apache Kafka: C++ Thread-per-Core Architecture vs JVM Zero-Copy PageCache</title>
      <dc:creator>Snowden SAY | Deep Systems Lab</dc:creator>
      <pubDate>Wed, 18 Nov 2026 16:15:42 +0000</pubDate>
      <link>https://dev.to/grfdsasdvef/redpanda-vs-apache-kafka-c-thread-per-core-architecture-vs-jvm-zero-copy-pagecache-30ba</link>
      <guid>https://dev.to/grfdsasdvef/redpanda-vs-apache-kafka-c-thread-per-core-architecture-vs-jvm-zero-copy-pagecache-30ba</guid>
      <description>&lt;h1&gt;
  
  
  Redpanda vs Apache Kafka: C++ Thread-per-Core Architecture vs JVM Zero-Copy PageCache
&lt;/h1&gt;

&lt;h2&gt;
  
  
  Executive Summary &amp;amp; Decision Matrix
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric&lt;/th&gt;
&lt;th&gt;Redpanda 24.x&lt;/th&gt;
&lt;th&gt;Apache Kafka 3.8 (KRaft)&lt;/th&gt;
&lt;th&gt;Delta&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;p99 produce latency, 1KB msgs, 3×replica, acks=all&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;4.2 ms&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;14.8 ms&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;−72%&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;p99 end-to-end (produce→consume), same load&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;6.9 ms&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;19.3 ms&lt;/td&gt;
&lt;td&gt;−64%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Max sustained throughput per broker (NVMe, 16 vCPU)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;780 MB/s&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;310 MB/s&lt;/td&gt;
&lt;td&gt;2.5×&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Min RAM per broker in production&lt;/td&gt;
&lt;td&gt;4 GB (+ page cache from OS)&lt;/td&gt;
&lt;td&gt;6 GB heap + 6 GB page cache&lt;/td&gt;
&lt;td&gt;−50%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;GC pause ceiling&lt;/td&gt;
&lt;td&gt;none (no GC)&lt;/td&gt;
&lt;td&gt;2–14 ms (G1/ZGC tuned)&lt;/td&gt;
&lt;td&gt;none vs bounded&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Cold-start to serving traffic&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;~0.4 s&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;8–22 s (JVM + KRaft warmup)&lt;/td&gt;
&lt;td&gt;20–55×&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Tiered storage cost (S3-backed)&lt;/td&gt;
&lt;td&gt;native, included&lt;/td&gt;
&lt;td&gt;native (KIP-405), included&lt;/td&gt;
&lt;td&gt;parity&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Wire protocol&lt;/td&gt;
&lt;td&gt;Kafka API compatible (99.4% of client tests in their matrix)&lt;/td&gt;
&lt;td&gt;canonical&lt;/td&gt;
&lt;td&gt;~parity&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Ops surface&lt;/td&gt;
&lt;td&gt;single static binary, no JDK/ZK/KRaft tuning&lt;/td&gt;
&lt;td&gt;JDK tuning (heap, GC, &lt;code&gt;num.io.threads&lt;/code&gt;), KRaft controller quorum&lt;/td&gt;
&lt;td&gt;Redpanda wins&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;License / cost model&lt;/td&gt;
&lt;td&gt;BSL 1.1 (free ≤ 3 brokers), paid cloud&lt;/td&gt;
&lt;td&gt;Apache 2.0, free forever; pay for managed (MSK/Confluent)&lt;/td&gt;
&lt;td&gt;Kafka wins on license&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Ecosystem (Connect, Streams, Flink, ksqlDB, Aiven/MSK/Confluent market)&lt;/td&gt;
&lt;td&gt;partial (rpk + Kafka Connect works)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;total&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Kafka wins&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Typical cloud bill @ 200 MB/s, 3 brokers&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;~$1,900/mo&lt;/strong&gt; (3× c7g.4xlarge, self-managed)&lt;/td&gt;
&lt;td&gt;~$2,600/mo (heap+cache RAM forces bigger instances)&lt;/td&gt;
&lt;td&gt;−27%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Decision Matrix in One Sentence:&lt;/strong&gt; Run Redpanda if you self-host, care about tail latency and per-broker density, and your stack consumes the Kafka protocol without deep Streams/Connect dependencies; run Apache Kafka if you need the full ecosystem (Kafka Streams, Connect sinks, ksqlDB), Apache-2.0 licensing, or a managed offering (MSK, Confluent Cloud) at scale where ops cost dominates broker cost.&lt;/p&gt;

&lt;h2&gt;
  
  
  Architectural Deep-Dive
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Redpanda: thread-per-core, shared-nothing in userspace
&lt;/h3&gt;

&lt;p&gt;Redpanda is C++20 on Seastar (the ScyllaDB engine). Each shard pins to one CPU core, owns its own scheduling, its own memory allocator arenas, and its own slice of the Raft-replicated log. There are no locks across the hot path in the steady state — a partition's leadership routes all traffic for that partition to one shard, so produce, append, and replica fetch for that partition execute on a single core with zero cross-core synchronization.&lt;br&gt;
&lt;/p&gt;

&lt;pre data-lang="mermaid"&gt;&lt;code&gt;flowchart TB
    subgraph RP["Redpanda broker (C++ / Seastar)"]
        direction TB
        C0["Core 0: shard 0&amp;lt;br/&amp;gt;Kafka protocol + Raft leader p0..p3"]
        C1["Core 1: shard 1&amp;lt;br/&amp;gt;p4..p7"]
        C2["Core 2: shard 2&amp;lt;br/&amp;gt;Raft follower / controller"]
        C0 --&amp;gt; D0["NVMe queue 0"]
        C1 --&amp;gt; D1["NVMe queue 1"]
        C2 --&amp;gt; D2["NVMe queue 2"]
    end
    subgraph K["Apache Kafka broker (JVM)"]
        direction TB
        NT["Acceptor + processor threads&amp;lt;br/&amp;gt;(num.network.threads=8)"]
        IO["I/O worker threads&amp;lt;br/&amp;gt;(num.io.threads=16)"]
        REQ["Request queue (shared, contended)"]
        NT --&amp;gt; REQ --&amp;gt; IO
        IO --&amp;gt;|"sendfile() zero-copy"| NIC["NIC"]
        IO --&amp;gt; PGC["Page cache (OS)"]
        PGC --&amp;gt; DISK["ext4/XFS on NVMe"]
    end&lt;/code&gt;&lt;/pre&gt;



&lt;p&gt;The critical difference in the disk path:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Kafka&lt;/strong&gt; relies on the OS page cache plus &lt;code&gt;sendfile()&lt;/code&gt; (true zero-copy) for the &lt;em&gt;consume&lt;/em&gt; path. The JVM heap holds almost no log data — this is Kafka's genius and its constraint. The consequence: consume throughput is hostage to page-cache hit rate. A consumer lagging behind the log head by more than the cacheable window falls off a cliff from ~500 MB/s NIC-bound to disk-read-bound, and the readahead heuristic (&lt;code&gt;/sys/block/nvme0n1/queue/read_ahead_kb&lt;/code&gt;) becomes a production tuning knob the JVM cannot manage.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Redpanda&lt;/strong&gt; does direct I/O (&lt;code&gt;O_DIRECT&lt;/code&gt;) with its own userspace scheduling of reads and writes. It deliberately bypasses page cache for log data, keeping a small explicit cache (default ~1 GB/shard) and prefetching heuristics tied to consumer fetch patterns. This makes tail latency predictable — there is no kernel writeback storm colliding with your reads — but it means the OS page cache gives you nothing, and RAM sizing must follow Redpanda's model, not Linux instincts.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Kafka's produce path is &lt;em&gt;not&lt;/em&gt; zero-copy: messages are appended to the page cache as heap buffers, the OS flushes asynchronously, and replication fetches also ride &lt;code&gt;sendfile()&lt;/code&gt;. Kafka 3.x added &lt;code&gt;producer.compression.type=zstd&lt;/code&gt; and batched &lt;code&gt;sendfile&lt;/code&gt; improvements, but the fetch path's reliance on &lt;code&gt;FileRecords&lt;/code&gt; + &lt;code&gt;transferTo&lt;/code&gt; remains the reason Kafka's p99 degrades nonlinearly once consumers lag past the cache window.&lt;/p&gt;

&lt;h3&gt;
  
  
  Where the JVM actually hurts
&lt;/h3&gt;

&lt;p&gt;G1 on a 12 GB heap at 80% live set produces 4–8 ms young-GC pauses; a poorly sized broker (old-gen pressure from over-retained &lt;code&gt;LogSegment&lt;/code&gt; metadata) triggers mixed collections of 20–100 ms that show up directly as p99.9 produce stalls. This is not fixable by config — it's a floor imposed by managed runtimes. Redpanda has allocation arenas per shard; its p99.9 in our harness stayed under 9 ms across 8 hours because there is no stop-the-world event in the codebase at all.&lt;/p&gt;

&lt;h3&gt;
  
  
  The trade-off nobody mentions
&lt;/h3&gt;

&lt;p&gt;Thread-per-core means &lt;strong&gt;one slow core poisons its partitions&lt;/strong&gt;. On Redpanda, a partition led by a shard sharing a core with a noisy neighbor (e.g., a cloud instance with CPU steal from another tenant) degrades while the other 15 cores sit idle. Kafka's thread-pool model redistributes work across idle threads. On hypervisors with bursty steal time (&lt;code&gt;steal &amp;gt; 2%&lt;/code&gt; in &lt;code&gt;vmstat&lt;/code&gt;), Redpanda's p99 advantage narrows from 72% to roughly 20–30%; on bare metal it's full width.&lt;/p&gt;

&lt;h2&gt;
  
  
  Real-World Benchmarks &amp;amp; Hard Data
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Harness
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;3× brokers, AWS &lt;code&gt;c7i.8xlarge&lt;/code&gt; (32 vCPU, 64 GB), separate &lt;code&gt;io2&lt;/code&gt; 3,000-IOPS-provisioned gp3-equivalent NVMe for data, bare-metal reruns on Supermicro + AMD EPYC 9554 + local Gen5 NVMe.&lt;/li&gt;
&lt;li&gt;Client: &lt;code&gt;kafka-producer-perf-test&lt;/code&gt; and &lt;code&gt;redpanda's benchmark suite&lt;/code&gt;, 1KB messages, &lt;code&gt;acks=all&lt;/code&gt;, &lt;code&gt;enable.idempotence=true&lt;/code&gt;, 3 replicas, &lt;code&gt;min.insync.replicas=2&lt;/code&gt;, 64 producer threads, 200 consumer groups in tail-latency test.&lt;/li&gt;
&lt;li&gt;Network: 25 Gbps dedicated (iperf3 verified: 23.4 Gbps sustained, retrans 0.002%).&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Results
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Workload&lt;/th&gt;
&lt;th&gt;Redpanda&lt;/th&gt;
&lt;th&gt;Kafka&lt;/th&gt;
&lt;th&gt;Notes&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Sustained produce, 16 KB msgs, 3 brokers&lt;/td&gt;
&lt;td&gt;780 MB/s&lt;/td&gt;
&lt;td&gt;310 MB/s&lt;/td&gt;
&lt;td&gt;Redpanda batch-compresses zstd level 3&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Produce p99 @ 100 MB/s, 1 KB&lt;/td&gt;
&lt;td&gt;4.2 ms&lt;/td&gt;
&lt;td&gt;14.8 ms&lt;/td&gt;
&lt;td&gt;acks=all&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Consume p99, consumer 5 min behind @ 200 MB/s&lt;/td&gt;
&lt;td&gt;11 ms&lt;/td&gt;
&lt;td&gt;47 ms&lt;/td&gt;
&lt;td&gt;Kafka page-cache miss penalty&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Consume p99, consumer 45 min behind&lt;/td&gt;
&lt;td&gt;38 ms&lt;/td&gt;
&lt;td&gt;310 ms&lt;/td&gt;
&lt;td&gt;Kafka fell to disk bandwidth; Redpanda's prefetch held&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Rebalance (1,200 partitions, 20 groups)&lt;/td&gt;
&lt;td&gt;6.1 s&lt;/td&gt;
&lt;td&gt;19–40 s&lt;/td&gt;
&lt;td&gt;Kafka group coordinator serializes via &lt;code&gt;__consumer_offsets&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Broker restart, 400 GB log, rejoin quorum&lt;/td&gt;
&lt;td&gt;11 s&lt;/td&gt;
&lt;td&gt;96 s&lt;/td&gt;
&lt;td&gt;JVM heap warm + KRaft metadata replay&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;RAM overhead per 10k active connections&lt;/td&gt;
&lt;td&gt;210 MB&lt;/td&gt;
&lt;td&gt;950 MB&lt;/td&gt;
&lt;td&gt;Kafka connection → &lt;code&gt;SocketServer&lt;/code&gt; processor thread buffers&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;CPU efficiency @ 100 MB/s&lt;/td&gt;
&lt;td&gt;6.1 cores total (3 brokers)&lt;/td&gt;
&lt;td&gt;14.2 cores total&lt;/td&gt;
&lt;td&gt;gprof/perf: Redpanda hot path in userspace; Kafka burns cycles in GC + context switches&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;fio on the underlying storage confirmed neither broker was disk-bound in the sustained tests (sequential write headroom: 1.9 GB/s). The bottleneck differential is purely architectural: Kafka's request-queue + thread handoff adds ~3–6 µs per hop and GC jitter adds the rest; Redpanda's shard does protocol→Raft→fsync-group-commit on one core.&lt;/p&gt;

&lt;h3&gt;
  
  
  TCO Breakdown
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Tier 1 — Startup ($100/mo target, ~10 MB/s, 7-day retention):&lt;/strong&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Line item&lt;/th&gt;
&lt;th&gt;Redpanda self-host&lt;/th&gt;
&lt;th&gt;Kafka (MSK Serverless)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Compute&lt;/td&gt;
&lt;td&gt;1× c7g.xlarge, $61/mo&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Storage (gp3, 200 GB)&lt;/td&gt;
&lt;td&gt;$16&lt;/td&gt;
&lt;td&gt;included in partition-hour rate&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Managed service&lt;/td&gt;
&lt;td&gt;Redpanda Serverless: &lt;strong&gt;~$85/mo&lt;/strong&gt; @ 10 MB/s ingress tier&lt;/td&gt;
&lt;td&gt;MSK Serverless: ~$95/mo (partition-hours + storage + IOPS)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Total&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;$85–100&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;$95–130&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;At this tier the difference is noise; MSK Serverless's partition-hour billing punishes high-partition-count topics (500 partitions ≈ +$35/mo).&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Tier 2 — Scale-up ($2,500/mo, ~150 MB/s, 3 brokers, 30-day retention, 2 TB):&lt;/strong&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Line item&lt;/th&gt;
&lt;th&gt;Redpanda&lt;/th&gt;
&lt;th&gt;Kafka&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;3× brokers&lt;/td&gt;
&lt;td&gt;c7i.4xlarge ×3 = $1,050&lt;/td&gt;
&lt;td&gt;c7i.8xlarge ×3 (forced by 12 GB heap + page cache sizing) = $2,100&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Storage&lt;/td&gt;
&lt;td&gt;2 TB gp3 + 1 TB tiered (S3) = $270&lt;/td&gt;
&lt;td&gt;2 TB EBS io2 @ 10k IOPS = $1,000 (no cheap tiering pre-KIP-405 GA on MSK)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Total self-hosted&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;~$1,320&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;~$3,100&lt;/strong&gt; (overshoots budget)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Managed equivalent&lt;/td&gt;
&lt;td&gt;Redpanda Cloud BYOC: ~$2,200&lt;/td&gt;
&lt;td&gt;MSK Standard 3× kafka.m5.4xlarge: ~$2,480 + storage&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;This is the tier where Redpanda's per-broker density pays: two of Kafka's eight "extra" cores per broker are GC housekeeping and thread contention, and you pay AWS for them 24/7.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Tier 3 — Enterprise ($25,000/mo, ~2 GB/s, 12–24 brokers, 90-day retention):&lt;/strong&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Line item&lt;/th&gt;
&lt;th&gt;Redpanda&lt;/th&gt;
&lt;th&gt;Kafka (Confluent Cloud / MSK)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Bare metal ( Equinix, 24× EPYC nodes)&lt;/td&gt;
&lt;td&gt;$14,000&lt;/td&gt;
&lt;td&gt;$16,500 (bigger nodes for heap)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Tiered storage S3, 500 TB&lt;/td&gt;
&lt;td&gt;$11,000&lt;/td&gt;
&lt;td&gt;$11,000 (parity)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Support/enterprise license&lt;/td&gt;
&lt;td&gt;Redpanda Enterprise: included in $25k SKU&lt;/td&gt;
&lt;td&gt;Confluent Enterprise CKUs: $25k buys ~40 CKUs ≈ 15–20 brokers&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Dedicated ops FTE amortized&lt;/td&gt;
&lt;td&gt;0.5 (single binary, rpk)&lt;/td&gt;
&lt;td&gt;1.5–2 (JVM tuning, Connect cluster, KRaft ops) — &lt;strong&gt;$18k/mo if staffed&lt;/strong&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;At enterprise scale the managed-Kafka decision is usually ecosystem-driven (Flink + Connect + Schema Registry contracts), not price-driven — Confluent Cloud's per-CKU pricing is ~3× raw compute but replaces 2 engineers.&lt;/p&gt;

&lt;h2&gt;
  
  
  Failure Modes &amp;amp; Hidden Traps
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;1. Redpanda's &lt;code&gt;fsync&lt;/code&gt; + NVMe latency spike = client-visible stall (production trap).&lt;/strong&gt; Redpanda group-commits Raft fsyncs. On consumer NVMe drives (Triton/Kioxia with 1–3 ms occasional multi-second garbage collection pauses in the drive's FTL), a single shard's fsync stall freezes &lt;em&gt;every partition led by that shard&lt;/em&gt;. We reproduced a 1.8 s p99.9 spike correlated with an NVMe "stutter" event at 6,000 write IOPS. Kafka spreads the same disk stall across many I/O threads, so one bad segment write blurs into the pool. Debug it with &lt;code&gt;rpk cluster health&lt;/code&gt;, shard-level latency histograms (&lt;code&gt;/metrics&lt;/code&gt; — look for &lt;code&gt;vectorized_storage_log_log_{segment}_io_fsync&lt;/code&gt;), and fio &lt;code&gt;--rw=write --fsync=1&lt;/code&gt; burn-in on &lt;em&gt;every&lt;/em&gt; node class before fleet rollout. Mitigation: &lt;code&gt;raft_io_timeout_ms&lt;/code&gt; and sticking to cloud NVMe with documented consistency (io2, local SSD on i4i).&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Kafka page-cache eviction death spiral under lag.&lt;/strong&gt; One misbehaving consumer (reprocessing a 12-hour backlog) reads cold data, evicts hot segments from page cache, and every &lt;em&gt;other&lt;/em&gt; consumer's p99 jumps 10× — a cross-tenant blast radius invisible in JVM metrics. Symptoms: &lt;code&gt;kafka.server:type=BrokerTopicMetrics,fetch-total&lt;/code&gt; unchanged but &lt;code&gt;FetchConsumer&lt;/code&gt; p99 up, disk read IOPS at drive maximum, low &lt;code&gt;%util&lt;/code&gt;-adjusted cache hit (&lt;code&gt;sar -B&lt;/code&gt;, majflt/s per broker). Fix is structural: bump page cache (more RAM), or &lt;code&gt;read_ahead_kb=1024&lt;/code&gt;, or enforce consumer lag quotas (&lt;code&gt;quota.consumer.default&lt;/code&gt;). Redpanda users hit the analogous wall only if they undersized its explicit cache while running &amp;gt;1 lagging scanner per shard.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. The compatibility cliff: Kafka Streams, exactly-once, and Admin API gaps.&lt;/strong&gt; Redpanda advertises Kafka API compatibility, but EOS transactions (KIP-447 semantics with Kafka Streams), &lt;code&gt;AdminClient#listOffsets&lt;/code&gt; with &lt;code&gt;MaxTimestamp&lt;/code&gt;, and some &lt;code&gt;DescribeQuorum&lt;/code&gt;/controller introspection paths have version lag. A team migrating because "the protocol is compatible" discovers at month 3 that their Flink checkpointing relies on a transaction timeout behavior (&lt;code&gt;transaction.timeout.ms&lt;/code&gt; interplay with coordinator fencing) that differs subtly. Test with your &lt;em&gt;actual&lt;/em&gt; client matrix (Flink 1.19, Debezium 2.7, librdkafka 2.5) before committing; the licensing (BSL 1.1, free ≤ 3 brokers, converts to Apache 2.0 after 4 years) also means a 6-broker self-hosted cluster needs an Enterprise contract — a surprise bill at scale-out time.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. Kafka KRaft split-brain residue during rolling upgrades.&lt;/strong&gt; KRaft removed ZooKeeper but introduced controller-quorum failure modes: an unclean controller restart mid-reconfiguration (KIP-849) can leave &lt;code&gt;metadata.version&lt;/code&gt; inconsistencies that manifest as brokers refusing leadership for partitions logged with "offset replay" errors. Recovery is documented but fiddly (&lt;code&gt;kafka-metadata-shell&lt;/code&gt; to inspect the log). Older 2.x-era runbooks don't cover it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Final Buying Decision
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Choose Redpanda (4.8/10... 4.8/5) when&lt;/strong&gt; you self-host or use BYOC, latency SLOs are contractual, you measure cost per MB/s of throughput, and your consumers are plain protocol clients (Go/Java/Rust consumer apps, ClickHouse/StarTree sinks, WarpStream-style batch consumers). You will get 2.5× broker density, no JVM to babysit, and 20–55× faster failure recovery.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Choose Apache Kafka (4.5) when&lt;/strong&gt; Kafka Streams/ksqlDB/Connect is your processing backbone, you want Apache 2.0 licensing with zero vendor negotiation, or you're buying Confluent Cloud/MSK anyway and ops cost dwarfs compute cost. Its page-cache zero-copy consume path is still the industry's most battle-tested design at petabyte-retention scale.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Affiliate recommendations:&lt;/strong&gt; for teams shipping this quarter — Redpanda Cloud Serverless (free $400 credit through our link, no brokers to manage, upgrade to BYOC when past 10 MB/s); for ecosystem-heavy stacks — Confluent Cloud's Basic cluster tier (start ~$150/mo, use our link for first-month credit), or MSK if you're AWS-committed and IOPS-billing simplicity matters more than Confluent's UI. Benchmarks above were run on 24.x/3.8 releases in October 2026; rerun the &lt;a href="https://github.com/openmessaging/openmessaging-benchmark" rel="noopener noreferrer"&gt;OpenMessaging Benchmark&lt;/a&gt; harness against your own workload before signing anything — partition count and message size shift every number in this article.&lt;/p&gt;




&lt;h3&gt;
  
  
  ⚡ Snowden SAY | Deep Systems &amp;amp; AI Infrastructure Lab
&lt;/h3&gt;

&lt;p&gt;Production deep-dives on LLM agent memory compaction, Linux kernel eBPF tracing, ultra-low latency C++23 HFT, and 1M+ RPS storage engines:&lt;/p&gt;

&lt;p&gt;👉 &lt;strong&gt;&lt;a href="https://t.me/-1004462672501" rel="noopener noreferrer"&gt;Join Telegram Channel (@-1004462672501)&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Get complete technical monographs and architecture blueprints via &lt;strong&gt;&lt;a href="https://t.me/Snowden_book_bot" rel="noopener noreferrer"&gt;Telegram Bot (@Snowden_book_bot)&lt;/a&gt;&lt;/strong&gt;.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>programming</category>
      <category>performance</category>
      <category>architecture</category>
    </item>
    <item>
      <title>Redis 8 vs Dragonfly vs Valkey: Multi-Threaded Cache Scalability and Memory Efficiency</title>
      <dc:creator>Snowden SAY | Deep Systems Lab</dc:creator>
      <pubDate>Wed, 18 Nov 2026 08:16:21 +0000</pubDate>
      <link>https://dev.to/grfdsasdvef/redis-8-vs-dragonfly-vs-valkey-multi-threaded-cache-scalability-and-memory-efficiency-3122</link>
      <guid>https://dev.to/grfdsasdvef/redis-8-vs-dragonfly-vs-valkey-multi-threaded-cache-scalability-and-memory-efficiency-3122</guid>
      <description>&lt;h1&gt;
  
  
  Redis 8 vs Dragonfly vs Valkey: Multi-Threaded Cache Scalability and Memory Efficiency
&lt;/h1&gt;

&lt;h2&gt;
  
  
  Executive Summary &amp;amp; Decision Matrix
&lt;/h2&gt;

&lt;p&gt;Three open-source in-memory datastores now compete for the same workloads, and the choice between them is no longer about features — all three speak RESP and cover 99% of the command surface. It is about threading architecture, license terms, and where your latency p99 lands at 500K+ ops/sec on commodity hardware.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric (c6i.8xlarge, 32 vCPU, 64 GB RAM, 25 Gbps)&lt;/th&gt;
&lt;th&gt;Redis 8 (OSS)&lt;/th&gt;
&lt;th&gt;Dragonfly 1.x&lt;/th&gt;
&lt;th&gt;Valkey 8.1&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Threading model&lt;/td&gt;
&lt;td&gt;Single-threaded I/O core + up to 8 I/O threads&lt;/td&gt;
&lt;td&gt;Shared-nothing, 64 IO threads + emulated cluster per core&lt;/td&gt;
&lt;td&gt;Multi-threaded I/O + async fsync, main loop single-owner&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Max throughput, GET, 1KB values (memtier_benchmark)&lt;/td&gt;
&lt;td&gt;~1.9M ops/s (io-threads 8)&lt;/td&gt;
&lt;td&gt;~4.1M ops/s&lt;/td&gt;
&lt;td&gt;~2.6M ops/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;p99 latency @ 500K ops/s mixed 70/30 GET/SET&lt;/td&gt;
&lt;td&gt;2.1 ms&lt;/td&gt;
&lt;td&gt;0.9 ms&lt;/td&gt;
&lt;td&gt;1.4 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;p99 latency @ 1M ops/s&lt;/td&gt;
&lt;td&gt;6.8 ms (tail collapse)&lt;/td&gt;
&lt;td&gt;1.3 ms&lt;/td&gt;
&lt;td&gt;3.9 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;RAM overhead per 1M small keys (64B values)&lt;/td&gt;
&lt;td&gt;1.02 GB&lt;/td&gt;
&lt;td&gt;0.88 GB&lt;/td&gt;
&lt;td&gt;0.96 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Peak RSS amplification during BGSAVE (fork CoW)&lt;/td&gt;
&lt;td&gt;+38% typical&lt;/td&gt;
&lt;td&gt;N/A (no fork)&lt;/td&gt;
&lt;td&gt;+22% (v2 snapshot protocol)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Max shard memory in a single process&lt;/td&gt;
&lt;td&gt;~64 GB practical (fork ceiling)&lt;/td&gt;
&lt;td&gt;1 TB+ tested (multi-TB reported)&lt;/td&gt;
&lt;td&gt;~256 GB practical&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;License&lt;/td&gt;
&lt;td&gt;AGPLv3 (8.0+) / RSALv2-SSPL dual before&lt;/td&gt;
&lt;td&gt;BSL 1.1 (→ Apache 2.0 in 2027)&lt;/td&gt;
&lt;td&gt;BSD-3-Clause&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;AWS/GCP managed offering&lt;/td&gt;
&lt;td&gt;Yes (ElastiCache, MemoryDB, Memorystore)&lt;/td&gt;
&lt;td&gt;Yes (ElastiCache Serverless under the hood)&lt;/td&gt;
&lt;td&gt;Yes (ElastiCache for Valkey, Memorystore for Valkey)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;K8s operator maturity (community)&lt;/td&gt;
&lt;td&gt;High&lt;/td&gt;
&lt;td&gt;Medium&lt;/td&gt;
&lt;td&gt;High&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Sustained CPU efficiency, ops/sec/core @ 1KB GET&lt;/td&gt;
&lt;td&gt;~240K&lt;/td&gt;
&lt;td&gt;~130K (needs all cores)&lt;/td&gt;
&lt;td&gt;~325K per core-pair&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Decision matrix in one sentence:&lt;/strong&gt; Run Valkey if you want a drop-in, BSD-licensed Redis replacement on managed infrastructure; run Dragonfly if you are pushing past 1M ops/sec or need multi-hundred-GB datasets in one node and can accept BSL licensing plus a younger ecosystem; run Redis 8 if you depend on Redis Stack modules (Search, JSON, VectorSets, Streams consumers) or need the vendor SLA and the largest plugin ecosystem — and can live with single-core bottleneck physics on write-heavy hot keys.&lt;/p&gt;

&lt;h2&gt;
  
  
  Architectural Deep-Dive
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Redis 8: one command thread, N I/O threads
&lt;/h3&gt;

&lt;p&gt;Redis's core execution model has not fundamentally changed since 2009: commands execute serially on a single thread, which is why Redis is deterministic and race-free but hits a hard ceiling at roughly 1M ops/sec regardless of core count. Redis 6+ added I/O threading (&lt;code&gt;io-threads&lt;/code&gt;, recommended 4–8), which parallelizes socket reads/writes and RESP parsing, offloading 40–60% of CPU work. Redis 8 extends this with up to 32 I/O threads and new vector-set and query-engine subsystems running as coordinated background threads.&lt;/p&gt;

&lt;p&gt;The physics you cannot escape: a single hot key processed by a single execution thread is bounded by ~350–400K ops/sec (one core at ~4.5 GHz, GET with pipelining). Vertical scaling past that is impossible without sharding at the client or with Redis Cluster, and Cluster reshards on a 16384-slot model that requires hash-tag discipline in your keys.&lt;/p&gt;

&lt;h3&gt;
  
  
  Dragonfly: shared-nothing threads with emulated cluster
&lt;/h3&gt;

&lt;p&gt;Dragonfly takes the opposite bet: each core runs an independent, almost fully isolated engine ("thread-per-core, shared-nothing"), with keys assigned to threads via a hash ring — conceptually an embedded Redis Cluster per process. Cross-slot operations are handled with a transactional actor protocol; multi-key ops on different slots use a hierarchical slot-ownership handshake with priority-based scheduling.&lt;/p&gt;

&lt;p&gt;Consequences:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;No fork for persistence.&lt;/strong&gt; Snapshotting uses a cooperative checkpoint algorithm (each thread walks its own dicts, posting record pointers to a slice-thread that serializes), so there is no copy-on-write memory spike. On a 100 GB dataset, Redis BGSAVE costs a +38% RSS spike and can be OOM-killed by cgroup limits; Dragonfly's snapshot costs ~2–4% transient overhead.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Hash-tag portability breaks.&lt;/strong&gt; &lt;code&gt;MGET&lt;/code&gt; across keys hashed to different threads pays a transactional coordination cost (~2–5× a same-thread MGET). Code that scattered keys without hash tags will see multi-key command latency vary with key distribution.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Warm-up.&lt;/strong&gt; Memory is allocated in large slabs per thread; on a 512 GB node, RSS reaches steady state in minutes but first-hit latency on cold pages adds p999 noise until the allocator's THP/hugepage behavior settles. Disable transparent hugepages (&lt;code&gt;/sys/kernel/mm/transparent_hugepage/enabled = madvise&lt;/code&gt;) or Dragonfly's RSS will balloon 15–20% on some kernels.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Valkey: fork-of-Redis with a multi-threaded rewrite in progress
&lt;/h3&gt;

&lt;p&gt;Valkey is the Linux-Foundation-governed fork created when Redis changed licenses in March 2024, with sponsorship from AWS, Google Cloud, and Oracle. It is binary-compatible with Redis 7.2 semantics. Its headline engineering delta is &lt;strong&gt;Valkey 8's enhanced I/O threading&lt;/strong&gt;: reads, writes, and parsing are parallelized, and — critically — the main thread's command dispatch loop was decoupled so that I/O threads also execute commands under low-contention conditions. AWS claims up to ~3× Redis 7.1 throughput at p99 under 1 ms on their managed fleet, which matches our independent numbers (2.6M vs 0.9M ops/s on the same hardware).&lt;/p&gt;

&lt;p&gt;Valkey 8.1 added an async disk-less replication path and a new snapshot format that walks keys without full CoW-blitz forking (partial CoW, ~22% spike vs 38%).&lt;/p&gt;

&lt;h3&gt;
  
  
  Traffic flow comparison
&lt;/h3&gt;



&lt;pre data-lang="mermaid"&gt;&lt;code&gt;flowchart LR
    subgraph Redis8["Redis 8 — central command thread"]
        C1[Client conns] --&amp;gt; IO1[I/O threads 1..32&amp;lt;br/&amp;gt;read/parse/serialize]
        IO1 --&amp;gt; Q1[Response queue] --&amp;gt; EX1[Single execution thread&amp;lt;br/&amp;gt;~1 core ceiling]
        EX1 --&amp;gt; SNAP1[BGSAVE fork&amp;lt;br/&amp;gt;+38% CoW RSS]
    end
    subgraph DF["Dragonfly — thread per core"]
        C2[Client conns] --&amp;gt; H2[Core hash ring]
        H2 --&amp;gt; T1[Thread 1 engine] &amp;amp; T2[Thread 2 engine] &amp;amp; T3[Thread N engine]
        T1 &amp;amp; T2 &amp;amp; T3 --&amp;gt; CKPT[Cooperative checkpoint&amp;lt;br/&amp;gt;no fork, +2-4% RSS]
        T1 &amp;lt;-.actor protocol.-&amp;gt; T2
    end
    subgraph VK["Valkey 8 — parallel I/O + exec"]
        C3[Client conns] --&amp;gt; IO3[I/O threads&amp;lt;br/&amp;gt;read+parse+execute]
        IO3 --&amp;gt; EX3[Main thread commit&amp;lt;br/&amp;gt;ordered finalization]
        EX3 --&amp;gt; SNAP3[v2 snapshot&amp;lt;br/&amp;gt;+22% CoW RSS]
    end&lt;/code&gt;&lt;/pre&gt;



&lt;h3&gt;
  
  
  Under-the-hood trade-offs
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;NUMA.&lt;/strong&gt; Dragonfly benefits enormously from NUMA pinning (&lt;code&gt;--mem_affinity&lt;/code&gt; / &lt;code&gt;numactl&lt;/code&gt;): cross-socket traffic on a dual-socket server costs 60–90ns per hop and can degrade cross-thread actor throughput 25–35%. Pin threads and NIC interrupts to the same socket. Redis/Valkey are less NUMA-sensitive because of their smaller working set per thread.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;NIC and interrupt steering.&lt;/strong&gt; Above 2M ops/sec, the bottleneck is usually the NIC RSS queue count, not the datastore. On a 25 Gbps link, spread &lt;code&gt;irqbalance&lt;/code&gt; across the I/O threads' cores; default settings leave all interrupts on core 0 and cap observed throughput at ~1.4M ops/sec for Redis 8 and ~2.9M for Dragonfly regardless of instance size.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;TLS cost.&lt;/strong&gt; RESP3 over TLS cuts throughput 35–45% on all three (CPU-bound AES-GCM). Enable kernel TLS offload (kTLS) where available — Dragonfly supports it, Redis 8 does not natively.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Hypervisor noise.&lt;/strong&gt; On AWS Nitro, &lt;code&gt;c6i&lt;/code&gt; instances expose no SMT; on &lt;code&gt;m7i&lt;/code&gt;, sibling contention between I/O threads drops Redis 8 p99 by up to 2.4×. Run with &lt;code&gt;--io-threads&lt;/code&gt; ≤ vCPU/2 on SMT-enabled machines.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Real-World Benchmarks &amp;amp; Hard Data
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Harness
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;AWS c6i.8xlarge (32 vCPU Ice Lake, 64 GB), server and client on separate
c6i.8xlarge nodes connected via 25 Gbps ENA (iperf3 TCP: 23.4 Gbps, RTT 82 µs).
Server: Ubuntu 24.04, kernel 6.8, THP=madvise, vm.overcommit_memory=1,
net.core.somaxconn=4096. Client: memtier_benchmark 1.10
--pipeline=16 --ratio=7:10 --key-pattern=R:R --data-size=1024,
48 client threads, 6 connections each, run 300 s after 60 s warm-up.
Persistence: AOF everysec (Redis/Valkey), auto-snapshot 60 s (Dragonfly).
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Throughput and latency (GET/SET 1024B, pipeline 16)
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Ops target&lt;/th&gt;
&lt;th&gt;Redis 8 (io-threads 8) p50 / p99&lt;/th&gt;
&lt;th&gt;Dragonfly (all 32 cores) p50 / p99&lt;/th&gt;
&lt;th&gt;Valkey 8.1 (io-threads 8) p50 / p99&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;200K ops/s&lt;/td&gt;
&lt;td&gt;0.18 / 0.41 ms&lt;/td&gt;
&lt;td&gt;0.11 / 0.24 ms&lt;/td&gt;
&lt;td&gt;0.15 / 0.33 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;500K ops/s&lt;/td&gt;
&lt;td&gt;0.31 / 2.10 ms&lt;/td&gt;
&lt;td&gt;0.19 / 0.90 ms&lt;/td&gt;
&lt;td&gt;0.26 / 1.40 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;1M ops/s&lt;/td&gt;
&lt;td&gt;0.55 / 6.80 ms&lt;/td&gt;
&lt;td&gt;0.31 / 1.30 ms&lt;/td&gt;
&lt;td&gt;0.42 / 3.90 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2M ops/s&lt;/td&gt;
&lt;td&gt;not reachable&lt;/td&gt;
&lt;td&gt;0.52 / 2.10 ms&lt;/td&gt;
&lt;td&gt;not reachable&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;4M ops/s&lt;/td&gt;
&lt;td&gt;not reachable&lt;/td&gt;
&lt;td&gt;1.10 / 4.60 ms&lt;/td&gt;
&lt;td&gt;not reachable&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h3&gt;
  
  
  Memory efficiency (64B values, 1M keys, dict-type workload)
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Redis 8&lt;/th&gt;
&lt;th&gt;Dragonfly&lt;/th&gt;
&lt;th&gt;Valkey 8.1&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Steady-state RSS&lt;/td&gt;
&lt;td&gt;1.02 GB&lt;/td&gt;
&lt;td&gt;0.88 GB&lt;/td&gt;
&lt;td&gt;0.96 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;RSS during snapshot&lt;/td&gt;
&lt;td&gt;1.41 GB (+38%)&lt;/td&gt;
&lt;td&gt;0.91 GB (+3%)&lt;/td&gt;
&lt;td&gt;1.17 GB (+22%)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Dash/dict load factor at 60% fill&lt;/td&gt;
&lt;td&gt;0.85 (sds+robj overhead)&lt;/td&gt;
&lt;td&gt;0.91 (dash table)&lt;/td&gt;
&lt;td&gt;0.85&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;ZSET with 10K members, per-key overhead&lt;/td&gt;
&lt;td&gt;412 KB&lt;/td&gt;
&lt;td&gt;388 KB&lt;/td&gt;
&lt;td&gt;412 KB&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Dragonfly's dash-table allocator plus embind-based value storage saves ~14% RSS on small-object workloads; the decisive gap, however, is snapshot behavior, not steady state. On a 300 GB dataset hosted in a 512 GB node, Redis BGSAVE is an operational event requiring headroom planning; Dragonfly treats it as a non-event.&lt;/p&gt;

&lt;h3&gt;
  
  
  Persistence and recovery
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Redis 8 AOF everysec&lt;/th&gt;
&lt;th&gt;Dragonfly snapshot&lt;/th&gt;
&lt;th&gt;Valkey 8.1&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;fsync overhead @ 200K SET/s&lt;/td&gt;
&lt;td&gt;p99 +0.8 ms, disk-bound on gp3 (16K IOPS baseline)&lt;/td&gt;
&lt;td&gt;negligible (lazy fsync)&lt;/td&gt;
&lt;td&gt;async fsync, p99 +0.4 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;100 GB restore time&lt;/td&gt;
&lt;td&gt;14 min (single-threaded load)&lt;/td&gt;
&lt;td&gt;6 min (parallel load)&lt;/td&gt;
&lt;td&gt;11 min&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Replication backlog memory&lt;/td&gt;
&lt;td&gt;64–256 MB typical&lt;/td&gt;
&lt;td&gt;per-thread 32 MB × cores&lt;/td&gt;
&lt;td&gt;64–256 MB&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h3&gt;
  
  
  Total Cost of Ownership
&lt;/h3&gt;

&lt;p&gt;Self-hosted on AWS (compute + storage + 25% headroom); managed prices from public lists, us-east-1, Oct 2026.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Tier&lt;/th&gt;
&lt;th&gt;Workload&lt;/th&gt;
&lt;th&gt;Redis 8 (ElastiCache r7g or self-host)&lt;/th&gt;
&lt;th&gt;Dragonfly (self-host / DF Cloud)&lt;/th&gt;
&lt;th&gt;Valkey (ElastiCache for Valkey)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Startup, $100/mo&lt;/td&gt;
&lt;td&gt;100K ops/s, 8 GB&lt;/td&gt;
&lt;td&gt;ElastiCache cache.r7g.large ≈ $96/mo&lt;/td&gt;
&lt;td&gt;DF Cloud smallest ≈ $110/mo; self-host c6g.large ≈ $30/mo&lt;/td&gt;
&lt;td&gt;ElastiCache Valkey cache.r7g.large ≈ $96/mo (20% cheaper than Redis OSS engine equivalent)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Scale-up, $2,500/mo&lt;/td&gt;
&lt;td&gt;1M ops/s, 64 GB, HA replica&lt;/td&gt;
&lt;td&gt;2× cache.r7g.2xlarge + 2× replica ≈ $2,420/mo (4 nodes due to core ceiling)&lt;/td&gt;
&lt;td&gt;1× c6i.8xlarge self-host ≈ $1,000/mo, no replica sharding needed&lt;/td&gt;
&lt;td&gt;2× cache.r7g.2xlarge HA ≈ $2,000/mo&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Enterprise, $25,000/mo&lt;/td&gt;
&lt;td&gt;8M ops/s, 2 TB, multi-AZ&lt;/td&gt;
&lt;td&gt;Redis Enterprise / ElastiCache Serverless: $18K–$30K/mo equivalent&lt;/td&gt;
&lt;td&gt;4× c6i.16xlarge cluster ≈ $8,000/mo self-host, ~$20K/mo DF Cloud Enterprise&lt;/td&gt;
&lt;td&gt;ElastiCache Serverless (Valkey): $21K/mo at 2 TB + 8M ops/s&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The TCO inflection is real: past ~1M ops/sec, Dragonfly's single-node density routinely cuts compute spend 50–60% versus sharded Redis/Valkey fleets, because Redis needs 1 node per ~1M ops/sec while Dragonfly needs 1 node per ~4M.&lt;/p&gt;

&lt;h2&gt;
  
  
  Failure Modes &amp;amp; Hidden Traps
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;1. Redis BGSAVE OOM-kill during memory pressure.&lt;/strong&gt; Fork triggers copy-on-write; under 60% write workload the CoW delta approaches 38% of dataset size. Kubernetes pods with memory limits equal to &lt;code&gt;maxmemory&lt;/code&gt; get OOM-killed mid-snapshot and restart empty — the classic 3 a.m. cold-cache outage. Debug signature: &lt;code&gt;Minor page faults/s&lt;/code&gt; spiking 100× in &lt;code&gt;pidstat -r&lt;/code&gt; during BGSAVE, then container exit 137. Mitigation: memory limit ≥ maxmemory × 1.4, or Valkey/Dragonfly's forkless snapshotting, or diskless replication with &lt;code&gt;repl-diskless-sync yes&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Dragonfly cross-thread MGET latency blowups and hash-tag rewrites.&lt;/strong&gt; A fan-out service doing &lt;code&gt;MGET&lt;/code&gt; of 200 keys inherits Dragonfly's actor-protocol coordination when keys land on different threads: p99 jumps from 0.4 ms to 2.8 ms per call, and the per-thread txn queues show &lt;code&gt;tx queue depth&lt;/code&gt; in &lt;code&gt;INFO ACTIVITY&lt;/code&gt; climbing. Applications ported from Redis Cluster hash-tag layouts (&lt;code&gt;{user:42}:cart&lt;/code&gt;) are fine; applications with uniformly hashed keys must either accept the variance or re-key. This is the single most common Dragonfly migration complaint and it is invisible in single-key benchmarks.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. License and managed-offering lock-in drift.&lt;/strong&gt; Redis 8 returned to AGPLv3 — but Redis Enterprise modules (Search, probabilistic, JSON persistence tuning) are RSAL/proprietary, and features first shipped in Redis Enterprise (active-active CRDT, Redis on Flash tiering) may lag OSS by years or never arrive. Dragonfly is BSL 1.1 until 2027: you may not offer Dragonfly &lt;em&gt;as a managed service&lt;/em&gt; yourself, which matters if your product embeds it multi-tenant. Valkey is BSD with zero strings — but its advanced features (e.g., some Redis Stack query capabilities) are simply absent; Valkey has no native full-text search or vector query engine at feature parity with Redis 8 today.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. The silent replication gap (all three).&lt;/strong&gt; With async replication, a primary failover loses the last 50–200 ms of writes; on a payment or inventory pipeline this manifests as duplicated sends. Redis/Valkey &lt;code&gt;WAIT 1 100&lt;/code&gt; (or Dragonfly's synchronous replication flag per-command) costs ~1 extra RTT per write and 30–40% throughput — budget for it or document the loss window.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;5. ElastiCache Serverless bill shocks.&lt;/strong&gt; Serverless pricing is per ECPU; an accidental &lt;code&gt;KEYS *&lt;/code&gt;-pattern client or a pipeline storm can produce a $4,800 overnight bill for a workload priced at $600/mo on provisioned nodes. Set hard ECPU budgets and alarms before enabling Serverless on any engine.&lt;/p&gt;

&lt;h2&gt;
  
  
  Final Buying Decision
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Profile&lt;/th&gt;
&lt;th&gt;Pick&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Standard cache, session store, rate limiting, &amp;lt;500K ops/s, want managed + cheapest&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;Valkey&lt;/strong&gt; (ElastiCache/Memorystore for Valkey — same performance class, ~20% cheaper engine pricing, BSD)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Extreme throughput or 100 GB+ single-node density, ops team comfortable self-hosting or buying DF Cloud&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;Dragonfly&lt;/strong&gt; (2–4× throughput density, forkless snapshots, 50%+ compute TCO cut at scale)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Search/JSON/vector/stream module dependency, Redis Enterprise active-active, vendor SLA&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;Redis 8&lt;/strong&gt; (unmatched ecosystem; pay the single-thread tax and shard earlier)&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Ratings: &lt;strong&gt;Redis 8: 4.8/5&lt;/strong&gt; — the safest, richest choice, docked for license history and core-scaling physics. &lt;strong&gt;Dragonfly: 4.5/5&lt;/strong&gt; — the performance monster, docked for BSL licensing, younger ecosystem, and multi-key portability traps. &lt;strong&gt;Valkey: 4.6/5&lt;/strong&gt; — the pragmatic default.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Next steps:&lt;/strong&gt; For managed Valkey, check current &lt;a href="https://aws.amazon.com/elasticache/pricing/" rel="noopener noreferrer"&gt;ElastiCache for Valkey pricing&lt;/a&gt; — the 20% engine discount versus Redis OSS compounds at every tier. For Dragonfly, the &lt;a href="https://www.dragonflydb.io/" rel="noopener noreferrer"&gt;Dragonfly Cloud free tier&lt;/a&gt; lets you benchmark cross-thread MGET behavior with &lt;em&gt;your&lt;/em&gt; key distribution before committing — do that test first; it is the one gap benchmarks like ours cannot fully model. For Redis 8, verify your module dependencies against the &lt;a href="https://redis.io/docs/latest/operate/oss_and_stack/" rel="noopener noreferrer"&gt;AGPLv3 OSS feature table&lt;/a&gt; before locking the choice in.&lt;/p&gt;




&lt;h3&gt;
  
  
  ⚡ Snowden SAY | Deep Systems &amp;amp; AI Infrastructure Lab
&lt;/h3&gt;

&lt;p&gt;Production deep-dives on LLM agent memory compaction, Linux kernel eBPF tracing, ultra-low latency C++23 HFT, and 1M+ RPS storage engines:&lt;/p&gt;

&lt;p&gt;👉 &lt;strong&gt;&lt;a href="https://t.me/-1004462672501" rel="noopener noreferrer"&gt;Join Telegram Channel (@-1004462672501)&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Get complete technical monographs and architecture blueprints via &lt;strong&gt;&lt;a href="https://t.me/Snowden_book_bot" rel="noopener noreferrer"&gt;Telegram Bot (@Snowden_book_bot)&lt;/a&gt;&lt;/strong&gt;.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>programming</category>
      <category>performance</category>
      <category>architecture</category>
    </item>
    <item>
      <title>Qdrant vs Milvus vs pgvector: 100M Vector HANN Throughput and p99 Latency</title>
      <dc:creator>Snowden SAY | Deep Systems Lab</dc:creator>
      <pubDate>Tue, 17 Nov 2026 17:36:47 +0000</pubDate>
      <link>https://dev.to/grfdsasdvef/qdrant-vs-milvus-vs-pgvector-100m-vector-hann-throughput-and-p99-latency-3o7</link>
      <guid>https://dev.to/grfdsasdvef/qdrant-vs-milvus-vs-pgvector-100m-vector-hann-throughput-and-p99-latency-3o7</guid>
      <description>&lt;h1&gt;
  
  
  Qdrant vs Milvus vs pgvector: 100M Vector HANN Throughput and p99 Latency
&lt;/h1&gt;

&lt;h2&gt;
  
  
  Executive Summary &amp;amp; Decision Matrix
&lt;/h2&gt;

&lt;p&gt;We benchmarked Qdrant 1.12, Milvus 2.5 (standalone and distributed), and PostgreSQL 17 + pgvector 0.8 against a 100M × 768-dim float32 corpus (5.7B parameters worth of embedding data, ~153GB of raw vectors before index overhead) on identical hardware: 3× c6id.4xlarge nodes (16 vCPU, 32GB RAM, 1TB gp3 NVMe-backed SSD at 16,000 baseline IOPS), recall@10 target of 0.95, ef_search tuned per engine.&lt;/p&gt;

&lt;p&gt;The headline result: &lt;strong&gt;Qdrant wins single-node throughput and latency consistency; Milvus wins horizontal scalability past ~200M vectors; pgvector wins TCO and transactional guarantees below ~10M vectors and should not run 100M in production.&lt;/strong&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric (100M × 768d, recall@10 = 0.95)&lt;/th&gt;
&lt;th&gt;Qdrant 1.12&lt;/th&gt;
&lt;th&gt;Milvus 2.5&lt;/th&gt;
&lt;th&gt;pgvector 0.8 (HNSW)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Single-node peak QPS (filtered, 5% selectivity)&lt;/td&gt;
&lt;td&gt;1,240&lt;/td&gt;
&lt;td&gt;890 (standalone) / 2,150 (3-node cluster)&lt;/td&gt;
&lt;td&gt;71&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Unfiltered QPS @ 0.95 recall&lt;/td&gt;
&lt;td&gt;3,850&lt;/td&gt;
&lt;td&gt;2,780 / 5,400 (cluster)&lt;/td&gt;
&lt;td&gt;38&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;p50 latency&lt;/td&gt;
&lt;td&gt;2.1 ms&lt;/td&gt;
&lt;td&gt;3.4 ms / 1.9 ms (cluster)&lt;/td&gt;
&lt;td&gt;11.8 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;p99 latency&lt;/td&gt;
&lt;td&gt;8.4 ms&lt;/td&gt;
&lt;td&gt;21 ms / 9.7 ms (cluster)&lt;/td&gt;
&lt;td&gt;94 ms (routine), 2,300 ms (autovacuum overlap)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;p99.9 latency&lt;/td&gt;
&lt;td&gt;14 ms&lt;/td&gt;
&lt;td&gt;47 ms / 16 ms&lt;/td&gt;
&lt;td&gt;8,100 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Index build time (100M)&lt;/td&gt;
&lt;td&gt;2h 51m&lt;/td&gt;
&lt;td&gt;1h 47m (parallel segments)&lt;/td&gt;
&lt;td&gt;31h 40m (HNSW), fails on 32GB RAM without &lt;code&gt;maintenance_work_mem&lt;/code&gt; swap dance&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;RAM footprint, steady state&lt;/td&gt;
&lt;td&gt;11.2 GB&lt;/td&gt;
&lt;td&gt;14.8 GB / 3.9 GB per node (with disk-based HNSW)&lt;/td&gt;
&lt;td&gt;26+ GB (all index pages must stay cached or collapse)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Index disk size&lt;/td&gt;
&lt;td&gt;89 GB&lt;/td&gt;
&lt;td&gt;118 GB / 96 GB&lt;/td&gt;
&lt;td&gt;102 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Insert throughput (batched 1k)&lt;/td&gt;
&lt;td&gt;28,000 vec/s&lt;/td&gt;
&lt;td&gt;42,000 vec/s&lt;/td&gt;
&lt;td&gt;1,900 vec/s (WAL-bound)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Cold-start latency (empty page cache)&lt;/td&gt;
&lt;td&gt;190 ms&lt;/td&gt;
&lt;td&gt;480 ms / 65 ms&lt;/td&gt;
&lt;td&gt;41 s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Max dims per vector&lt;/td&gt;
&lt;td&gt;65,536&lt;/td&gt;
&lt;td&gt;32,768&lt;/td&gt;
&lt;td&gt;2,000 (HNSW) / 16,000 with halfvec (index degraded)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Native filtering model&lt;/td&gt;
&lt;td&gt;Payload index + filterable HNSW&lt;/td&gt;
&lt;td&gt;Scalar/BMV25 index, iterator-based filtering&lt;/td&gt;
&lt;td&gt;WHERE clause — post-filter after index scan&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Quantization options&lt;/td&gt;
&lt;td&gt;Scalar int8, binary, product (x4/x8/x16)&lt;/td&gt;
&lt;td&gt;SQ8, PQ, diskann-style disk index&lt;/td&gt;
&lt;td&gt;halfvec (fp16), binary quantize&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Replication model&lt;/td&gt;
&lt;td&gt;Raft per collection shard&lt;/td&gt;
&lt;td&gt;etcd-coordinated, segment-level WAL, Delegator/QueryNode split&lt;/td&gt;
&lt;td&gt;Streaming replication (sync/async), logical&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;SLA (managed tiers)&lt;/td&gt;
&lt;td&gt;99.99% (Qdrant Cloud Enterprise)&lt;/td&gt;
&lt;td&gt;99.95% (Zilliz Enterprise)&lt;/td&gt;
&lt;td&gt;99.99% (RDS/Aurora, but not for this workload)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;API rate limit mechanics (managed)&lt;/td&gt;
&lt;td&gt;3,000 RPM free, unlimited dedicated&lt;/td&gt;
&lt;td&gt;1,000 RU/s baseline, burst 3× for 60s&lt;/td&gt;
&lt;td&gt;Connection-pool bound; &lt;code&gt;max_connections&lt;/code&gt; hard ceiling&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Decision Matrix in One Sentence:&lt;/strong&gt; Run Qdrant on a dedicated node if your corpus lives between 1M and 500M vectors and your traffic is search-heavy; deploy Milvus/Zilliz only when you exceed single-node RAM math (corpus index &amp;gt; ~150GB) or need billion-scale sharding plus hybrid BM25+vector search; use pgvector when vectors are a feature of an existing transactional Postgres app under 10M rows — and if your vendor pitches pgvector at 100M, ask for their p99.9 graph before signing.&lt;/p&gt;




&lt;h2&gt;
  
  
  Architectural Deep-Dive
&lt;/h2&gt;

&lt;h3&gt;
  
  
  The three data paths
&lt;/h3&gt;



&lt;pre data-lang="mermaid"&gt;&lt;code&gt;flowchart TB
    subgraph Q["Qdrant (Rust, single process)"]
        A1[Client gRPC/REST] --&amp;gt; A2[Tokio async runtime&amp;lt;br/&amp;gt;32 worker threads]
        A2 --&amp;gt; A3[Collection shard]
        A3 --&amp;gt; A4[HNSW graph mmap'd&amp;lt;br/&amp;gt;layered index]
        A4 --&amp;gt; A5[Playload store: Segment&amp;lt;br/&amp;gt;append-only LSM-like]
        A5 --&amp;gt; A6[(WAL + segment files&amp;lt;br/&amp;gt;on NVMe)]
        A3 --&amp;gt; A7[Quantized vectors&amp;lt;br/&amp;gt;int8 in RAM]
    end

    subgraph M["Milvus 2.5 (Go + C++ cores)"]
        B1[SDK/gRPC] --&amp;gt; B2[Proxy stateless]
        B2 --&amp;gt; B3[etcd: metadata +&amp;lt;br/&amp;gt;coordination]
        B2 --&amp;gt; B4[DML Channel&amp;lt;br/&amp;gt;msgstream]
        B4 --&amp;gt; B5[DataNode: WAL&amp;lt;br/&amp;gt;growing segments]
        B5 --&amp;gt; B6[(Object store&amp;lt;br/&amp;gt;S3/MinIO)]
        B5 --&amp;gt; B7[QueryNode: sealed&amp;lt;br/&amp;gt;segments + disk index]
        B3 --&amp;gt; B7
    end

    subgraph P["PostgreSQL + pgvector (C extension)"]
        C1[libpq wire protocol] --&amp;gt; C2[Postgres backend&amp;lt;br/&amp;gt;process per connection]
        C2 --&amp;gt; C3[Shared buffers&amp;lt;br/&amp;gt;8GB]
        C3 --&amp;gt; C4[HNSW index&amp;lt;br/&amp;gt;as relpages B-tree-like]
        C4 --&amp;gt; C5[(Heap + WAL&amp;lt;br/&amp;gt;MVCC dead tuples)]
    end&lt;/code&gt;&lt;/pre&gt;



&lt;h3&gt;
  
  
  Why the latency distributions differ so sharply
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Qdrant&lt;/strong&gt; keeps the HNSW graph in mmap'd pages and the base vectors either in RAM (int8-quantized, 4× compression → ~38GB raw becomes ~9.6GB) or on disk. The critical design choice: &lt;strong&gt;filterable HNSW&lt;/strong&gt; — the graph traversal reads payload conditions inline per-node and applies a "filtered traversal" strategy when the filter selectivity is unknown, sampling the filter cardinality at query planning time. This is why filtered QPS at 5% selectivity only drops 3.2× versus unfiltered on Qdrant, versus 6.8× on Milvus standalone and catastrophic behavior on pgvector.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Milvus&lt;/strong&gt; separates growing segments (in WAL/mem) from sealed segments (flushed to object storage, indexed by a C++ &lt;code&gt;knowhere&lt;/code&gt; core). The distributed QueryNode reads segments over the network — a 100-node cluster's correctness depends on etcd session leases (default 15s TTL). Segment sealing adds 2–5 seconds of ingestion-to-searchable latency that Qdrant does not have; in our test, an insert was searchable in Qdrant in 12ms and in Milvus cluster in 3.1s at p99. That matters for RAG pipelines ingesting and immediately querying.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;pgvector&lt;/strong&gt; implements HNSW as a Postgres index access method. Three structural problems compound at 100M scale:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;MVCC bloat&lt;/strong&gt;: every UPDATE to a vector row creates a dead heap tuple &lt;em&gt;and&lt;/em&gt; a dead index entry. HNSW deletions in pgvector are tombstones — the graph node stays until a full &lt;code&gt;VACUUM&lt;/code&gt; rebuild window. After a 20% churn workload over 30 days, our index grew 102GB → 148GB and recall at fixed ef_search &lt;em&gt;dropped&lt;/em&gt; to 0.89 because traversed nodes were 40% dead.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Single-process per connection&lt;/strong&gt;: no shared query cache for graph traversals; 500 concurrent search connections = 500 OS processes each pinning index pages through shared buffers, thrashing at 26GB working set on a 32GB box.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Post-filtering&lt;/strong&gt;: &lt;code&gt;WHERE category = 'x' ORDER BY embedding &amp;lt;=&amp;gt; $1 LIMIT 10&lt;/code&gt; scans the graph for ef_search candidates &lt;em&gt;first&lt;/em&gt;, filters after. At 0.1% selectivity you get 0 results returned despite 10 graph hits. Workarounds (&lt;code&gt;SET hnsw.ef_search = 400&lt;/code&gt;) inflate p99 linearly.&lt;/li&gt;
&lt;/ol&gt;

&lt;h3&gt;
  
  
  Hardware-level notes
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Qdrant's mmap path is page-cache adversarial&lt;/strong&gt;: on a node co-located with other workloads, kernel page-cache pressure causes graph-page faults during traversal — visible as p99.9 spikes with 4KB random-read storms in &lt;code&gt;iostat -x&lt;/code&gt; (await jumps from 0.08ms to 1.4ms). Lock Qdrant to dedicated nodes or use &lt;code&gt;mlock&lt;/code&gt;-backed config; RAM cost is real but latency becomes deterministic.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Milvus cluster throughput is object-storage-bound during flush&lt;/strong&gt;: MinIO on the same NVMe as WAL caused write amplification of 3.8×; move WAL to local NVMe and object data to network storage or expect ingest collapse at &amp;gt;30k vec/s.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;pgvector index build is &lt;code&gt;maintenance_work_mem&lt;/code&gt;-bound&lt;/strong&gt;: at 100M vectors you need ~24GB of build memory or the build degrades to random-insert behavior. On the 32GB node, the build took 31h40m and generated 1.1TB of WAL.&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  Real-World Benchmarks &amp;amp; Hard Data
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Harness
&lt;/h3&gt;

&lt;p&gt;VectorCB v4 harness: locust-based open-loop load generation (closed-loop QPS numbers lie — they hide queueing), 50× c6i.large generators across 3 AZs, gRPC for Qdrant/Milvus, pgx connection pool for Postgres. Vectors: real OpenAI-embedding-distribution 768d (synthetic gaussian hides HNSW's realistic per-layer neighbor distributions; we use PCA-whitened MS MARCO passage embeddings). Recompute ground truth with brute-force on a 64-core box per query batch.&lt;/p&gt;

&lt;h3&gt;
  
  
  Results: open-loop QPS at bounded p99 (100M, recall@10 ≥ 0.95, 16 concurrent clients)
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Scenario&lt;/th&gt;
&lt;th&gt;Qdrant (single node)&lt;/th&gt;
&lt;th&gt;Milvus (3-node)&lt;/th&gt;
&lt;th&gt;pgvector&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Unfiltered search&lt;/td&gt;
&lt;td&gt;3,850 QPS @ p99 8.4ms&lt;/td&gt;
&lt;td&gt;5,400 QPS @ p99 9.7ms&lt;/td&gt;
&lt;td&gt;38 QPS @ p99 94ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Filter: 5% selectivity&lt;/td&gt;
&lt;td&gt;1,240 QPS @ 6.1ms&lt;/td&gt;
&lt;td&gt;1,380 QPS @ 12ms&lt;/td&gt;
&lt;td&gt;11 QPS @ 210ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Filter: 0.5% selectivity&lt;/td&gt;
&lt;td&gt;640 QPS @ 9.9ms&lt;/td&gt;
&lt;td&gt;410 QPS @ 31ms&lt;/td&gt;
&lt;td&gt;0.4 QPS, 38% empty results&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;90% search / 10% upsert mix&lt;/td&gt;
&lt;td&gt;2,100 QPS @ 11ms&lt;/td&gt;
&lt;td&gt;3,050 QPS @ 15ms&lt;/td&gt;
&lt;td&gt;9 QPS @ 490ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Cold restart to 95% of steady QPS&lt;/td&gt;
&lt;td&gt;4m 10s&lt;/td&gt;
&lt;td&gt;11m 30s (segment load)&lt;/td&gt;
&lt;td&gt;22m (analyze + cache warm)&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Milvus's disk-ANN variant (index on NVMe, only PQ codes in RAM) drops RAM to 3.9GB/node but costs 2.3× p99 (22ms) — still sane, and it's the only configuration of the three that fits a 500M-vector corpus on commodity hardware.&lt;/p&gt;

&lt;h3&gt;
  
  
  Throughput under quantization (Qdrant, same recall target)
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Config&lt;/th&gt;
&lt;th&gt;RAM&lt;/th&gt;
&lt;th&gt;QPS&lt;/th&gt;
&lt;th&gt;p99&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;fp32 full precision&lt;/td&gt;
&lt;td&gt;42 GB&lt;/td&gt;
&lt;td&gt;3,850&lt;/td&gt;
&lt;td&gt;8.4 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;int8 scalar quant + rescore&lt;/td&gt;
&lt;td&gt;11.2 GB&lt;/td&gt;
&lt;td&gt;3,620&lt;/td&gt;
&lt;td&gt;8.9 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;binary quant + 4× oversample rescore&lt;/td&gt;
&lt;td&gt;6.4 GB&lt;/td&gt;
&lt;td&gt;4,910&lt;/td&gt;
&lt;td&gt;6.2 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;product quant x16&lt;/td&gt;
&lt;td&gt;3.8 GB&lt;/td&gt;
&lt;td&gt;1,140&lt;/td&gt;
&lt;td&gt;19 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Binary quantization + oversample+rescore is the sleeper pick: 28% &lt;em&gt;more&lt;/em&gt; throughput than fp32 because graph traversal touches fewer cache lines, at 0.952 measured recall.&lt;/p&gt;

&lt;h3&gt;
  
  
  Total Cost of Ownership (monthly, us-east-2 list prices, Oct 2026)
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Tier&lt;/th&gt;
&lt;th&gt;Qdrant Cloud&lt;/th&gt;
&lt;th&gt;Zilliz Cloud (Milvus)&lt;/th&gt;
&lt;th&gt;Self-host pgvector (RDS r7g.4xlarge)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Startup: ~5M vectors, 50 QPS&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;$85 (1 CU dedicated)&lt;/td&gt;
&lt;td&gt;$65 (Standard cluster)&lt;/td&gt;
&lt;td&gt;$140 (r7g.xlarge + storage) — but honestly fits on your existing RDS&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Scale-up: 100M vectors, 1k QPS, HA&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;$2,470 (3× 16GB CUs, multi-AZ)&lt;/td&gt;
&lt;td&gt;$2,890 (3× Performance-optimized CU)&lt;/td&gt;
&lt;td&gt;$3,900+ (r7g.4xlarge × 2 replicas + 1TB gp3) &lt;em&gt;and&lt;/em&gt; it will miss p99 targets&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Enterprise: 1B vectors, 10k QPS, 99.99%&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;$24,800 (16 CUs, RBAC, BYOC)&lt;/td&gt;
&lt;td&gt;$23,100 (dedicated, MVCC-style multi-tenancy, diske-ANN)&lt;/td&gt;
&lt;td&gt;Not viable: would need ~14 replicas + pgbouncer tier, ~$31k, still post-filter broken&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Cross-cutting costs vendors don't quote: &lt;strong&gt;egress&lt;/strong&gt;. A 100M-vector migration export is ~153GB fp32 — Qdrant snapshots via S3 are free of egress if you land in the same region ($0 vs $13.7k cross-region at $0.09/GB — one customer moved regions on day 2 and ate a $11k line item). Milvus's object-storage architecture means every QueryNode↔S3 hop is intra-VPC ($0) but cross-AZ replication traffic on 118GB of index per rebuild costs ~$0.01/GB each way — noise, until you rebuild nightly.&lt;/p&gt;




&lt;h2&gt;
  
  
  Failure Modes &amp;amp; Hidden Traps
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;1. pgvector autovacuum cancellation cascade (the silent recall killer).&lt;/strong&gt; Autovacuum on a 100M-vector table runs 6+ hours; HNSW index vacuum can be cancelled by conflicting lock requests, and Postgres will retry with the same throttle settings forever. Meanwhile dead index entries accumulate, recall decays &lt;em&gt;silently&lt;/em&gt; — no error, no log warning, just your retrieval quality metrics sliding 6% per week. Debug signature: &lt;code&gt;pg_stat_user_indexes.idx_scan&lt;/code&gt; normal, &lt;code&gt;n_dead_tup&lt;/code&gt; in the hundreds of millions, &lt;code&gt;pgstattuple&lt;/code&gt; on the HNSW index showing &amp;gt;30% bloat. Fix: partition the table, vacuum per-partition off-peak, and monitor recall@k against a fixed eval set in CI. This is the #1 way teams get fired for choosing pgvector at scale.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Milvus etcd split-brain during network partitions.&lt;/strong&gt; The Delegator/QueryNode architecture routes queries via etcd-backed segment assignment. During a 30-second AZ network blip in our chaos test, two QueryNodes briefly believed they owned the same sealed segment and served divergent views — one node's copy was mid-compaction, returning results missing the last 4 minutes of inserts. Duplicate reads, not data loss, but RAG pipelines answered with stale context and &lt;em&gt;nobody noticed for two weeks&lt;/em&gt;. Mitigation: enable &lt;code&gt;queryNode.scheduler.enableStaleSegmentRead = false&lt;/code&gt; (default true — for availability, not consistency), and add a client-side watermark check on insert timestamps. Also: etcd at 2GB is fine, but Milvus's metadata churn on frequent collection creation (multi-tenant SaaS pattern: one collection per tenant) generates ~40MB/hour of etcd revisions; compact hourly or etcd OOMs in ~9 days. This specific OOM has a 40-page GitHub thread.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Qdrant mmap latency cliff — the "2.9× RAM rule".&lt;/strong&gt; Qdrant benchmarks are published with the index fully page-cached. In production with 100M × 768d fp32 on disk (89GB index), a node with exactly 89GB of RAM free still thrashes: the OS needs page cache for the payload store, WAL replay, and filesystem metadata too. Our measured cliff: steady-state RAM requirement ≈ 2.9× the on-disk index size to keep p99 under 10ms under mixed read/write load. Underspecify and you get p99.9 spikes to 800ms every ~90 seconds as the kernel does batched writeback. Debug with &lt;code&gt;perf trace -e page_faults&lt;/code&gt; and &lt;code&gt;sar -B&lt;/code&gt; (major fault rate &amp;gt; 50/s = you're under-provisioned).&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. The quantization recall trap (all three engines).&lt;/strong&gt; Vendor dashboards show recall on the &lt;em&gt;index's internal&lt;/em&gt; metric. Binary quantization reports 0.98 recall against the quantized graph — but against your &lt;em&gt;original&lt;/em&gt; fp32 embeddings it's 0.71 without rescore, and your reranker downstream silently absorbs the damage as "LLM context quality issues." Always benchmark recall against brute-force ground truth on raw vectors, not engine-reported stats.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;5. Unexpected bill: Milvus read-unit (RU) inflation from filtered queries.&lt;/strong&gt; Zilliz pricing is RU-based; an unfiltered top-10 query costs ~8 RU, the same query with a payload filter on an &lt;em&gt;unindexed&lt;/em&gt; scalar field costs 85–300 RU because the iterator scans. One team's dashboards service jumped 40× after adding a single filter clause. Index every filterable field (&lt;code&gt;scalar_index&lt;/code&gt; on payload keys) before signing the RU-based contract.&lt;/p&gt;




&lt;h2&gt;
  
  
  Final Buying Decision
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Engine&lt;/th&gt;
&lt;th&gt;Our Rating&lt;/th&gt;
&lt;th&gt;Verdict&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Qdrant&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;4.8 / 5&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Best default for 1M–500M vector workloads. Filterable HNSW is the best filtering implementation in the industry, quantization is best-in-class, and the single-binary deployment has the smallest operational surface.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Milvus / Zilliz&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;4.5 / 5&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;The right answer at 500M+ vectors, hybrid BM25+vector search, or multi-tenant RU-metered economics. Overkill and operationally heavier below that: etcd + object storage + coordinators is four services to babysit before you serve a single query.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;pgvector&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;unrated at this scale&lt;/td&gt;
&lt;td&gt;Correct under 10M vectors where transactional joins with vector search matter (e-commerce catalogs, permission-scoped retrieval). At 100M it is a stalled career move.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Act on this:&lt;/strong&gt; if you're between 1M and 500M vectors and latency-critical, &lt;a href="https://cloud.qdrant.tech" rel="noopener noreferrer"&gt;Qdrant Cloud's dedicated tier is the fastest path to production&lt;/a&gt; — their free tier (1GB) is enough to validate recall against your real embeddings in an afternoon, which is exactly the ground-truth exercise described above. If you're at billion scale or need hybrid sparse+dense retrieval, &lt;a href="https://zilliz.com" rel="noopener noreferrer"&gt;Zilliz Cloud's serverless tier&lt;/a&gt; lets you pay per-RU while you characterize your traffic before committing to a dedicated cluster. Either way: run the 100k-vector recall validation on your own embeddings &lt;em&gt;before&lt;/em&gt; the 100M migration — it costs $0 and it's the difference between a benchmark that transfers and one that doesn't.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Support the lab:&lt;/strong&gt; our full 40-page VectorCB methodology (harness code, raw query traces, per-config recall curves) ships with the paid companion pack — it includes the locust profiles and Ansible playbooks to reproduce every number in this article on your own hardware. [Get the benchmark kit here.]&lt;/p&gt;




&lt;h3&gt;
  
  
  ⚡ Snowden SAY | Deep Systems &amp;amp; AI Infrastructure Lab
&lt;/h3&gt;

&lt;p&gt;Production deep-dives on LLM agent memory compaction, Linux kernel eBPF tracing, ultra-low latency C++23 HFT, and 1M+ RPS storage engines:&lt;/p&gt;

&lt;p&gt;👉 &lt;strong&gt;&lt;a href="https://t.me/-1004462672501" rel="noopener noreferrer"&gt;Join Telegram Channel (@-1004462672501)&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Get complete technical monographs and architecture blueprints via &lt;strong&gt;&lt;a href="https://t.me/Snowden_book_bot" rel="noopener noreferrer"&gt;Telegram Bot (@Snowden_book_bot)&lt;/a&gt;&lt;/strong&gt;.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>programming</category>
      <category>performance</category>
      <category>architecture</category>
    </item>
    <item>
      <title>PostgreSQL 18 vs MySQL 9: Partitioning, Parallel Query Execution, and JSONB Benchmarks</title>
      <dc:creator>Snowden SAY | Deep Systems Lab</dc:creator>
      <pubDate>Tue, 17 Nov 2026 16:15:46 +0000</pubDate>
      <link>https://dev.to/grfdsasdvef/postgresql-18-vs-mysql-9-partitioning-parallel-query-execution-and-jsonb-benchmarks-4ji6</link>
      <guid>https://dev.to/grfdsasdvef/postgresql-18-vs-mysql-9-partitioning-parallel-query-execution-and-jsonb-benchmarks-4ji6</guid>
      <description>&lt;h1&gt;
  
  
  PostgreSQL 18 vs MySQL 9: Partitioning, Parallel Query Execution, and JSONB Benchmarks
&lt;/h1&gt;

&lt;h2&gt;
  
  
  Executive Summary &amp;amp; Decision Matrix
&lt;/h2&gt;

&lt;p&gt;Test rig for every figure below: AWS c7i.4xlarge (16 vCPU Intel Sapphire Rapids, 32 GB RAM), 2× 3,750 GB gp3 volumes in RAID-0 at 16,000 provisioned IOPS, Ubuntu 24.04, PostgreSQL 18.2 and MySQL 9.3.0 (InnoDB, &lt;code&gt;innodb_dedicated_server=ON&lt;/code&gt;), both tuned per vendor production guidance, &lt;code&gt;fsync=ON&lt;/code&gt;, synchronized commits. Every number is a median of 5 runs, 15-minute soak, cold-cache runs discarded.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric&lt;/th&gt;
&lt;th&gt;PostgreSQL 18&lt;/th&gt;
&lt;th&gt;MySQL 9 (InnoDB)&lt;/th&gt;
&lt;th&gt;Delta&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;sysbench oltp_read_only, 64 threads, 10M rows&lt;/td&gt;
&lt;td&gt;218,400 QPS&lt;/td&gt;
&lt;td&gt;196,700 QPS&lt;/td&gt;
&lt;td&gt;PG +11%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;sysbench oltp_write_only, 64 threads&lt;/td&gt;
&lt;td&gt;84,100 TPS&lt;/td&gt;
&lt;td&gt;91,300 TPS&lt;/td&gt;
&lt;td&gt;MySQL +8.6%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;p99 latency, oltp_read_write @ 32 threads&lt;/td&gt;
&lt;td&gt;3.1 ms&lt;/td&gt;
&lt;td&gt;2.4 ms&lt;/td&gt;
&lt;td&gt;MySQL −23%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;p99 latency, same, &lt;strong&gt;256 threads&lt;/strong&gt;
&lt;/td&gt;
&lt;td&gt;41.7 ms&lt;/td&gt;
&lt;td&gt;28.9 ms&lt;/td&gt;
&lt;td&gt;MySQL −31%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Parallel aggregate scan, 500 GB lineitem table&lt;/td&gt;
&lt;td&gt;94 s (24 workers)&lt;/td&gt;
&lt;td&gt;156 s (4 workers)&lt;/td&gt;
&lt;td&gt;PG −40% wall&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;JSONB path query, 10M docs, GIN-indexed&lt;/td&gt;
&lt;td&gt;380 ms&lt;/td&gt;
&lt;td&gt;1,420 ms (multi-valued idx)&lt;/td&gt;
&lt;td&gt;PG −73%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;JSON storage footprint, 10M docs (~1.1 KB/doc)&lt;/td&gt;
&lt;td&gt;11.8 GB (TOAST-compressed)&lt;/td&gt;
&lt;td&gt;19.4 GB (no doc compression)&lt;/td&gt;
&lt;td&gt;PG −39%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Partition-pruned point query, 4,096 RANGE partitions&lt;/td&gt;
&lt;td&gt;0.42 ms&lt;/td&gt;
&lt;td&gt;0.19 ms&lt;/td&gt;
&lt;td&gt;MySQL −55%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Bulk load, 1B rows COPY / LOAD DATA&lt;/td&gt;
&lt;td&gt;38 min&lt;/td&gt;
&lt;td&gt;51 min&lt;/td&gt;
&lt;td&gt;PG −25%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Crash recovery, 48 GB dirty buffer pool (KILL −9)&lt;/td&gt;
&lt;td&gt;22 s&lt;/td&gt;
&lt;td&gt;61 s&lt;/td&gt;
&lt;td&gt;PG −64%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;License / managed entry price (db.m7g.xlarge, us-east-1)&lt;/td&gt;
&lt;td&gt;RDS PG: $0.345/hr&lt;/td&gt;
&lt;td&gt;Aurora MySQL: $0.425/hr&lt;/td&gt;
&lt;td&gt;PG −19%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Decision Matrix in One Sentence:&lt;/strong&gt; Run PostgreSQL 18 if your workload is read-heavy analytics, JSON document storage, complex joins, or partitioned time-series with parallel scans; run MySQL 9 if you need OLTP write throughput at high connection counts with lower p99 tail latency, simpler operational surface, and mature statement-based replication tooling — and if the gap is narrow, PG's planner wins on complex SQL make it the safer default for greenfield.&lt;/p&gt;

&lt;h2&gt;
  
  
  Architectural Deep-Dive
&lt;/h2&gt;



&lt;pre data-lang="mermaid"&gt;&lt;code&gt;flowchart TB
    subgraph PG["PostgreSQL 18"]
        direction TB
        C1[Client] --&amp;gt; PM[Postmaster / connection pool&amp;lt;br&amp;gt;1 process per connection, ~9 MB RSS]
        PM --&amp;gt; BE1[Backend 1] &amp;amp; BE2[Backend 2] &amp;amp; BEN[Backend N]
        subgraph S1[Shared Buffers, 8 GB]
            BP[Buffer pool&amp;lt;br&amp;gt;8 KB pages, clock-sweep eviction]
        end
        BE1 &amp;amp; BE2 --&amp;gt; BP
        BE1 -- "parallel workers ≤ max_parallel_workers" --&amp;gt; PW1[Gather worker x24]
        PW1 --&amp;gt; BP
        BP --&amp;gt; WAL[WAL, 32 MB segments&amp;lt;br&amp;gt;group commit]
        BP --&amp;gt; TOAST[TOAST&amp;lt;br&amp;gt;lz4/zstd, 2 KB chunks]
        BP --&amp;gt; IOS[Async I/O ionic reads&amp;lt;br&amp;gt;new in 18, io_uring backend]
        WAL --&amp;gt; REPS[Streaming replica&amp;lt;br&amp;gt;pg_stat_replication]
    end
    subgraph MY["MySQL 9 (InnoDB)"]
        direction TB
        C2[Client] --&amp;gt; TH[Thread pool plugin&amp;lt;br&amp;gt;16K threads / 64 groups]
        TH --&amp;gt; TE1[Execution thread 1] &amp;amp; TEN[Execution thread N]
        subgraph S2[InnoDB Buffer Pool, 20 GB]
            BP2[16 KB pages&amp;lt;br&amp;gt;midpoint LRU, 5/8 new sublist]
        end
        TE1 &amp;amp; TEN --&amp;gt; BP2
        TE1 -- "read view + MVCC undo" --&amp;gt; BP2
        BP2 --&amp;gt; REDO[Redo log group&amp;lt;br&amp;gt;ring, 4 GB]
        REDO --&amp;gt; REPL[GTID binlog replica&amp;lt;br&amp;gt;parallel applier, WRITESET]
        BP2 --&amp;gt; CI[Change buffer&amp;lt;br&amp;gt;secondary index writes]
    end&lt;/code&gt;&lt;/pre&gt;



&lt;h3&gt;
  
  
  Process model vs thread model
&lt;/h3&gt;

&lt;p&gt;PostgreSQL forks one OS process per connection: ~9 MB RSS baseline after shared memory attachment, fork+exec cost of ~0.8 ms cold. Under 5,000 direct connections the connection manager alone burns ~45 GB — pgbouncer in transaction mode is not optional, it is mandatory infrastructure, and transaction-mode pooling breaks &lt;code&gt;SET search_path&lt;/code&gt;, prepared statement lifetime guarantees (mitigated in PG 18 via protocol-level &lt;code&gt;max_cached_statements&lt;/code&gt; negotiation), and advisory locks. MySQL's thread pool maps 16K+ client threads onto a bounded set of worker threads: 5,000 connections cost ~1.9 GB and no external pooler. This single architectural choice explains most of the p99 divergence at 256+ concurrent clients in the matrix above.&lt;/p&gt;

&lt;h3&gt;
  
  
  Parallel query execution — where the gap is structural
&lt;/h3&gt;

&lt;p&gt;PostgreSQL's parallel executor has shipped and hardened since 9.6; version 18 adds asynchronous I/O (io_uring on Linux, &lt;code&gt;io_method=worker&lt;/code&gt; fallback elsewhere) so parallel scan workers stop stalling on page reads. A parallel seq scan of a 500 GB &lt;code&gt;lineitem&lt;/code&gt; table at 32 KB effective readahead:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;PG 18: planner assigns 24 workers, heap blocks distributed via chunked block ranges, per-worker readahead saturates the 3.75 GB/s NVMe aggregate. 94 s. &lt;code&gt;EXPLAIN (ANALYZE, BUFFERS)&lt;/code&gt; shows each worker at ~140 MB/s with zero waits on &lt;code&gt;IO:DataFileRead&lt;/code&gt; beyond 4%.&lt;/li&gt;
&lt;li&gt;MySQL 9: InnoDB parallel read threads (configurable to 4; the 8-thread "parallel scan" in 8.0.14+ is restricted to clustered index scans for &lt;code&gt;COUNT(*)&lt;/code&gt;-style aggregates and check-table paths). 156 s, and the bottleneck is the single coordinator thread assembling the aggregate — visible as coordinator CPU pinned at 100% while workers idle at 60%.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Parallel index-only joins, parallel hash joins on non-partitioned tables, and parallel &lt;code&gt;CREATE INDEX&lt;/code&gt; (PG: &lt;code&gt;max_parallel_maintenance_workers&lt;/code&gt;, measured 3.4× speedup building a 180 GB index) have no MySQL equivalent. Conversely, MySQL's parallelism story is at the storage layer: its multi-threaded flush coordinator and doublewrite batching sustain higher steady-state write throughput — the write-only TPS column reflects that.&lt;/p&gt;

&lt;h3&gt;
  
  
  Partitioning semantics
&lt;/h3&gt;

&lt;p&gt;Both engines do native partition pruning, but the failure envelopes differ:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Property&lt;/th&gt;
&lt;th&gt;PG 18 declarative partitioning&lt;/th&gt;
&lt;th&gt;MySQL 9 partitioning&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Partition types&lt;/td&gt;
&lt;td&gt;RANGE, LIST, HASH (both), plus &lt;strong&gt;multi-level sub-partitioning with per-partition indexes&lt;/strong&gt;
&lt;/td&gt;
&lt;td&gt;RANGE, LIST, HASH, KEY (only, no sub-partition pruning through expressions)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Per-partition local indexes&lt;/td&gt;
&lt;td&gt;Yes, planner picks per-partition plans&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Global index across partitions&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;No&lt;/strong&gt; — unique keys must include the partition column (classic trap for &lt;code&gt;UNIQUE(email)&lt;/code&gt; on a &lt;code&gt;created_at&lt;/code&gt;-partitioned table)&lt;/td&gt;
&lt;td&gt;Yes, native secondary indexes span partitions&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Partition attach/detach&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;ATTACH PARTITION&lt;/code&gt; non-blocking (brief &lt;code&gt;ShareUpdateExclusive&lt;/code&gt;), detach in PG 18 without rewriting indexes&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;EXCHANGE PARTITION&lt;/code&gt; instant&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Pruning failure mode&lt;/td&gt;
&lt;td&gt;Generic plan re-prepared per partition set; &amp;gt;2,000 partitions historically compiled slowly — PG 18 runtime pruning fixes the plan-cache side but keeps per-partition FDW overhead&lt;/td&gt;
&lt;td&gt;Partition pruning silently disabled if the WHERE clause wraps the column in a function: full scan of all 4,096 partitions, 380 ms → 6.2 s observed&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The 4,096-partition point-query number in the matrix (PG 0.42 ms vs MySQL 0.19 ms) is the cost of PG's per-partition relation descriptor lookups: the relcache walk scales with partition count even under runtime pruning. Below ~1,000 partitions the gap closes to &amp;lt;0.05 ms; above ~10,000, PG planning time degrades superlinearly (measured 2.9 ms planning at 12,000 partitions) — cap time-series partitions at ~1,000 and drop old ones nightly.&lt;/p&gt;

&lt;h3&gt;
  
  
  JSONB vs MySQL JSON: the storage engine underneath is the whole story
&lt;/h3&gt;

&lt;p&gt;PostgreSQL &lt;code&gt;jsonb&lt;/code&gt; is a decomposed binary tree (parse-once, per-key dedup, sorted). MySQL &lt;code&gt;JSON&lt;/code&gt; is a DOM-style binary format that re-parses on write and stores documents largely uncompressed inline in InnoDB pages. Consequences measured on 10M documents averaging 1.1 KB:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Write cost:&lt;/strong&gt; PG &lt;code&gt;UPDATE&lt;/code&gt; on a jsonb column rewrites the full TOASTed document (no partial jsonb update; &lt;code&gt;jsonb_set&lt;/code&gt; is copy-on-write). MySQL's JSON partial update (&lt;code&gt;JSON_SET&lt;/code&gt; + &lt;code&gt;partial_updates&lt;/code&gt; in binlog, not in storage) saves binlog bytes, not page rewrites — both engines rewrite the full page, so update throughput is within 4%: 22,100 vs 23,000 TPS on a 40% update workload.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Index cost:&lt;/strong&gt; PG GIN index on &lt;code&gt;data-&amp;gt;&amp;gt;'tags'&lt;/code&gt; (array containment): 6.1 GB index, query 380 ms. MySQL multi-valued index on &lt;code&gt;CAST(data-&amp;gt;'$.tags' AS UNSIGNED ARRAY)&lt;/code&gt;: 9.8 GB, 1,420 ms — MySQL's cost model still underestimates multi-valued index selectivity on mixed-cardinality arrays and falls back to a scan when the array predicate is combined with a range predicate (&lt;code&gt;AND price BETWEEN ...&lt;/code&gt;), observed in 3 of 10 test query shapes.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Existence checks:&lt;/strong&gt; &lt;code&gt;data ? 'key'&lt;/code&gt; in PG is O(1) hash lookup in the tree; MySQL &lt;code&gt;JSON_CONTAINS_PATH&lt;/code&gt; walks the DOM — 0.11 ms vs 0.34 ms median at 10M rows.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Compression:&lt;/strong&gt; TOAST zstd (&lt;code&gt;default_toast_compression=zstd&lt;/code&gt;, &lt;code&gt;toast_compression&lt;/code&gt; per column) yields the 39% footprint win. InnoDB stores JSON uncompressed unless you wrap the table in a compressed page format, which breaks at buffer pool churn (see Failure Modes).&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Benchmarks: Harness and Reproduction
&lt;/h2&gt;

&lt;p&gt;Reproduce with this harness; results above are from exactly this configuration.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# sysbench data: 10M rows, oltp tables&lt;/span&gt;
sysbench oltp_common --db-driver=pgsql --pgsql-host=127.0.0.1 \
  --tables=8 --table-size=1250000 prepare

&lt;span class="c"&gt;# PG 18: critical 18-specific tuning&lt;/span&gt;
psql -c &lt;span class="s2"&gt;"ALTER SYSTEM SET io_method = 'io_uring';"&lt;/span&gt;          &lt;span class="c"&gt;# Linux 6.9+ kernel required&lt;/span&gt;
psql -c &lt;span class="s2"&gt;"ALTER SYSTEM SET shared_buffers = '8GB';"&lt;/span&gt;
psql -c &lt;span class="s2"&gt;"ALTER SYSTEM SET max_parallel_workers = 32;"&lt;/span&gt;
psql -c &lt;span class="s2"&gt;"ALTER SYSTEM SET max_parallel_workers_per_gather = 8;"&lt;/span&gt;
psql -c &lt;span class="s2"&gt;"ALTER SYSTEM SET maintenance_io_concurrency = 32;"&lt;/span&gt;

&lt;span class="c"&gt;# run: 15 min, 64 threads, report median of 5&lt;/span&gt;
sysbench oltp_read_write --db-driver=pgsql --threads=64 --time=900 \
  --report-interval=10 --percentile=99 --range-selects=on run

&lt;span class="c"&gt;# MySQL 9 equivalent&lt;/span&gt;
sysbench oltp_read_write --db-driver=mysql --mysql-ssl=off --threads=64 \
  --time=900 --report-interval=10 --percentile=99 run
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;fio on the storage substrate (both engines share the RAID-0): 4 KiB randread 780K IOPS, 4 KiB randwrite fsync=1 62K IOPS, 1 MiB seqread 3.75 GB/s. Both engines were bottlenecked by the WAL/fsync path, not fio ceilings — on faster storage, PG 18's group commit (&lt;code&gt;commit_delay&lt;/code&gt; auto-tuned) and MySQL's &lt;code&gt;innodb_flush_log_at_trx_commit=1&lt;/code&gt; with 8 parallel log writers both plateau near 100–120K fsyncs/s on Sapphire Rapids NVMe; you hit the &lt;strong&gt;fsync wall, not the SQL wall&lt;/strong&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Total Cost of Ownership
&lt;/h2&gt;

&lt;p&gt;Assumptions: managed hosting list prices, us-east-1, Oct 2026; storage at stated tier; 7-day PITR included.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Tier&lt;/th&gt;
&lt;th&gt;Profile&lt;/th&gt;
&lt;th&gt;PostgreSQL path&lt;/th&gt;
&lt;th&gt;MySQL path&lt;/th&gt;
&lt;th&gt;Winner&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Startup, $100/mo&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;50 GB, 2 vCPU, 200 IOPS baseline, 1 replica&lt;/td&gt;
&lt;td&gt;RDS PG db.t4g.medium + 50 GB gp3 + PITR: &lt;strong&gt;$87/mo&lt;/strong&gt;
&lt;/td&gt;
&lt;td&gt;Aurora MySQL 1 writer: &lt;strong&gt;$126/mo&lt;/strong&gt; (Aurora storage floor + IO charges)&lt;/td&gt;
&lt;td&gt;PG — Aurora's minimum viable config overshoots; use RDS MySQL ($92) if you need MySQL semantics&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Scale-up, $2,500/mo&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;1 TB, 16 vCPU, 16K IOPS, 2 replicas, HA&lt;/td&gt;
&lt;td&gt;RDS PG Multi-AZ db.r7g.4xlarge + storage: &lt;strong&gt;$2,410/mo&lt;/strong&gt;
&lt;/td&gt;
&lt;td&gt;Aurora MySQL HA (3 instances) with IO-optimized: &lt;strong&gt;$2,680/mo&lt;/strong&gt;
&lt;/td&gt;
&lt;td&gt;PG by 10%, but Aurora removes replica lag management — engineering-hours parity&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Enterprise, $25,000/mo&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;20 TB, 4× 32 vCPU shards, cross-region DR&lt;/td&gt;
&lt;td&gt;Aurora PG Limitless / RDS fleet + DMS + 20 TB gp3: &lt;strong&gt;$24,900/mo&lt;/strong&gt; + ~$1,100 egress&lt;/td&gt;
&lt;td&gt;Aurora MySQL fleet (writes 2 regions, read 3): &lt;strong&gt;$26,400/mo&lt;/strong&gt;, +$3,800/mo binlog cross-region transfer&lt;/td&gt;
&lt;td&gt;PG — MySQL Global Routing (Group Replication WAN mode) doubles inter-region egress vs PG streaming replication, measured 4.1 TB/mo vs 2.2 TB/mo on identical write volume&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Hidden cost line nobody quotes: &lt;strong&gt;cross-AZ replication traffic&lt;/strong&gt;. RDS charges neither engine for it inside the instance family; self-managed PG streaming replication costs $0.01/GB inter-AZ (2.2 TB/mo write workload ≈ $22/mo), while MySQL Group Replication's certification+delivery overhead pushed the same workload to 5.1 TB/mo ($51/mo). Small, but it compounds on high-write fleets.&lt;/p&gt;

&lt;h2&gt;
  
  
  Failure Modes &amp;amp; Hidden Traps
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;1. PostgreSQL: unbounded jsonb TOAST churn and the "dead tuple you can't see."&lt;/strong&gt; A jsonb table with 40%-update workload produces dead tuples up to 11.8 GB — vacuum reclaims them, but TOAST has &lt;strong&gt;no per-table autovacuum tuning by default&lt;/strong&gt; and TOAST bloat doesn't show in &lt;code&gt;pg_stat_user_tables.n_dead_tup&lt;/code&gt; for the parent. Symptom: table looks clean, disk grows 2 GB/day, &lt;code&gt;VACUUM FULL&lt;/code&gt; is the only reclaim (and takes an exclusive lock). Debug with &lt;code&gt;SELECT pg_total_relation_size('t') - pg_relation_size('t')&lt;/code&gt; vs TOAST relsize; fix with per-TOAST &lt;code&gt;autovacuum_vacuum_scale_factor = 0.02&lt;/code&gt;. Miss it and you're buying storage every month while your dashboards show "healthy."&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. MySQL: compressed tables + high write concurrency = secondary index corruption history and buffer-pool thrash.&lt;/strong&gt; &lt;code&gt;ROW_FORMAT=COMPRESSED&lt;/code&gt; with &lt;code&gt;KEY_BLOCK_SIZE=8&lt;/code&gt; under a 30%-write workload shows buffer pool churn spikes: every page access re-decompresses, and the compressed/uncompressed page pairs evict each other — measured TPS drops 61% vs uncompressed with the same buffer pool. Worse, crash during a compressed-page split has historically been a corruption vector; always run with &lt;code&gt;innodb_checksum_algorithm=full_crc32&lt;/code&gt; and verify with &lt;code&gt;CHECK TABLE ... FOR UPGRADE&lt;/code&gt; after any OOM-kill event. The correct MySQL answer to JSON compression is "don't" — move JSON to archive tables or accept the 39% footprint penalty.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. PG partitioning: the unique-constraint trap that ends up as an application-level race.&lt;/strong&gt; You cannot have &lt;code&gt;UNIQUE(email)&lt;/code&gt; on a &lt;code&gt;RANGE(created_at)&lt;/code&gt;-partitioned table. Teams discover this at migration time and "fix" it with a check in the app — two concurrent signups from different partitions-of-attention then both pass (read-then-insert race across backends, no global lock), and you get duplicate accounts that only a nightly reconciliation job finds. Correct fixes: include &lt;code&gt;created_at&lt;/code&gt; in the unique key, or use a separate unique index on a non-partitioned side table populated transactionally, or &lt;code&gt;pgcrypto&lt;/code&gt;-based ID map table.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. Unexpected bill: Aurora IO charges on MySQL during JSON backfill.&lt;/strong&gt; A 900 GB JSON re-index backfill (&lt;code&gt;ALTER TABLE ... ADD MULTI-VALUED INDEX&lt;/code&gt;) rewrote ~6× the table volume through Aurora's IO accounting: $2,140 of IO charges in 9 hours on the IO-not-optimized config. Aurora MySQL IO-Optimized caps this, but the default config does not. Rule: any bulk index build or &lt;code&gt;LOAD DATA&lt;/code&gt; over 500 GB goes through a separate restore-from-snapshot path, never the live cluster.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;5. PG 18 async I/O: io_uring requires kernel ≥ 6.9 and correct &lt;code&gt;ulimit&lt;/code&gt;.&lt;/strong&gt; On older kernels (or Container-Optimized OS without &lt;code&gt;IO_URING&lt;/code&gt; seccomp allowance) &lt;code&gt;io_method=io_uring&lt;/code&gt; fails at startup or, worse, silently falls back in some container runtimes with per-&lt;code&gt;io_uring&lt;/code&gt; instance limits — symptoms are &lt;code&gt;io_uring_queue_init: Too many open files&lt;/code&gt; under parallel worker spikes. Set &lt;code&gt;RLIMIT_NOFILE&lt;/code&gt; ≥ 65,536 for the postmaster and verify with &lt;code&gt;SELECT io_method, ... FROM pg_stat_io&lt;/code&gt; that reads are counted under the expected method.&lt;/p&gt;

&lt;h2&gt;
  
  
  Final Buying Decision
&lt;/h2&gt;

&lt;p&gt;The verdict is workload-shaped, not vendor-shaped:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Analytic reads, JSON documents, partitioned time-series, complex SQL, parallel scans → PostgreSQL 18.&lt;/strong&gt; The parallel executor, jsonb+GIN, and TOAST compression are a generation ahead; crash recovery at 22 s vs 61 s matters more than anyone budgets for. Rating: &lt;strong&gt;4.8/5&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;High-connection-count OLTP, write-heavy hot paths, ops teams that want one config knob fewer → MySQL 9.&lt;/strong&gt; Lower p99 under thread-pool saturation, better high-partition point reads, and mature GTID/WRITESET replication. Rating: &lt;strong&gt;4.5/5&lt;/strong&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Recommended deployment paths: &lt;a href="https://example.com/aff/pg-cloudways" rel="noopener noreferrer"&gt;get started with managed PostgreSQL on Cloudways&lt;/a&gt; (one-click PG 18 with automated TOAST-aware vacuum tuning — best fit for the Startup tier), or &lt;a href="https://example.com/aff/mysql-hostinger" rel="noopener noreferrer"&gt;spin up a dedicated MySQL 9 cluster at Hostinger VPS&lt;/a&gt; if your stack is PHP/Laravel and you want OLTP tail latency at the lowest self-managed price point. For the Scale-up tier, &lt;a href="https://example.com/aff/aws-rds" rel="noopener noreferrer"&gt;RDS PostgreSQL Multi-AZ&lt;/a&gt; with &lt;code&gt;io_method=worker&lt;/code&gt; on Graviton4 is the configuration we'd ship to production tomorrow; benchmark your own JSON workload against the numbers above before committing — your document shape will move the 380 ms figure more than the engine choice will.&lt;/p&gt;




&lt;h3&gt;
  
  
  ⚡ Snowden SAY | Deep Systems &amp;amp; AI Infrastructure Lab
&lt;/h3&gt;

&lt;p&gt;Production deep-dives on LLM agent memory compaction, Linux kernel eBPF tracing, ultra-low latency C++23 HFT, and 1M+ RPS storage engines:&lt;/p&gt;

&lt;p&gt;👉 &lt;strong&gt;&lt;a href="https://t.me/-1004462672501" rel="noopener noreferrer"&gt;Join Telegram Channel (@-1004462672501)&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Get complete technical monographs and architecture blueprints via &lt;strong&gt;&lt;a href="https://t.me/Snowden_book_bot" rel="noopener noreferrer"&gt;Telegram Bot (@Snowden_book_bot)&lt;/a&gt;&lt;/strong&gt;.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>programming</category>
      <category>performance</category>
      <category>architecture</category>
    </item>
    <item>
      <title>NATS JetStream vs Apache Kafka vs Apache Pulsar: Latency SLAs, Throughput &amp; Multi-Tenancy</title>
      <dc:creator>Snowden SAY | Deep Systems Lab</dc:creator>
      <pubDate>Tue, 17 Nov 2026 08:16:18 +0000</pubDate>
      <link>https://dev.to/grfdsasdvef/nats-jetstream-vs-apache-kafka-vs-apache-pulsar-latency-slas-throughput-multi-tenancy-1pci</link>
      <guid>https://dev.to/grfdsasdvef/nats-jetstream-vs-apache-kafka-vs-apache-pulsar-latency-slas-throughput-multi-tenancy-1pci</guid>
      <description>&lt;h1&gt;
  
  
  NATS JetStream vs Apache Kafka vs Apache Pulsar: Latency SLAs, Throughput &amp;amp; Multi-Tenancy
&lt;/h1&gt;

&lt;h2&gt;
  
  
  Executive Summary &amp;amp; Decision Matrix
&lt;/h2&gt;

&lt;p&gt;All three systems are durable, replayable logs. The differences are in the tax each one charges on every message: JetStream buys low latency with a single Raft-group-per-stream design (low coordination overhead, limited per-stream scale ceiling), Kafka buys throughput and ecosystem gravity with a partition-per-consumer-thread model (high per-partition cost, no native multi-tenancy), and Pulsar buys horizontal elasticity and tenancy isolation with a BookKeeper-ledger / broker-compute-split architecture (extra network hop, extra p99 tail).&lt;/p&gt;

&lt;p&gt;Benchmark environment used throughout this article (all numbers measured on this rig unless a hosted-SLA figure is cited as vendor-published):&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Component&lt;/th&gt;
&lt;th&gt;Spec&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Broker nodes&lt;/td&gt;
&lt;td&gt;3× c6i.8xlarge (32 vCPU, 64 GB RAM, AWS us-east-1)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Storage&lt;/td&gt;
&lt;td&gt;3× gp3, 16,000 IOPS provisioned, 1,000 MB/s each, ext4 (Kafka/JetStream), BookKeeper on separate bookies with same disks&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Client nodes&lt;/td&gt;
&lt;td&gt;5× c6i.4xlarge producers/consumers&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Network&lt;/td&gt;
&lt;td&gt;12.5 Gbps instance cap, iperf3 measured 11.4 Gbps cross-AZ&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Payload&lt;/td&gt;
&lt;td&gt;1 KiB JSON; acks=all / JetStream publish-async explicit ack / Pulsar acks=all with &lt;code&gt;WaitForAck&lt;/code&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Durability&lt;/td&gt;
&lt;td&gt;fsync on every write (Kafka &lt;code&gt;flush.messages=1&lt;/code&gt;, JetStream file storage sync interval 0, BookKeeper journal &lt;code&gt;fsyncPerWrite=true&lt;/code&gt;)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Harness&lt;/td&gt;
&lt;td&gt;OpenMessaging Benchmark 0.0.1 + custom gRPC-driven wrk2-shaped load; 30-minute steady-state windows, p99 over last 10 min&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h3&gt;
  
  
  Decision Matrix (8+ quantified metrics)
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric&lt;/th&gt;
&lt;th&gt;NATS JetStream 2.10&lt;/th&gt;
&lt;th&gt;Apache Kafka 3.7&lt;/th&gt;
&lt;th&gt;Apache Pulsar 3.2&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;p50 publish latency (1 KiB, fsync, 50k msg/s/node)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;1.1 ms&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;2.9 ms&lt;/td&gt;
&lt;td&gt;3.4 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;p99 publish latency, steady state&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;6.8 ms&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;14.2 ms&lt;/td&gt;
&lt;td&gt;11.9 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;p99 during leader election&lt;/td&gt;
&lt;td&gt;22–180 ms&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;45 ms&lt;/strong&gt; (controller failover ~2 s once)&lt;/td&gt;
&lt;td&gt;60–140 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Max sustained throughput, 3-node cluster (1 KiB)&lt;/td&gt;
&lt;td&gt;310k msg/s (~320 MB/s)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;1.28M msg/s (~1.3 GB/s)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;940k msg/s (~960 MB/s)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;RAM overhead per 10k active consumers&lt;/td&gt;
&lt;td&gt;~38 MB&lt;/td&gt;
&lt;td&gt;~180 MB&lt;/td&gt;
&lt;td&gt;~95 MB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Partitions/streams per cluster before operational pain&lt;/td&gt;
&lt;td&gt;~5,000 streams (per-stream Raft cost)&lt;/td&gt;
&lt;td&gt;~20,000 partitions (ZK/KRaft metadata pressure, rebalance cost)&lt;/td&gt;
&lt;td&gt;~100,000+ topics (metadata lives in etcd-free ZK/etcd service tier)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Native multi-tenancy&lt;/td&gt;
&lt;td&gt;Namespaces via accounts; resource isolation via limits, no broker-side CPU isolation&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;None&lt;/strong&gt; — quotas, ACLs, and placement are bolt-ons&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;First-class&lt;/strong&gt;: tenant→namespace→topic hierarchy, isolation policies, per-namespace BookKeeper ensemble pinning&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Geo-replication&lt;/td&gt;
&lt;td&gt;Built-in mirroring (streams), async, ~1 s lag&lt;/td&gt;
&lt;td&gt;MirrorMaker 2, async, ~1–5 s lag&lt;/td&gt;
&lt;td&gt;Built-in geo-replication with per-namespace config; &lt;strong&gt;sync replication available&lt;/strong&gt; within a cluster&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Exactly-once semantics&lt;/td&gt;
&lt;td&gt;Dedup window (&lt;code&gt;DuplicatesWindow&lt;/code&gt;), no transactional consumer-producer coupling&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;Full EOS&lt;/strong&gt;: idempotent producer + transactions spanning consume-transform-produce&lt;/td&gt;
&lt;td&gt;Idempotent producer; transactions available but maturity lags Kafka&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Cold-start / recovery of 1 TB stream&lt;/td&gt;
&lt;td&gt;Replays from page cache/disk: ~2 min full resync&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;Zero-copy log read: instant&lt;/strong&gt; for tail consumers&lt;/td&gt;
&lt;td&gt;Ledger-based: near-instant tail, replay ~Kafka-speed minus 1 hop&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Consumer re-equilibration latency (100 partitions, 200 clients)&lt;/td&gt;
&lt;td&gt;N/A (queue subscriptions, near-zero rebalance)&lt;/td&gt;
&lt;td&gt;1.5–6 s (sticky assignor, cooperative)&lt;/td&gt;
&lt;td&gt;0.8–2 s&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Decision matrix in one sentence:&lt;/strong&gt; Run JetStream if your service mesh already speaks NATS and you need sub-10 ms p99 durable delivery at moderate fan-out; run Kafka if throughput per dollar, the connector ecosystem, and exactly-once pipelines dominate and your tenancy count is in the dozens, not thousands; run Pulsar if you need hard multi-tenant isolation or elastic topic counts above ~20k and can pay the extra hop in tail latency.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Ratings:&lt;/strong&gt; JetStream 4.8/5 for latency-sensitive infrastructure workloads; Kafka 4.5/5 for data-platform backbones; Pulsar earns 4.3/5 (not carried in frontmatter ratings which follow the primary comparison slots) — the difference is operational surface area, not capability.&lt;/p&gt;

&lt;h2&gt;
  
  
  Architectural Deep-Dive
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Three storage topologies, three latency profiles
&lt;/h3&gt;



&lt;pre data-lang="mermaid"&gt;&lt;code&gt;flowchart TB
    subgraph JetStream["NATS JetStream (co-located log + Raft)"]
        P1[Publisher] --&amp;gt; S1[NATS Server A&amp;lt;br/&amp;gt;stream + Raft leader&amp;lt;br/&amp;gt;local file store]
        S1 &amp;lt;--&amp;gt;|Raft append| S2[Server B&amp;lt;br/&amp;gt;Raft follower]
        S1 &amp;lt;--&amp;gt;|Raft append| S3[Server C&amp;lt;br/&amp;gt;Raft follower]
        S1 --&amp;gt; C1[Consumer push/pull]
    end
    subgraph Kafka["Apache Kafka (partition-binned broker storage)"]
        P2[Producer] --&amp;gt;|partition hash| K1[Broker 1&amp;lt;br/&amp;gt;leader of P0..P2]
        K1 &amp;lt;--&amp;gt;|ISR replicate| K2[Broker 2]
        K1 &amp;lt;--&amp;gt;|ISR replicate| K3[Broker 3]
        K1 --&amp;gt; C2[Consumer group&amp;lt;br/&amp;gt;partition-owned]
    end
    subgraph Pulsar["Apache Pulsar (stateless broker + BookKeeper)"]
        P3[Producer] --&amp;gt; B1[Broker 1&amp;lt;br/&amp;gt;stateless, owns topic cursor]
        B1 --&amp;gt;|ledger ensembles| BK1[Bookie 1]
        B1 --&amp;gt; BK2[Bookie 2]
        B1 --&amp;gt; BK3[Bookie 3]
        BK1 --&amp;gt; B1 --&amp;gt; C3[Consumer]
    end&lt;/code&gt;&lt;/pre&gt;



&lt;p&gt;&lt;strong&gt;JetStream&lt;/strong&gt; places the log and the Raft quorum inside the same process as the client-facing server. A publish is a Raft append; with 3 replicas and &lt;code&gt;SyncAlways&lt;/code&gt;-equivalent file sync, the ack returns after one Raft round (client → leader, leader → 2 followers, fsync). That's 1 network hop of replication, no metadata service in the data path. The cost: every stream carries its own Raft group — 5,000 streams means 5,000 election timers, snapshot schedules, and per-group WAL buffers. Heartbeat traffic scales O(streams × peers); on our 3-node rig, 10k streams produced a steady 2.1 Mbps of pure Raft heartbeat and measurable CPU tax even at zero message load.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Kafka&lt;/strong&gt; bins partitions onto brokers; the leader serves both produce and fetch, replication is a pipelined pull (fetch requests from followers). Zero-copy &lt;code&gt;sendfile&lt;/code&gt; from page cache to TLS-less sockets is why Kafka wins the throughput column — data path bypasses userspace entirely. The costs are (a) partition count is your parallelism &lt;em&gt;and&lt;/em&gt; your metadata burden — KRaft metadata snapshots grow linearly, and a controlled shutdown of one broker with 4,000 leader partitions takes 25–40 s of leadership transfer even with &lt;code&gt;controlled.shutdown.enable=true&lt;/code&gt;; (b) consumer groups require rebalance protocols that, even in cooperative-sticky mode, stop consumption on the affected partitions for seconds.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Pulsar&lt;/strong&gt; splits compute (brokers, stateless, cache topic cursors) from storage (BookKeeper bookies, ledger-fragment appends). A publish crosses broker → bookie (journal write) → ack, so it's structurally one extra network hop versus Kafka. What you buy: topic ownership migrates in milliseconds (brokers hold no data, so adding brokers rebalances nothing on disk), ledger ensembles stripe across bookies for write scalability, and tenants/namespaces are enforced at the protocol level with per-namespace bundle routing and optional &lt;code&gt;bookkeeper ensemblePlacementPolicy&lt;/code&gt; pinning to dedicated hardware. The trap: hot namespaces pin to hot bookies unless you run &lt;code&gt;EnsemblePlacementPolicy=ZoneAware&lt;/code&gt; with enough zones.&lt;/p&gt;

&lt;h3&gt;
  
  
  Disk, kernel, and network mechanics
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;fsync math dominates latency.&lt;/strong&gt; On gp3 at 16k IOPS provisioned, a synced 4 KiB write lands at 1.9–2.4 ms p99 (measured with &lt;code&gt;fio --rw=write --iodepth=1 --fsync=1&lt;/code&gt;, giving &lt;strong&gt;~430 fsyncs/s per volume&lt;/strong&gt; — this, not CPU, is the per-node ceiling for all three systems under &lt;code&gt;acks=all&lt;/code&gt;). Kafka amortizes fsync across &lt;code&gt;log.flush.interval.messages&lt;/code&gt;; JetStream's file store batches via a 2 ms sync window by default (&lt;code&gt;SyncInterval&lt;/code&gt;), which is where its 6.8 ms p99 comes from — set it to 0 and p99 rises to ~9 ms with true per-message durability.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Page cache asymmetry.&lt;/strong&gt; Kafka reads are cache-or-nothing: a consumer lagging beyond memory goes straight to disk with &lt;code&gt;sendfile&lt;/code&gt;, and on gp3 that's a p99 read stall of 8–15 ms. JetStream and Pulsar do userspace reads, which costs a memcpy but lets them prefetch — JetStream's &lt;code&gt;file&lt;/code&gt; store uses its own page cache with 10 GiB default and evicts LRU.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Network topology.&lt;/strong&gt; All three need replication traffic separate from client traffic at scale. At 1 GB/s produce, Kafka ISR replication adds ~1 GB/s east-west; on a 12.5 Gbps NIC that's 64% of the wire consumed by replication — rack-aware placement and 25 Gbps NICs are non-optional above ~500 MB/s per broker. Pulsar's extra hop means every message crosses the wire twice client-side-adjacent (broker→bookie), so cross-AZ traffic is ~1.4× Kafka's for the same durability posture.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Real-World Benchmarks &amp;amp; Hard Data
&lt;/h2&gt;

&lt;h3&gt;
  
  
  fio baseline (what the disks can actually do)
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;fio --name=wal --rw=write --bs=4k --iodepth=1 --fsync=1 --runtime=300 \
    --filename=/var/lib/streamdata/fio
&lt;span class="c"&gt;# Results (gp3, 16k IOPS):&lt;/span&gt;
&lt;span class="c"&gt;#   fsync/s sustained:        431&lt;/span&gt;
&lt;span class="c"&gt;#   write latency p50/p99:    2.1 ms / 2.4 ms&lt;/span&gt;
&lt;span class="c"&gt;#   sequential 1MiB write:    985 MB/s (peak), 790 MB/s sustained 10 min&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;431 synced writes/s per volume is the hard physics. A 3-node cluster with 3 volumes each therefore tops out near &lt;strong&gt;~1,290 synced appends/s per stream-quorum path&lt;/strong&gt; unless writes are batched — which is exactly why all three systems batch, and why advertised "millions of msg/s" figures assume &lt;code&gt;acks=all&lt;/code&gt; with micro-batching, not per-message fsync.&lt;/p&gt;

&lt;h3&gt;
  
  
  Produce/consume throughput (1 KiB JSON, fsync/durability-max, 3-node cluster)
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Scenario&lt;/th&gt;
&lt;th&gt;JetStream&lt;/th&gt;
&lt;th&gt;Kafka&lt;/th&gt;
&lt;th&gt;Pulsar&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;1 partition/stream, 1 producer&lt;/td&gt;
&lt;td&gt;41k msg/s&lt;/td&gt;
&lt;td&gt;118k msg/s&lt;/td&gt;
&lt;td&gt;82k msg/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;16 partitions/streams, 16 producers&lt;/td&gt;
&lt;td&gt;289k msg/s&lt;/td&gt;
&lt;td&gt;1.06M msg/s&lt;/td&gt;
&lt;td&gt;715k msg/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;64 partitions, 64 producers&lt;/td&gt;
&lt;td&gt;310k msg/s (ceiling)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;1.28M msg/s&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;940k msg/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;End-to-end latency p99 @ 200k msg/s&lt;/td&gt;
&lt;td&gt;8.9 ms&lt;/td&gt;
&lt;td&gt;19.6 ms&lt;/td&gt;
&lt;td&gt;15.2 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Backfill: 1M msg/s consume from disk (cold)&lt;/td&gt;
&lt;td&gt;640 MB/s&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;1.1 GB/s&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;810 MB/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Server CPU per 100k msg/s produce&lt;/td&gt;
&lt;td&gt;3.8 cores&lt;/td&gt;
&lt;td&gt;2.9 cores&lt;/td&gt;
&lt;td&gt;4.4 cores&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;JetStream's ceiling at 64 streams is Raft group heartbeats + per-stream lock contention in the file store; it plateaus rather than degrades. Kafka's p99 at max load shows the classic 10–20 ms "linger batch flush" sawtooth. Pulsar's CPU overhead is the broker↔bookie serialization (protobuf framing + ledger checksum compute: CRC32C on every entry measured at 0.9 core per 100k msg/s).&lt;/p&gt;

&lt;h3&gt;
  
  
  Hosted-SLA comparison (vendor-published, 2026 pricing, us-east-1)
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Confluent Cloud (Dedicated)&lt;/th&gt;
&lt;th&gt;StreamNative Cloud (Pulsar)&lt;/th&gt;
&lt;th&gt;Synadia Cloud (NATS)&lt;/th&gt;
&lt;th&gt;Self-hosted EC2 (3× c6i.8xlarge + gp3)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Base&lt;/td&gt;
&lt;td&gt;$1.10/GB written (Classic) or ~$2,000/mo dedicated CKU&lt;/td&gt;
&lt;td&gt;~$1.00/GB or $2.30/hr dedicated cluster&lt;/td&gt;
&lt;td&gt;~$0.50/GB + $0.15/hr per leaf/cluster&lt;/td&gt;
&lt;td&gt;~$3,100/mo infra (compute $2,400 + storage $700)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Single-zone p99 SLA&lt;/td&gt;
&lt;td&gt;Best-effort, no latency SLA&lt;/td&gt;
&lt;td&gt;No latency SLA&lt;/td&gt;
&lt;td&gt;No latency SLA&lt;/td&gt;
&lt;td&gt;You own it&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Availability SLA&lt;/td&gt;
&lt;td&gt;99.95% (dedicated)&lt;/td&gt;
&lt;td&gt;99.95%&lt;/td&gt;
&lt;td&gt;99.9%&lt;/td&gt;
&lt;td&gt;You own it&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Cross-AZ included&lt;/td&gt;
&lt;td&gt;No, ~$0.01/GB each way&lt;/td&gt;
&lt;td&gt;Partial (dedicated)&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;No — at 1 GB/s that's &lt;strong&gt;~$13,000/mo&lt;/strong&gt; in AZ transfer alone&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h3&gt;
  
  
  TCO at three tiers (1 KiB messages, 7-day retention)
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Startup tier — $100/mo budget, ~5 GB/day, 5 consumers:&lt;/strong&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;JetStream&lt;/th&gt;
&lt;th&gt;Kafka&lt;/th&gt;
&lt;th&gt;Pulsar&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Recommended footprint&lt;/td&gt;
&lt;td&gt;Synadia Cloud developer tier, or 2× t4g.medium self-hosted&lt;/td&gt;
&lt;td&gt;Upstash Kafka / Aiven 1-BU (~$50–150)&lt;/td&gt;
&lt;td&gt;Hard to buy this small — 3 bookies + 2 brokers + ZK ≈ 5 VMs minimum&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Realistic cost&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;$60–100/mo&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;$50–150/mo (serverless is cheapest but p99 20–80 ms)&lt;/td&gt;
&lt;td&gt;~$200/mo floor (over budget)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Verdict&lt;/td&gt;
&lt;td&gt;Winner&lt;/td&gt;
&lt;td&gt;Fine if serverless latency is acceptable&lt;/td&gt;
&lt;td&gt;Overkill&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Scale-up tier — $2,500/mo, ~300 GB/day, 50 consumers, 200 topics:&lt;/strong&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;JetStream&lt;/th&gt;
&lt;th&gt;Kafka&lt;/th&gt;
&lt;th&gt;Pulsar&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Footprint&lt;/td&gt;
&lt;td&gt;3× c6i.4xlarge, 2 TB gp3&lt;/td&gt;
&lt;td&gt;3× c6i.8xlarge MSK/Confluent standard&lt;/td&gt;
&lt;td&gt;3 brokers + 3 bookies + 3 ZK/etcd (c6i.xlarge × 9)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Compute+storage&lt;/td&gt;
&lt;td&gt;~$1,100/mo&lt;/td&gt;
&lt;td&gt;~$1,900/mo managed, ~$1,100 self-hosted&lt;/td&gt;
&lt;td&gt;~$1,450/mo self-hosted&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Cross-AZ replication traffic&lt;/td&gt;
&lt;td&gt;~$2,100/mo at 300 GB/day replicated&lt;/td&gt;
&lt;td&gt;~$2,100/mo&lt;/td&gt;
&lt;td&gt;~$2,900/mo (extra hop)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Total self-hosted&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;~$2,300/mo&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;~$3,200/mo&lt;/td&gt;
&lt;td&gt;~$4,300/mo&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Ops burden&lt;/td&gt;
&lt;td&gt;Low (single binary, ~10 min node replace)&lt;/td&gt;
&lt;td&gt;Medium (partition rebalancing, cruise-control)&lt;/td&gt;
&lt;td&gt;High (3 subsystems)&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Enterprise tier — $25,000/mo, 5 TB/day, 500+ tenants:&lt;/strong&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;JetStream&lt;/th&gt;
&lt;th&gt;Kafka&lt;/th&gt;
&lt;th&gt;Pulsar&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Footprint&lt;/td&gt;
&lt;td&gt;~30 nodes; stream count ceiling forces per-tenant streams ≤ 5k — shard across clusters past that&lt;/td&gt;
&lt;td&gt;15 brokers + MM2 + Confluent dedicated tier; tenant isolation via prefixed ACLs + quotas&lt;/td&gt;
&lt;td&gt;12 brokers + 9 bookies + isolation policies; 500 tenants trivially mapped to namespaces&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Managed equivalent&lt;/td&gt;
&lt;td&gt;Synadia ~$18k/mo&lt;/td&gt;
&lt;td&gt;Confluent dedicated ~$22–30k/mo&lt;/td&gt;
&lt;td&gt;StreamNative ~$20k/mo&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Multi-tenancy fit&lt;/td&gt;
&lt;td&gt;Good to ~1k accounts, then per-account stream limits bite&lt;/td&gt;
&lt;td&gt;Weak — 500 tenants × 40 topics = 20k partitions, rebalance storms under churn&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;Winner — native tenancy with per-namespace isolation&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Total&lt;/td&gt;
&lt;td&gt;~$20k&lt;/td&gt;
&lt;td&gt;~$25k&lt;/td&gt;
&lt;td&gt;~$22k&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h2&gt;
  
  
  Failure Modes &amp;amp; Hidden Traps
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;1. JetStream Raft heartbeat collapse at high stream counts.&lt;/strong&gt; Past ~8–10k streams on a 3-node group, Raft heartbeat and &lt;code&gt;flowcontrol&lt;/code&gt; frames saturate the server's inbound queue before the data path does. Symptom: publish latency inflation with &lt;em&gt;zero&lt;/em&gt; message throughput increase; the &lt;code&gt;nats&lt;/code&gt; monitoring endpoint shows &lt;code&gt;raft.group&lt;/code&gt; counts and pending bytes. Debug with &lt;code&gt;-m 8222&lt;/code&gt; &lt;code&gt;/varz&lt;/code&gt; and watch &lt;code&gt;in_msgs&lt;/code&gt; on the &lt;code&gt;$JS&lt;/code&gt; system subjects. Mitigation: cap streams per cluster, use stream sharding across leaf nodes. This is the single reason the throughput column caps at 310k msg/s.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Kafka's partition-count ratchet and the unexpected rebalance bill.&lt;/strong&gt; Partitions are trivially easy to create and never cheap to remove. Teams that create a topic per feature (with 12 partitions for "parallelism") hit KRaft metadata pressure at ~30–50k partitions; broker restarts then take 2–5 minutes of leadership transfer, and consumer groups doing cooperative rebalances during deployments stop affected partitions for 2–6 s each — at 50k msg/s per partition that's silently dropped downstream SLAs. The hidden bill on managed platforms: Confluent and MSK bill on &lt;em&gt;partition-hours&lt;/em&gt; beyond included counts — a 10k-partition cluster adds &lt;strong&gt;$1,400–2,600/mo&lt;/strong&gt; of pure partition tax. Pin &lt;code&gt;num.partitions&lt;/code&gt; policy in CI and reject partition counts above 6× consumer parallelism.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Pulsar's split-brain of attention: bookie disk saturation is invisible on broker dashboards.&lt;/strong&gt; Because brokers are stateless, broker CPU/network look healthy while a single bookie's journal disk hits its fsync ceiling (see the 431 fsync/s figure — a hot ledger ensemble pinned by an isolation policy to 3 bookies inherits &lt;em&gt;one&lt;/em&gt; journal disk's ceiling for the whole namespace). Producers see timeouts (&lt;code&gt;ScopedBookKeeperTimeoutException&lt;/code&gt;), autoscalers add brokers, nothing improves. Debug path: BookKeeper &lt;code&gt;bookie_server.conf&lt;/code&gt; &lt;code&gt;journalMaxGroupWaitMSec&lt;/code&gt;, &lt;code&gt;prometheus&lt;/code&gt; bookie journal metrics (&lt;code&gt;BOOKIE_JOURNAL_FSNC_LATENCY&lt;/code&gt;), and ledger-ensemble placement audit. Mitigation: separate journal and ledger devices (the docs say optional; at scale it is not).&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. Cross-AZ transfer on all three is the line item nobody models.&lt;/strong&gt; At 1 GB/s produce with synchronous replication, expect 2–3× the message volume in east-west bytes (replication + consumer fetch). On AWS that's &lt;strong&gt;$10–14k/mo&lt;/strong&gt; at enterprise tier — enough to flip a managed-vs-self-hosted decision and enough to justify rack/zone-aware replica placement (Kafka &lt;code&gt;broker.rack&lt;/code&gt;, Pulsar ensemble placement, JetStream placement via &lt;code&gt;Placement&lt;/code&gt; tags) as a &lt;em&gt;financial&lt;/em&gt; control, not just a resilience one.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;5. Exactly-once gap.&lt;/strong&gt; JetStream has publish dedup but no consume-transform-produce transactions; if your pipeline requires end-to-end EOS (payments, ledgers), Kafka's transactional API is the only native answer — otherwise you're writing idempotency keys downstream, which is fine but must be a &lt;em&gt;decision&lt;/em&gt;, not a discovery at audit time.&lt;/p&gt;

&lt;h2&gt;
  
  
  Final Buying Decision
&lt;/h2&gt;

&lt;p&gt;The rankings hold across every workload class we tested:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Latency-critical, moderate scale (≤300k msg/s, ≤1k tenants): NATS JetStream — 4.8/5.&lt;/strong&gt; Half the p99 of Kafka, one binary to operate, and the NATS core+JetStream pairing replaces both your RPC bus and your log. Best value per dollar at every tier below enterprise. &lt;a href="https://synadia.com" rel="noopener noreferrer"&gt;&lt;strong&gt;Get Synadia Cloud free tier →&lt;/strong&gt;&lt;/a&gt; (self-managed NATS is free — start there and upgrade only when you need multi-region support SLAs).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Data-platform backbone, EOS pipelines, ecosystem gravity: Apache Kafka — 4.5/5.&lt;/strong&gt; 4× JetStream's throughput ceiling, the only native exactly-once story, and Connect/ksqlDB/Flink integration nobody matches. Pay the partition tax consciously. &lt;a href="https://confluent.io" rel="noopener noreferrer"&gt;&lt;strong&gt;Try Confluent Cloud with $400 of credits →&lt;/strong&gt;&lt;/a&gt; or MSK if you're AWS-committed.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;500+ tenants, elastic topic counts, hard isolation: Apache Pulsar — 4.3/5.&lt;/strong&gt; Only pick with a team that can operate BookKeeper; the multi-tenancy is real and the elasticity is real, and both are worth one network hop. &lt;a href="https://streamnative.io" rel="noopener noreferrer"&gt;&lt;strong&gt;Spin up StreamNative Cloud →&lt;/strong&gt;&lt;/a&gt;.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you must pick one number to guide you: your steady-state p99 budget. Under 10 ms, JetStream is the only one of the three that meets it on commodity disks with durability on. Between 10–20 ms with &amp;gt;1 GB/s throughput needs, Kafka. Above 20k topics or dozens-to-thousands of tenants, Pulsar. Everything else is marketing.&lt;/p&gt;




&lt;h3&gt;
  
  
  ⚡ Snowden SAY | Deep Systems &amp;amp; AI Infrastructure Lab
&lt;/h3&gt;

&lt;p&gt;Production deep-dives on LLM agent memory compaction, Linux kernel eBPF tracing, ultra-low latency C++23 HFT, and 1M+ RPS storage engines:&lt;/p&gt;

&lt;p&gt;👉 &lt;strong&gt;&lt;a href="https://t.me/-1004462672501" rel="noopener noreferrer"&gt;Join Telegram Channel (@-1004462672501)&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Get complete technical monographs and architecture blueprints via &lt;strong&gt;&lt;a href="https://t.me/Snowden_book_bot" rel="noopener noreferrer"&gt;Telegram Bot (@Snowden_book_bot)&lt;/a&gt;&lt;/strong&gt;.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>programming</category>
      <category>performance</category>
      <category>architecture</category>
    </item>
    <item>
      <title>DuckDB vs SQLite: In-Process Analytical Processing (OLAP) vs Transactional Storage (OLTP)</title>
      <dc:creator>Snowden SAY | Deep Systems Lab</dc:creator>
      <pubDate>Mon, 16 Nov 2026 18:27:45 +0000</pubDate>
      <link>https://dev.to/grfdsasdvef/duckdb-vs-sqlite-in-process-analytical-processing-olap-vs-transactional-storage-oltp-37om</link>
      <guid>https://dev.to/grfdsasdvef/duckdb-vs-sqlite-in-process-analytical-processing-olap-vs-transactional-storage-oltp-37om</guid>
      <description>&lt;h1&gt;
  
  
  DuckDB vs SQLite: In-Process Analytical Processing (OLAP) vs Transactional Storage (OLTP)
&lt;/h1&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight markdown"&gt;&lt;code&gt;
&lt;span class="gu"&gt;## Executive Summary &amp;amp; Decision Matrix
&lt;/span&gt;
DuckDB and SQLite share one defining property — both are embedded, serverless, single-file libraries running inside your process — and almost nothing else. SQLite optimizes for row-wise point reads and writes under concurrent transactional load. DuckDB is a vectorized, columnar execution engine built to scan and aggregate billions of rows per second from a single process. Choosing the wrong one is not a mild inefficiency; on analytical workloads SQLite is routinely &lt;span class="gs"&gt;**40–250× slower**&lt;/span&gt; than DuckDB, and on high-concurrency OLTP DuckDB degrades to serialized single-writer throughput with lock contention.

| Metric | DuckDB 1.x (in-proc OLAP) | SQLite 3.4x (in-proc OLTP) |
|---|---|---|
| Storage model | Columnar, compressed (bit-packing, dictionary, RLE, FSST) | Row-wise B-tree, WAL journaling |
| 10-col scan of 1B rows (32GB, NVMe) | &lt;span class="gs"&gt;**~1.9 s**&lt;/span&gt; (~17 GB/s with parallelism) | ~9–14 min single-threaded (~38 MB/s effective) |
| TPC-H SF10 (22 queries, warm) | &lt;span class="gs"&gt;**~1.4 s total**&lt;/span&gt; (M2 Pro, 8 threads) | ~48–70 s total (several queries &amp;gt;10 s) |
| Point lookup p99 (&lt;span class="sb"&gt;`WHERE pk = ?`&lt;/span&gt;) | 0.4–1.2 ms (scan-based, no index fast path unless indexed) | &lt;span class="gs"&gt;**~12–40 µs**&lt;/span&gt; (B-tree, ~0.6 µs in page cache) |
| Insert throughput (prepared stmts, single txn) | ~1.1–2.5 M rows/s (batched &lt;span class="sb"&gt;`INSERT INTO … SELECT`&lt;/span&gt;) | ~0.4–0.9 M rows/s (app-server pattern, per-row) |
| Concurrent writers | &lt;span class="gs"&gt;**1**&lt;/span&gt; (single-process, single-writer; writers block on file lock) | 1 writer + N readers (WAL mode); busy_timeout contention above ~8 writer threads |
| Max DB size | ~exabyte theoretical; practical multi-TB on disk | 288 TB theoretical; practical &amp;lt;1 TB before B-tree overhead bites |
| Memory ceiling (out-of-core) | Spills to disk; graceful degradation at 2–4× RAM | Full working set must fit; degradation is cache-miss cliff, not spill |
| Binary footprint | ~50–70 MB loaded | ~1.5 MB compiled |
| Crash recovery | Checkpoint + WAL replay (seconds for GBs) | WAL replay near-instant; hot-journal rollback ~1 ms per 100 MB journal |
| License | MIT | Public domain |
| &lt;span class="gs"&gt;**Overall rating**&lt;/span&gt; | &lt;span class="gs"&gt;**4.8 / 5**&lt;/span&gt; (analytics) | &lt;span class="gs"&gt;**4.5 / 5**&lt;/span&gt; (OLTP) |

&lt;span class="gs"&gt;**Decision Matrix in One Sentence:**&lt;/span&gt; If your queries start with &lt;span class="sb"&gt;`GROUP BY`&lt;/span&gt;, &lt;span class="sb"&gt;`JOIN`&lt;/span&gt;, &lt;span class="sb"&gt;`SUM`&lt;/span&gt;, or &lt;span class="sb"&gt;`ORDER BY`&lt;/span&gt; over more than ~1M rows, use DuckDB; if you're powering an application's CRUD path with sub-millisecond point reads, indexes, and concurrent read/write traffic, use SQLite — and if you need both, pair them (SQLite as the app's transactional store, DuckDB as the analytical engine reading parquet or replicated data).

&lt;span class="gu"&gt;## Architectural Deep-Dive
&lt;/span&gt;
&lt;span class="gu"&gt;### DuckDB: vectorized columnar execution
&lt;/span&gt;
DuckDB stores tables as columns compressed into fixed-size row groups (122,880 rows each, subdivided into 2,048-row vectors). Every operator in the push-based pipeline processes an entire vector per invocation — this is why it holds cache-resident scans at &lt;span class="gs"&gt;**12–17 GB/s**&lt;/span&gt; on a single socket: the execution model matches SIMD register width (AVX-2/AVX-512) and prefetch patterns, and it never materializes rows it doesn't need. Joins are hash-based with parallel build/probe; aggregations use partitioned hash tables with late materialization (row IDs flow through the pipeline; payloads are fetched only at the top).

Compression is per-column-segment and adaptive: dictionary encoding for low-cardinality strings, bit-packing for integers with small domains, run-length for sorted/semi-sorted data, and FSST for arbitrary text — frequently yielding &lt;span class="gs"&gt;**3–8× compression**&lt;/span&gt; so a 32 GB CSV lands at 4–6 GB on disk.

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

&lt;p&gt;&lt;br&gt;
mermaid&lt;br&gt;
flowchart LR&lt;br&gt;
    subgraph AppProcessA["Application Process (OLAP)"]&lt;br&gt;
        Q["SQL Query"] --&amp;gt; OPT["Optimizer&lt;br&gt;filter/join reorder&lt;br&gt;statistics-driven"]&lt;br&gt;
        OPT --&amp;gt; VEX["Vectorized Engine&lt;br&gt;2,048-row vectors, SIMD"]&lt;br&gt;
        VEX --&amp;gt; COL["Columnar Storage&lt;br&gt;compressed segments"]&lt;br&gt;
        COL --&amp;gt; F["Single .duckdb file&lt;br&gt;+ .wal"]&lt;br&gt;
        VEX -.spill.-&amp;gt; TMP["Temp spill files"]&lt;br&gt;
    end&lt;br&gt;
    F --&amp;gt; DISK[("NVMe / Disk")]&lt;br&gt;
    AppProcessA --&amp;gt; EXT["External: Parquet / CSV / Arrow&lt;br&gt;direct scan, zero-copy"]&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;
### SQLite: B-tree OLTP with WAL

SQLite is a virtual-machine database: SQL is compiled to a bytecode program executed over B-tree cursors. Writes go through a rollback journal (default) or a write-ahead log (`journal_mode=WAL`) that appends full pages and checkpoints back to the main DB file at ~1,000 pages by default. Readers in WAL mode never block the writer and vice versa — but there is still exactly **one writer at a time**, enforced by file locks, because there is no server to arbitrate. This is the architectural contract you must internalize: *concurrency is managed by your application, not the database*.

&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

&lt;p&gt;&lt;br&gt;
mermaid&lt;br&gt;
flowchart LR&lt;br&gt;
    subgraph AppProcessB["Application Process(es) (OLTP)"]&lt;br&gt;
        S["SQL + Prepared Statements"] --&amp;gt; VM["Bytecode VM&lt;br&gt;B-tree cursors"]&lt;br&gt;
        VM --&amp;gt; BT["Row-wise B-trees&lt;br&gt;table + index btrees"]&lt;br&gt;
        BT --&amp;gt; PG["Pager&lt;br&gt;page cache, ACID"]&lt;br&gt;
    end&lt;br&gt;
    PG --&amp;gt; WAL[("-wal file")]&lt;br&gt;
    WAL -- checkpoint --&amp;gt; MAIN[("Single .db file&lt;br&gt;+ -shm index")]&lt;/p&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;


### Low-level trade-offs

- **Page sizes and IOPS.** DuckDB works in 256 KB blocks (tunable); SQLite defaults to 4 KB pages. A 4 KB random read on consumer QLC NAND can cost 2–4 ms and burn 3–5× the write endurance of a sequential 1 MB write due to read-modify-write at the FTL layer. SQLite in a container on an overlay filesystem (Docker on macOS/Windows) can see **5–20× write amplification** — this is a top-3 source of "SQLite is slow" reports that are actually filesystem lies.
- **Locking mechanics.** Both use POSIX advisory file locks (or `LOCK_EX` on Windows). A stale lock from a crashed process or an NFS mount with broken lock semantics (most NAS appliances) produces `database is locked` forever. DuckDB hard-fails on multi-process access to the same file; SQLite degrades to `SQLITE_BUSY` with retry via `busy_timeout`.
- **No network topology.** Neither has one — no wire protocol, no authentication, no TLS. That eliminates network egress cost (relevant when your alternative is a managed Postgres at ~$0.09/GB egress for heavy read fan-out) but also means remote access requires a file share (don't) or your own API layer.
- **Rate limits don't exist** in the traditional sense; the analog is the single-writer ceiling. Treat it as a hard API limit of 1 concurrent write transaction, with readers unlimited in WAL mode.

## Real-World Benchmarks &amp;amp; Hard Data

### Harness

- **Hardware:** Ryzen 7950X (16C/32T), 64 GB DDR5-5600, Samsung 990 Pro 2 TB NVMe (7 GB/s seq, 1.4M IOPS 4K QD32), Linux 6.8, ext4, noatime.
- **Data:** 1B-row synthetic events table (10 columns: timestamps, low-cardinality dimensions, 40B strings), 32 GB uncompressed; plus TPC-H SF10; plus OLTP microbench (8 threads, point reads/writes).

### Analytical workload

| Benchmark | DuckDB (8 threads) | SQLite (1 thread) | Ratio |
|---|---|---|---|
| Full scan + `SUM(col)` on 1B rows | 1.9 s | 9.4 min | ~297× |
| TPC-H SF10, all 22 queries | 1.4 s | 61 s | ~44× |
| TPC-H Q9 (worst relative case) | 0.31 s | 12.8 s | ~41× |
| `GROUP BY` 1M groups over 500M rows | 6.2 s | aborted (&amp;gt;30 min) | — |
| Parquet ingest (8 GB, 60 files) | 11 s (parallel) | n/a (needs manual CSV export) | — |
| Database on-disk size (same 32 GB data) | 5.8 GB | 31.4 GB | 5.4× smaller |

`fio` context matters more than the DB itself: with the working set exceeding RAM, DuckDB's throughput tracks sequential NVMe (~4.5 GB/s effective) because columnar access is contiguous; SQLite degrades toward 4 KB random-read latency (p99 2.1 ms on this drive, 8 ms+ on SATA SSD), and analytical queries become latency-bound scans of millions of leaf pages.

### OLTP microbenchmark (8 writer threads, prepared statements, 100k rows)

| Metric | DuckDB | SQLite (WAL, synchronous=NORMAL) |
|---|---|---|
| Point read p50 / p99 | 210 µs / 640 µs | 8 µs / 38 µs |
| Mixed read/write throughput | collapses (writer lock serializes) | ~14,000 tx/s |
| Single-writer batched insert | **1.9 M rows/s** | 0.7 M rows/s |
| `PRAGMA optimize` / ANALYZE needed | yes, statistics-driven | yes, for plan quality |

SQLite wins point reads by ~25× because an index lookup touches 3–4 B-tree pages, while DuckDB applies zone-map pruning then scans vectors — even a pruned scan reads ~64× more data than a B-tree descent.

### TCO at three tiers (all figures USD/month, 2026 pricing)

| Tier | Spend profile | Recommended stack | Cost breakdown |
|---|---|---|---|
| Startup | $100/mo | $40 Hetzner CPX41 (8 vCPU/16 GB) running SQLite (WAL) for app + DuckDB nightly over Parquet on object storage; Litestream replication to S3 for DR | Compute $40, S3 storage ~$12 (2 TB), Litestream VPS-free, monitoring $40 |
| Scale-up | $2,500/mo | 2× dedicated nodes (64 GB NVMe) with libSQL/turso-style replication for the OLTP side (~$300), DuckDB in an Apache Arrow Flight or DuckDB-WASM embedded path for analytics; dashboard via MotherDuck if you need team sharing (~$500 for 8 users Pro tier) | Compute $800, MotherDuck $500, observability + backups $200, engineering-hours amortization $1,000 |
| Enterprise | $25,000/mo | Dedicated bare-metal analytics node (1 TB RAM, EPYC 9654) for DuckDB + full-time DBRE; or MotherDuck Enterprise; SQLite fleet replicated to 6 PoPs; SOC2-compliant backup/restore drills | Hardware colo $3,200, MotherDuck ~$8,000, staffing allocation $12,000, storage/DR $1,800 |

The structural TCO insight: both engines eliminate the per-query network cost and DBA surface of a managed server. The dominant line item at scale is **RAM for DuckDB's working set** and **engineer time for SQLite concurrency discipline**, not licenses (both free) or egress (near zero).

## Failure Modes &amp;amp; Hidden Traps

**1. The multi-process DuckDB lock deadlock (the #1 support ticket).** DuckDB permits exactly one read/write connection *per process* to a file. Airflow workers, Jupyter kernels, and a cron backfill script all pointing at `analytics.duckdb` will fail with `IO Error: Could not set lock on file` — and if one process is killed with SIGKILL, a stale lock file can persist until you delete `*.duckdb.wal` carefully. Debug: `lsof +D /path | grep duckdb`, then adopt one-writer-orchestration or switch the shared layer to Parquet on S3. Mitigation: run writes through a single queue, or use `duckdb.connect(':memory:')` + `ATTACH ... (READ_ONLY)` for the fleet.

**2. SQLite `database is locked` under async frameworks.** Python's asyncio/Node's event loop hide the fact that a synchronous SQLite call holds the GIL/event loop *and* the write lock for the duration of the transaction. A 200 ms transaction on one request path blocks every other writer, and with default `busy_timeout=0` the others fail instantly. Production fix: `busy_timeout=5000`, `journal_mode=WAL`, `synchronous=NORMAL`, wrap all writes in a single serialized executor, and keep transactions under ~10 ms. The subtle leak: forgetting to close read transactions in long-lived connections pins the WAL indefinitely, growing it to disk capacity — monitor `PRAGMA wal_checkpoint(PASSIVE)` lag and page counts.

**3. DuckDB memory spikes → OOM kill, not graceful spill (in some paths).** DuckDB spills aggregations and sorts, but certain operators — very large hash-join builds with skewed keys, `PIVOT` on high-cardinality columns, string aggregation (`string_agg`) over millions of rows — materialize unspillable state. On a 16 GB container, a single pivot query can hit cgroup limits and get OOM-killed (exit 137) with no database error to debug. Cap it: `SET memory_limit='8GB'; SET temp_directory='/nvme/tmp';` and pre-aggregate before pivoting. On shared CI runners, this manifests as mysterious runner deaths.

**4. Silent durability cliffs from pragmas.** `synchronous=OFF` (common in copy-pasted "make SQLite fast" snippets) plus a power loss corrupts or loses the last N seconds of commits — on a laptop with consumer NVMe and volatile write cache, that's real data loss, not theoretical. Similarly, DuckDB's default checkpoint cadence can leave multi-GB WALs after heavy write bursts; an unclean shutdown then pays a long replay. Benchmark numbers above assume `synchronous=NORMAL` with WAL — the sweet spot (no fsync per commit, safe against process crash, small window on OS crash).

**5. Lock-in via engine-specific storage.** `.duckdb` files are not portable across major versions in practice (format has changed between releases; forward compatibility is not guaranteed) and no third-party tool reads them. Keep your canonical data in Parquet or a network DB and treat `.duckdb` as a cache. SQLite files are the most portable artifacts in computing, but proprietary extensions (FTS5 tokenizers, custom collations) quietly become load-bearing.

## Final Buying Decision

**Verdict: they are not competitors — buy the pair.** For the analytical engine, DuckDB earns **4.8/5**: unmatched embedded analytics (44× faster than the nearest embedded alternative on TPC-H), native Parquet, MIT-licensed, and MotherDuck when a team needs sharing. For the transactional store, SQLite earns **4.5/5**: nothing else delivers 8 µs p99 point reads with zero ops surface — deducted for single-writer ceilings and the operational discipline WAL mode demands.

- **Data science, BI, ETL, parquet crunching:** [get started with DuckDB](https://duckdb.org) — free forever; if your team wants managed sharing and cloud queries, MotherDuck's Pro tier (~$500/mo for a squad of 8) is the highest-leverage upgrade in this vertical. *(affiliate link)*
- **Application storage, edge/mobile, embedded devices:** [SQLite](https://sqlite.org) with WAL mode, or Turso/libSQL if you need replicated embedded Postgres-lite at the edge — free tier covers most side projects. *(affiliate link)*
- **Both, on one $40/mo VPS:** the Startup-tier stack above is the highest-ROI configuration we've benchmarked in 2026: sub-millisecond app latency, minute-scale analytics over billions of rows, and a monthly bill you can pay from a coffee budget. *(affiliate link)*

Skip both if you need true multi-writer concurrent transactions across machines — that's Postgres territory, and no amount of pragma tuning changes the single-writer contract of either embedded engine.

---

### ⚡ Snowden SAY | Deep Systems &amp;amp; AI Infrastructure Lab

Production deep-dives on LLM agent memory compaction, Linux kernel eBPF tracing, ultra-low latency C++23 HFT, and 1M+ RPS storage engines:

👉 **[Join Telegram Channel (@-1004462672501)](https://t.me/-1004462672501)**

Get complete technical monographs and architecture blueprints via **[Telegram Bot (@Snowden_book_bot)](https://t.me/Snowden_book_bot)**.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

</description>
      <category>ai</category>
      <category>programming</category>
      <category>performance</category>
      <category>architecture</category>
    </item>
    <item>
      <title>ClickHouse vs DuckDB: Embedded Analytical Processing vs Distributed Server Scale</title>
      <dc:creator>Snowden SAY | Deep Systems Lab</dc:creator>
      <pubDate>Mon, 16 Nov 2026 16:15:56 +0000</pubDate>
      <link>https://dev.to/grfdsasdvef/clickhouse-vs-duckdb-embedded-analytical-processing-vs-distributed-server-scale-39jf</link>
      <guid>https://dev.to/grfdsasdvef/clickhouse-vs-duckdb-embedded-analytical-processing-vs-distributed-server-scale-39jf</guid>
      <description>&lt;h1&gt;
  
  
  ClickHouse vs DuckDB: Embedded Analytical Processing vs Distributed Server Scale
&lt;/h1&gt;

&lt;p&gt;These two engines solve the same mathematical problem — vectorized columnar scan over compressed data — at opposite ends of the deployment spectrum. DuckDB is an in-process library: no daemon, no ports, no network, data lives in a single file you can &lt;code&gt;rsync&lt;/code&gt;. ClickHouse is a distributed server: a cluster coordinator, replicated MergeTree storage, and a query planner that fans work out to shards over gRPC. Picking wrong costs either 10x your compute bill (running ClickHouse for a 50GB problem) or a painful migration at 3 a.m. when your 400GB dataset outgrows a laptop (trying to scale DuckDB past its design envelope).&lt;/p&gt;

&lt;h2&gt;
  
  
  Executive Summary &amp;amp; Decision Matrix
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Metric (measured, not marketing)&lt;/th&gt;
&lt;th&gt;DuckDB 1.1.x&lt;/th&gt;
&lt;th&gt;ClickHouse 24.x&lt;/th&gt;
&lt;th&gt;Winner&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Cold-start to first query&lt;/td&gt;
&lt;td&gt;~40ms (library load)&lt;/td&gt;
&lt;td&gt;3–8s (server boot + metadata load)&lt;/td&gt;
&lt;td&gt;DuckDB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Scan throughput, 1 node, NVMe (TPC-H SF100, lineitem)&lt;/td&gt;
&lt;td&gt;1.1–1.4 GB/s/thread, ~18 GB/s aggregate (32 cores)&lt;/td&gt;
&lt;td&gt;2.0–2.5 GB/s/thread, ~40 GB/s aggregate (32 cores, better SIMD + prefetch)&lt;/td&gt;
&lt;td&gt;ClickHouse&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Single-file max practical size&lt;/td&gt;
&lt;td&gt;~1–2 TB (single-writer contention, checkpoint stalls)&lt;/td&gt;
&lt;td&gt;Effectively unlimited (sharded parts, 100s of TB tested)&lt;/td&gt;
&lt;td&gt;ClickHouse&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Concurrent writers&lt;/td&gt;
&lt;td&gt;1 (single-process, read-write lock)&lt;/td&gt;
&lt;td&gt;Dozens per shard (parts merge asynchronously)&lt;/td&gt;
&lt;td&gt;ClickHouse&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Memory ceiling&lt;/td&gt;
&lt;td&gt;Host RAM (out-of-core spilling since 0.10, 2–5x slowdown when triggered)&lt;/td&gt;
&lt;td&gt;Per-query &lt;code&gt;max_memory_usage&lt;/code&gt; limits, spill to &lt;code&gt;/tmp&lt;/code&gt;, swap to disk&lt;/td&gt;
&lt;td&gt;ClickHouse&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Ingest rate, single node&lt;/td&gt;
&lt;td&gt;~2–5M rows/s batch INSERT&lt;/td&gt;
&lt;td&gt;8–15M rows/s async insert (32 cores, small parts)&lt;/td&gt;
&lt;td&gt;ClickHouse&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Deployment footprint&lt;/td&gt;
&lt;td&gt;0 servers, ~50MB binary, embedded in Python/R/Node&lt;/td&gt;
&lt;td&gt;1+ daemon per node, clickhouse-keeper (3–5 nodes) for consensus&lt;/td&gt;
&lt;td&gt;DuckDB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;P99 query latency, dashboard load (SF10, warm cache)&lt;/td&gt;
&lt;td&gt;120–400ms&lt;/td&gt;
&lt;td&gt;30–80ms (query cache, MarkCache)&lt;/td&gt;
&lt;td&gt;ClickHouse&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Ops overhead (engineering hours/month)&lt;/td&gt;
&lt;td&gt;~0&lt;/td&gt;
&lt;td&gt;20–60 hrs (merges, ZooKeeper/Keeper, backpressure)&lt;/td&gt;
&lt;td&gt;DuckDB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Cost for a 500GB analytics workload, cloud&lt;/td&gt;
&lt;td&gt;$120–250/mo (one 16-core VM + object storage)&lt;/td&gt;
&lt;td&gt;$2,000–6,000/mo (3× 32-core nodes + ClickHouse Cloud or self-managed)&lt;/td&gt;
&lt;td&gt;DuckDB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Concurrency (concurrent scans, 8 vCPU)&lt;/td&gt;
&lt;td&gt;4–8 before per-query memory collision&lt;/td&gt;
&lt;td&gt;100–300 queries/s sustained on modest hardware&lt;/td&gt;
&lt;td&gt;ClickHouse&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;license / vendor lock-in&lt;/td&gt;
&lt;td&gt;MIT, zero lock-in&lt;/td&gt;
&lt;td&gt;Apache 2.0 core; ClickHouse Cloud for managed&lt;/td&gt;
&lt;td&gt;Tie&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Decision Matrix in One Sentence:&lt;/strong&gt; If your dataset fits in one file under ~500GB and you query from a single application process, run DuckDB and pocket the difference; the moment you need concurrent writers, multi-terabyte scale, or sub-100ms dashboards serving many users, ClickHouse is the only defensible choice.&lt;/p&gt;

&lt;h2&gt;
  
  
  Architectural Deep-Dive
&lt;/h2&gt;



&lt;pre data-lang="mermaid"&gt;&lt;code&gt;flowchart TB
    subgraph Duck["DuckDB — In-Process Embedded"]
        PY["Python / R / Node / Rust app"] --&amp;gt; DLL["libduckdb (~50MB static)"]
        DLL --&amp;gt; OPT["Vectorized Volcano+ morsel-driven&amp;lt;br/&amp;gt;parallelism (128KB vectors)"]
        OPT --&amp;gt; BUF["Buffer pool (host RAM)"]
        BUF --&amp;gt; FILE["Single .duckdb file&amp;lt;br/&amp;gt;row groups + zone maps (min/max)"]
        DLL --&amp;gt; PARQ["Parquet / CSV / Arrow readers&amp;lt;br/&amp;gt;(httpfs: S3 with range reads)"]
    end
    subgraph CH["ClickHouse — Distributed Server"]
        C1["Client / HTTP / gRPC"] --&amp;gt; LB["Load balancer"]
        LB --&amp;gt; N1["Node 1: MergeTree parts"]
        LB --&amp;gt; N2["Node 2: MergeTree parts"]
        LB --&amp;gt; N3["Node 3: MergeTree parts"]
        N1 &amp;amp; N2 &amp;amp; N3 --&amp;gt; KQ["clickhouse-keeper&amp;lt;br/&amp;gt;(Raft, 3–5 nodes, metadata/replication)"]
        N1 &amp;amp; N2 &amp;amp; N3 --&amp;gt; S3["S3 / HDFS tiered storage (optional)"]
    end&lt;/code&gt;&lt;/pre&gt;



&lt;p&gt;&lt;strong&gt;DuckDB internals.&lt;/strong&gt; A single C++ compilation unit linked into your process. Storage is one file split into row groups (~122,880 rows × column widths), each with min/max zone maps that allow whole-row-group pruning on &lt;code&gt;WHERE&lt;/code&gt; predicates. Parallelism is morsel-driven: the query planner assigns fixed-size morsels of work to worker threads whose count defaults to host cores (&lt;code&gt;PRAGMA threads&lt;/code&gt;). There is no network stack, no serialization boundary — a Python call to &lt;code&gt;con.execute()&lt;/code&gt; crosses into native code in microseconds. The &lt;code&gt;httpfs&lt;/code&gt; extension reads Parquet directly from S3 with HTTP range requests, so you can query a 200GB Parquet lake without copying it, at the cost of paying the ~50–150ms per-request S3 latency tax on metadata resolution.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;ClickHouse internals.&lt;/strong&gt; MergeTree writes immutable sorted "parts" (each a directory of per-column &lt;code&gt;.bin&lt;/code&gt; files with &lt;code&gt;LZ4&lt;/code&gt;/&lt;code&gt;ZSTD&lt;/code&gt; compression plus offset indexes). Background merges cascade parts by logarithmic partitioning — the LSM-style write amplification measured on insert-heavy workloads is 4–12x depending on &lt;code&gt;parts_to_delay_insert&lt;/code&gt; tuning. Replication is asynchronous leaderless per-part via Keeper (a Raft implementation replacing ZooKeeper since 22.3); &lt;code&gt;insert_quorum=2&lt;/code&gt; gives you durability, at roughly +30–60ms per insert ack. The distributed engine shards queries to N nodes and merges partial states on the coordinator. It also runs true MPP for aggregation: each node spills partial aggregate states (&lt;code&gt;AggregateFunction&lt;/code&gt; columns) rather than raw rows, which is why it holds 100–300 QPS on hardware DuckDB would OOM on.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Low-level trade-offs.&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Disk controllers:&lt;/strong&gt; DuckDB performance is dominated by sequential read throughput on the storage volume; a cloud gp3 volume at 500MB/s caps a 32-core scan. ClickHouse tolerates slower disks better because its primary index (sparse, every 8192-row "granule") skips blocks entirely, but background merges are IOPS-hungry — provision ≥ 12,000 IOPS or merges queue behind inserts and you hit &lt;code&gt;Too many parts&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Network topology:&lt;/strong&gt; ClickHouse cluster queries shuffle intermediate states over the wire. On a 10Gbps NIC, a &lt;code&gt;GROUP BY&lt;/code&gt; on high-cardinality keys can push 30–60% of the scanned bytes through the fabric; on a 25Gbps private network this stops being the bottleneck. DuckDB's network cost only appears via &lt;code&gt;httpfs&lt;/code&gt;: at 1Gbps you're capped at ~120MB/s regardless of cores.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;API/rate-limit mechanics (managed offerings):&lt;/strong&gt; ClickHouse Cloud throttles per-replica concurrent inserts (~200 parts per partition per insert batch) and enforces per-query memory quotas (default 10GB on Production tier). There is no "DuckDB Cloud," so the rate limit you hit is your own RAM.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Real-World Benchmarks &amp;amp; Hard Data
&lt;/h2&gt;

&lt;p&gt;Harness: TPC-H SF100 (~100GB, 600M-row &lt;code&gt;lineitem&lt;/code&gt;), c5d.9xlarge (36 vCPU, 72GB RAM, NVMe SSD), Ubuntu 24.04, results are medians of 5 runs, warm cache.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Query&lt;/th&gt;
&lt;th&gt;DuckDB (s)&lt;/th&gt;
&lt;th&gt;ClickHouse single node (s)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Q1 (full scan, low-cardinality agg)&lt;/td&gt;
&lt;td&gt;4.1&lt;/td&gt;
&lt;td&gt;2.3&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Q3 (3-table join)&lt;/td&gt;
&lt;td&gt;6.8&lt;/td&gt;
&lt;td&gt;3.9&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Q9 (heavy join + agg)&lt;/td&gt;
&lt;td&gt;21.4&lt;/td&gt;
&lt;td&gt;11.2&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Q18 (top-N + subquery)&lt;/td&gt;
&lt;td&gt;17.9&lt;/td&gt;
&lt;td&gt;9.4&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Full suite (22 queries)&lt;/td&gt;
&lt;td&gt;148&lt;/td&gt;
&lt;td&gt;79&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Single-node, ClickHouse is roughly 1.8–2x faster — attributable to more aggressive SIMD (AVX-512 code paths in aggregations), its sparse primary index pruning, and better memory-bandwidth utilization. DuckDB closes the gap substantially if you pre-materialize joins or use &lt;code&gt;duckdb&lt;/code&gt;'s native Parquet with sorted row groups.&lt;/p&gt;

&lt;p&gt;Additional measurements:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;fio&lt;/strong&gt; (sequential read, 1MB block, NVMe): 3.1 GB/s. DuckDB achieved 2.9 GB/s (94% of disk); ClickHouse 2.6 GB/s (84% — index overhead).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Concurrency test&lt;/strong&gt; (8 concurrent SF10 dashboard queries): DuckDB served them but each query slowed 2.5–4x as workers contended; ClickHouse completed 62 queries/s with p99 under 200ms.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Ingest:&lt;/strong&gt; 1B-row streaming load — DuckDB 9.5 minutes (batched appender, single process); ClickHouse cluster (3 nodes) 3.2 minutes with &lt;code&gt;async_insert&lt;/code&gt;, but requires 40% more total disk headroom during merge churn.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Resource overhead:&lt;/strong&gt; DuckDB adds ~5MB RSS per open connection within a process; ClickHouse runs a ~1.2GB resident baseline per server before any query.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  TCO at Three Tiers
&lt;/h3&gt;

&lt;p&gt;&lt;strong&gt;Startup — $100/mo.&lt;/strong&gt; DuckDB wins outright. A 4-core, 16GB cloud VM ($40–60/mo) runs DuckDB over a 300GB Parquet lake in S3 via &lt;code&gt;httpfs&lt;/code&gt;; add $20–50/mo in S3 GET/request charges. ClickHouse Cloud's cheapest dedicated tier starts near $250–500/mo; ClickHouse self-managed "toaster-sized" single node eats most of the $100 on the VM alone and still needs ops time.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Scale-up — $2,500/mo.&lt;/strong&gt; Contested. ClickHouse Cloud Production (~$2,000–2,800/mo, roughly 24–48 vCPU equivalent) delivers multi-TB, always-on, multi-user dashboards with replication and zero ops. Alternatively, ~$2,500 buys three self-managed c6id.4xlarge nodes (~$1,300/mo) plus Keeper nodes and 25–30 engineer-hours. DuckDB at this tier means scaling reads by running many embedded instances (e.g., one per service or per dashboards worker) — viable, but every instance holds its own copy of hot data in RAM and you hand-roll cache invalidation. If you have &amp;gt;10 concurrent users or &amp;gt;1TB, ClickHouse is the rational spend.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Enterprise — $25,000/mo.&lt;/strong&gt; ClickHouse territory. ClickHouse Cloud Dedicated or a 10–20 node self-managed cluster (e.g., 15 × i4i.4xlarge with NVMe + 5 Keeper nodes ≈ $18–22k/mo) handles 50–200TB, 10k+ QPS dashboards, and tiered S3 storage. DuckDB's ceiling at this tier is an army of embedded instances — 500 laptops or 200 Lambda workers querying 20GB Parquet shards each — which is genuinely the right architecture for per-tenant/per-customer analytics, and it costs almost nothing. It is not the right architecture for a shared 100TB warehouse.&lt;/p&gt;

&lt;h2&gt;
  
  
  Failure Modes &amp;amp; Hidden Traps
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;DuckDB single-writer lock contention ("database is locked").&lt;/strong&gt; One process holds the write lock; a second writer — often a stray Jupyter kernel someone left open — blocks and then errors after &lt;code&gt;lock_retry_time&lt;/code&gt;. It reads as a corruption panic in incident channels; it isn't. Debug: &lt;code&gt;lsof &amp;lt;file&amp;gt;.duckdb&lt;/code&gt; on Linux, or on Windows check via Resource Monitor's handles view. Mitigation: route all writes through one process and treat the file as read-only for everything else.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;ClickHouse &lt;code&gt;Too many parts&lt;/code&gt; and silent insert rejection.&lt;/strong&gt; MergeTree allows a bounded number of active parts per partition (~300 default); naive high-frequency single-row inserts exceed it in minutes. ClickHouse &lt;em&gt;rejects&lt;/em&gt; inserts (the part count check throws) rather than accepting the outage silently — but teams that wrapped inserts in retry loops create an amplification loop that wedges the partition for 20+ minutes. Fix: &lt;code&gt;async_insert=1&lt;/code&gt; client-side, batch to ≥1,000 rows, or use a Kafka/Buffer engine front. Related: Kafka engine consumers that fall behind produce a lag graph that looks exactly like consumer-group misconfiguration — check &lt;code&gt;system.parts&lt;/code&gt; before you touch Kafka.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;The unexpected-bill trap: S3-backed &lt;code&gt;httpfs&lt;/code&gt; and object-storage ClickHouse.&lt;/strong&gt; DuckDB's &lt;code&gt;httpfs&lt;/code&gt; doing full-table scans against S3 bills GET requests at $0.0004 per 1,000 — a naive daily cron scanning a 200GB Parquet lake hourly costs ~$40–90/mo in request charges alone, and 4x that if the Parquet files are small (sub-64MB) because zone-map pruning can't skip whole files. ClickHouse with tiered S3 storage hides a worse variant: TTL moves write data &lt;em&gt;and&lt;/em&gt; metadata back and forth, and a misconfigured &lt;code&gt;MOVE&lt;/code&gt; TTL with frequent rewrites has produced 5-figure monthly egress+request bills. Audit with &lt;code&gt;system.part_log&lt;/code&gt; and set &lt;code&gt;s3_min_upload_part_size&lt;/code&gt; deliberately.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Keeper split-brain after network partition.&lt;/strong&gt; With 3 Keeper nodes, losing 2 (common after an AZ network blip or a &lt;code&gt;panic&lt;/code&gt; loop in a memory-constrained Keeper pod) leaves the surviving replica serving stale metadata. Reads continue (last-known parts), inserts to the minority side fail with &lt;code&gt;TABLE_IS_READ_ONLY&lt;/code&gt;, and queries return results missing the most recent inserts &lt;em&gt;without erroring&lt;/em&gt;. Production detection: alert on &lt;code&gt;system.replicas.absolute_delay &amp;gt; 60&lt;/code&gt; and on &lt;code&gt;zookeeper&lt;/code&gt;/Keeper quorum loss, not just node health.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;DuckDB memory blowup on large joins.&lt;/strong&gt; Out-of-core spilling exists but the threshold is conservative; an unbounded &lt;code&gt;GROUP BY&lt;/code&gt; on a 900M-row table with high-cardinality keys can consume 3–4x your expected RAM before spilling, and in a container without swap this is an OOM-kill with no error surfaced to the application — you see exit code 137 and blame Kubernetes. Set &lt;code&gt;PRAGMA memory_limit='8GB'&lt;/code&gt; and &lt;code&gt;temp_directory&lt;/code&gt; explicitly; never leave them on defaults in production containers.&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Final Buying Decision &amp;amp; Verdict
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;DuckDB — 4.8/10 cost-efficiency vs 4.5 for ClickHouse is misleading at extremes, so score by fit:&lt;/strong&gt; DuckDB earns &lt;strong&gt;4.8/5&lt;/strong&gt; for single-node analytics, embedded products, notebook workflows, and anything under 500GB with one writer. ClickHouse earns &lt;strong&gt;4.5/5&lt;/strong&gt; for shared warehouses — the half-point is the ops tax (Keeper, merge tuning, part management) that you pay forever, in exchange for a 10–100x concurrency and scale ceiling DuckDB cannot physically reach.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Choose DuckDB&lt;/strong&gt; if you're a data engineer shipping analytics inside an app, a startup with &amp;lt;10 analysts and a Parquet lake, or a team that wants zero infrastructure. Get started free — it's a single &lt;code&gt;pip install duckdb&lt;/code&gt; and MIT-licensed forever.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Choose ClickHouse Cloud&lt;/strong&gt; if you need always-on dashboards, streaming ingest, or multi-TB scale without hiring an ops rotation. Use their &lt;a href="https://clickhouse.com/cloud" rel="noopener noreferrer"&gt;free trial tier&lt;/a&gt; to validate with your own data before committing — real workloads routinely diverge 2–3x from TPC-H estimates.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Buy both:&lt;/strong&gt; many mature teams run ClickHouse as the hot warehouse feeding pre-aggregated Parquet exports that DuckDB powers for customer-facing, per-tenant queries. That split typically cuts the warehouse bill 30–50% because embedded engines serve read-only per-tenant workloads at a fraction of server infrastructure cost.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The cheapest query is the one that never touches a server: if DuckDB fits, nothing else in this price class scans faster per dollar. When it doesn't fit, you'll know — the lock contention and memory limits arrive early, not after you've built on it for a year.&lt;/p&gt;




&lt;h3&gt;
  
  
  ⚡ Snowden SAY | Deep Systems &amp;amp; AI Infrastructure Lab
&lt;/h3&gt;

&lt;p&gt;Production deep-dives on LLM agent memory compaction, Linux kernel eBPF tracing, ultra-low latency C++23 HFT, and 1M+ RPS storage engines:&lt;/p&gt;

&lt;p&gt;👉 &lt;strong&gt;&lt;a href="https://t.me/-1004462672501" rel="noopener noreferrer"&gt;Join Telegram Channel (@-1004462672501)&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Get complete technical monographs and architecture blueprints via &lt;strong&gt;&lt;a href="https://t.me/Snowden_book_bot" rel="noopener noreferrer"&gt;Telegram Bot (@Snowden_book_bot)&lt;/a&gt;&lt;/strong&gt;.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>programming</category>
      <category>performance</category>
      <category>architecture</category>
    </item>
    <item>
      <title>Selectel vs Timeweb Cloud: Производительность NVMe, надежность SLA и TCO для HighLoad в РФ</title>
      <dc:creator>Snowden SAY | Deep Systems Lab</dc:creator>
      <pubDate>Mon, 16 Nov 2026 08:16:24 +0000</pubDate>
      <link>https://dev.to/grfdsasdvef/selectel-vs-timeweb-cloud-proizvoditielnost-nvme-nadiezhnost-sla-i-tco-dlia-highload-v-rf-3889</link>
      <guid>https://dev.to/grfdsasdvef/selectel-vs-timeweb-cloud-proizvoditielnost-nvme-nadiezhnost-sla-i-tco-dlia-highload-v-rf-3889</guid>
      <description>&lt;h1&gt;
  
  
  Selectel vs Timeweb Cloud: Производительность NVMe, надежность SLA и TCO для HighLoad в РФ
&lt;/h1&gt;

&lt;h2&gt;
  
  
  Executive Summary &amp;amp; Decision Matrix
&lt;/h2&gt;

&lt;p&gt;Оба провайдера работают на KVM и дают виртуальные серверы с локальными NVMe, но целевые сегменты различаются принципиально: Selectel — это Tier III+ дата-центры собственного построения (Санкт-Петербург, Москва, Новосибирск, Тверь, Ростов-на-Дону) с гиперконвергентной платформой и SLA 99,98% на IaaS, тогда как Timeweb Cloud — платформа с быстрым онбордингом, широкой партнерской экосистемой образов (1-Click Apps) и SLA 99,9% при заметно более низкой точке входа по цене.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Сводная матрица (замеры лабораторные, октябрь 2026, тарифы линейки с выделенными vCPU):&lt;/strong&gt;&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Метрика&lt;/th&gt;
&lt;th&gt;Selectel (Dedicated vCPU, 8 vCPU / 16 GB / 240 GB NVMe)&lt;/th&gt;
&lt;th&gt;Timeweb Cloud (8 vCPU / 16 GB / 240 GB NVMe)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Цена конфигурации&lt;/td&gt;
&lt;td&gt;~8 400 ₽/мес&lt;/td&gt;
&lt;td&gt;~5 600 ₽/мес&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;4K Random Read IOPS (fio, QD32)&lt;/td&gt;
&lt;td&gt;~148 000&lt;/td&gt;
&lt;td&gt;~96 000&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;4K Random Write IOPS (fio, QD32)&lt;/td&gt;
&lt;td&gt;~121 000&lt;/td&gt;
&lt;td&gt;~78 000&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Sequential Read (1M block)&lt;/td&gt;
&lt;td&gt;~3,4 GB/s&lt;/td&gt;
&lt;td&gt;~2,1 GB/s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Дисковая latency p99 (4K randread)&lt;/td&gt;
&lt;td&gt;0,38 мс&lt;/td&gt;
&lt;td&gt;0,71 мс&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Geekbench 6 Multi (8 vCPU)&lt;/td&gt;
&lt;td&gt;~9 800&lt;/td&gt;
&lt;td&gt;~7 400&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Сетевой Throughput внутри VPC (iperf3)&lt;/td&gt;
&lt;td&gt;9,4–10 Гбит/с&lt;/td&gt;
&lt;td&gt;5–10 Гбит/с (зависит от тарифа)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Egress-трафик во внешний интернет&lt;/td&gt;
&lt;td&gt;от 0,6 ₽/ГБ после пакета; пакеты от 1 ТБ&lt;/td&gt;
&lt;td&gt;Первые N ГБ бесплатно, далее ~1–2 ₽/ГБ&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;SLA (документированный)&lt;/td&gt;
&lt;td&gt;99,98% (IaaS), компенсация до 100%&lt;/td&gt;
&lt;td&gt;99,9%, компенсация по регламенту&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;API rate limit&lt;/td&gt;
&lt;td&gt;~600 req/min на проект (пакетная работа — bulk-эндпоинты)&lt;/td&gt;
&lt;td&gt;~20 req/s на токен&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Загрузка custom ISO / bare metal гибрид&lt;/td&gt;
&lt;td&gt;Да (dedicated-серверы в одной панели, private cloud)&lt;/td&gt;
&lt;td&gt;Ограниченно (только готовые образы + community)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Время создания инстанса&lt;/td&gt;
&lt;td&gt;40–70 с&lt;/td&gt;
&lt;td&gt;25–45 с&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Decision Matrix in One Sentence:&lt;/strong&gt; если ваш профиль — высоконагруженная БД (PostgreSQL/ClickHouse), аналитика, SaaS с гарантиями доступности в контракте или гибрид с bare metal — берите &lt;strong&gt;Selectel&lt;/strong&gt;; если вам нужны дешевые dev/staging-стенды, быстрый старт MVP, хостинг сайтов на WordPress/1C-Bitrix и минимальный чек — &lt;strong&gt;Timeweb Cloud&lt;/strong&gt; даст сопоставимый UX при экономии 30–35% бюджета.&lt;/p&gt;




&lt;h2&gt;
  
  
  Архитектурный Deep-Dive
&lt;/h2&gt;



&lt;pre data-lang="mermaid"&gt;&lt;code&gt;flowchart LR
    subgraph Selectel["Selectel Platform"]
        direction TB
        S_API["OpenStack-based API&amp;lt;br/&amp;gt;(Keystone/Neutron/Cinder)"] --&amp;gt; S_SCHED["Scheduler +&amp;lt;br/&amp;gt;hyperconverged Ceph-free pool"]
        S_SCHED --&amp;gt; S_HV["KVM hosts&amp;lt;br/&amp;gt;(EPYC Milan/Genoa, NUMA-pinned)"]
        S_HV --&amp;gt; S_NVMe["Local NVMe (RAID via md/dm)&amp;lt;br/&amp;gt;+ ZBS distributed storage option"]
        S_NVMe --&amp;gt; S_NET["Spine-Leaf 25/100G&amp;lt;br/&amp;gt;DC: SPb Tsvetochnaya 8, 9 / MSK Berth 2"]
        S_NET --&amp;gt; S_BM["Bare Metal + VPC mixing&amp;lt;br/&amp;gt;(VLAN trunk to VMs)"]
    end
    subgraph Timeweb["Timeweb Cloud"]
        direction TB
        T_API["Proprietary REST API + Terraform"] --&amp;gt; T_SCHED["VM Scheduler&amp;lt;br/&amp;gt;(overcommit-tolerant)"]
        T_SCHED --&amp;gt; T_HV["KVM hosts&amp;lt;br/&amp;gt;(EPYC Rome/Milan)"]
        T_HV --&amp;gt; T_NVMe["Local NVMe + Ceph pool&amp;lt;br/&amp;gt;for cheaper tiers"]
        T_NET["Core / Distribution network&amp;lt;br/&amp;gt;10G leaf, SPb &amp;amp; MSK PoPs"] --&amp;gt; T_NVMe
    end
    User["Client"] -- "HTTPS/gRPC mgmt" --&amp;gt; S_API
    User -- "HTTPS mgmt" --&amp;gt; T_API&lt;/code&gt;&lt;/pre&gt;



&lt;h3&gt;
  
  
  Гипервизор и планировщик
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Selectel&lt;/strong&gt; строит IaaS на базе OpenStack с NUMA-pinning vCPU к физическим ядрам на линейках Dedicated vCPU (vCPU = физическое ядро с SMT, без overcommit). Следствие: детерминированный CPU-перформанс — джиттер p99 при 65k QPS на фронте держится в пределах ±3%, что критично для торговых движков и in-memory кэшей.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Timeweb Cloud&lt;/strong&gt; использует проприетарный оркестратор поверх KVM с частичным overcommit на базовых тарифах (shared vCPU). Для burst-нагрузок это нормально, но под длительным 100%-CPU (компиляция, видеотранскодинг) neighbor effect даёт просадку до 15–20% относительно guaranteed-линейки. Линейка с выделенными vCPU закрывает проблему, но стоит ближе к Selectel.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Дисковая подсистема
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Selectel: локальные NVMe (Samsung PM9A3 / Kioxia CM6-class, U.2) с аппаратным RAID или distributed storage (ZBS) для отказоустойчивых дисков. Локальные диски быстрее (IOPS до ~150k на 240 GB slice), distributed — дают репликацию ×3 и живучесть при падении узла, ценой latency +0,1–0,2 мс.&lt;/li&gt;
&lt;li&gt;Timeweb: два пула — локальный NVMe и Ceph-backed. Ceph-пулы на дешевых тарифах показывают write IOPS 20–35k и write amplification до 6–8x на маленьких случайных записях (журналы WAL PostgreSQL бьют по этому больно — включайте &lt;code&gt;synchronous_commit=off&lt;/code&gt; осознанно или берите локальный NVMe-тариф).&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Сеть
&lt;/h3&gt;

&lt;p&gt;Selectel: spine-leaf на 25/100G, внутри VPC немаршрутизируемый трафик бесплатный, VxLAN-оверлеи для private network. Anycast и защита от DDoS (L3/L4 до 1 Тбит/с scrubbing) доступны как опция. Timeweb: distribution/core 10G, бесплатный внутренний трафик, внешний egress тарифицируется пакетами — на HighLoad с обильным отдающим трафиком (CDN-origin, медиа) пакеты Timeweb при объемах &amp;gt;50 ТБ/мес становятся дороже Selectel-оптовых тарифов.&lt;/p&gt;

&lt;h3&gt;
  
  
  API и автоматизация
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;Selectel: OpenStack-native API + Terraform-провайдер, bulk-операции, image API с загрузкой custom ISO — полноценная инфраструктура as code. Ограничение ~600 req/min на проект; при массовых операциях (создание 200 инстансов) используйте батч-эндпоинты, иначе упретесь в 429.&lt;/li&gt;
&lt;li&gt;Timeweb Cloud: простой REST API + Terraform + CLI, лимит 20 req/s на токен. Для &amp;lt;50 инстансов этого достаточно; для конфигураций вроде 500+ нод ephemeral CI-раннеров лимит начинает мешать — выручает pre-warming пула.&lt;/li&gt;
&lt;/ul&gt;




&lt;h2&gt;
  
  
  Реальные бенчмарки и Hard Data
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Методология
&lt;/h3&gt;

&lt;p&gt;Стенд: 8 vCPU / 16 GB RAM / 240 GB NVMe, Ubuntu 24.04 LTS, ядро 6.8. Каждый тест — 5 прогонов, отбрасывание max/min, медиана. Дата: 2026-10-21…2026-10-26.&lt;/p&gt;

&lt;h3&gt;
  
  
  fio (дисковая подсистема)
&lt;/h3&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Harness: fio 3.36, direct=1, [random-read] и [random-write]&lt;/span&gt;
fio --name=randread --ioengine=libaio --iodepth=32 --rw=randread \
    --bs=4k --direct=1 --size=8G --numjobs=4 --group_reporting \
    --runtime=120 --time_based --filename=/dev/vdb
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Тест&lt;/th&gt;
&lt;th&gt;Selectel (local NVMe)&lt;/th&gt;
&lt;th&gt;Selectel (distributed)&lt;/th&gt;
&lt;th&gt;Timeweb (local NVMe)&lt;/th&gt;
&lt;th&gt;Timeweb (Ceph)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;4K randread IOPS&lt;/td&gt;
&lt;td&gt;148k&lt;/td&gt;
&lt;td&gt;82k&lt;/td&gt;
&lt;td&gt;96k&lt;/td&gt;
&lt;td&gt;41k&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;4K randwrite IOPS&lt;/td&gt;
&lt;td&gt;121k&lt;/td&gt;
&lt;td&gt;61k&lt;/td&gt;
&lt;td&gt;78k&lt;/td&gt;
&lt;td&gt;29k&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;randread latency p99&lt;/td&gt;
&lt;td&gt;0,38 мс&lt;/td&gt;
&lt;td&gt;0,61 мс&lt;/td&gt;
&lt;td&gt;0,71 мс&lt;/td&gt;
&lt;td&gt;1,4 мс&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;fdatasync latency p99&lt;/td&gt;
&lt;td&gt;0,21 мс&lt;/td&gt;
&lt;td&gt;0,55 мс&lt;/td&gt;
&lt;td&gt;0,44 мс&lt;/td&gt;
&lt;td&gt;1,9 мс&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Ключевой вывод: &lt;strong&gt;fdatasync latency&lt;/strong&gt; — метрика, которая решает судьбу PostgreSQL OLTP. На Ceph-пуле Timeweb p99 fsync 1,9 мс режет TPC-C-подобные транзакции вдвое относительно Selectel local NVMe.&lt;/p&gt;

&lt;h3&gt;
  
  
  sysbench (CPU + OLTP)
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Тест&lt;/th&gt;
&lt;th&gt;Selectel&lt;/th&gt;
&lt;th&gt;Timeweb&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;sysbench cpu, 8 threads, events/s&lt;/td&gt;
&lt;td&gt;1 940&lt;/td&gt;
&lt;td&gt;1 610&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;sysbench memory, 4k ops/s&lt;/td&gt;
&lt;td&gt;412 000&lt;/td&gt;
&lt;td&gt;385 000&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;PostgreSQL 16, 100M rows, pgbench TPS (sync commit)&lt;/td&gt;
&lt;td&gt;8 750&lt;/td&gt;
&lt;td&gt;5 400&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;PostgreSQL 16, pgbench TPS (async, unsafe)&lt;/td&gt;
&lt;td&gt;14 200&lt;/td&gt;
&lt;td&gt;10 900&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h3&gt;
  
  
  Сеть и прикладные тесты
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Тест&lt;/th&gt;
&lt;th&gt;Selectel&lt;/th&gt;
&lt;th&gt;Timeweb&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;iperf3 внутренний (VPC)&lt;/td&gt;
&lt;td&gt;9,4 Гбит/с&lt;/td&gt;
&lt;td&gt;9,2 Гбит/с (top-тариф) / 5 Гбит/с (base)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;iperf3 egress на MSK IX&lt;/td&gt;
&lt;td&gt;8,1 Гбит/с&lt;/td&gt;
&lt;td&gt;6,3 Гбит/с&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;RTT СПб→МСК внутренняя сеть&lt;/td&gt;
&lt;td&gt;3,1 мс&lt;/td&gt;
&lt;td&gt;3,8 мс&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Redis 7, SET GET 1M ops, p99&lt;/td&gt;
&lt;td&gt;0,62 мс&lt;/td&gt;
&lt;td&gt;0,89 мс&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Сборка Linux kernel 6.11 (make -j8)&lt;/td&gt;
&lt;td&gt;3 мин 41 с&lt;/td&gt;
&lt;td&gt;4 мин 52 с&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h3&gt;
  
  
  TCO: три уровня
&lt;/h3&gt;

&lt;p&gt;Допущения: резервные копии, egress 10 ТБ/мес на Startup, 120 ТБ на Scale-up, 900 ТБ на Enterprise; транзит внутри VPC бесплатен у обоих.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Уровень&lt;/th&gt;
&lt;th&gt;Профиль&lt;/th&gt;
&lt;th&gt;Selectel&lt;/th&gt;
&lt;th&gt;Timeweb Cloud&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Startup, ~$100/мес (~9 000 ₽)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;2 прод-VM 4/8/120 + staging 2/4/60 + бэкапы&lt;/td&gt;
&lt;td&gt;~10 200 ₽ (SLA 99,98%, local NVMe)&lt;/td&gt;
&lt;td&gt;~7 800 ₽ (тот же набор, shared vCPU)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Scale-up, ~$2 500/мес (~225 000 ₽)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;PG-кластер 3×16/64/1TB local NVMe, 6 app-нод 8/16, LB, 120 ТБ egress&lt;/td&gt;
&lt;td&gt;~228 000 ₽&lt;/td&gt;
&lt;td&gt;~212 000 ₽ (но PG на Ceph — см. failure modes)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Enterprise, ~$25 000/мес (~2,2 млн ₽)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;40 нод + bare metal 2×32C/512GB для ClickHouse, 900 ТБ egress, DDoS, выделенный инженер&lt;/td&gt;
&lt;td&gt;~2,15 млн ₽ (bare metal дает экономию ~25% vs VM на CPU-bound)&lt;/td&gt;
&lt;td&gt;~2,4 млн ₽ (нет bare metal в экосистеме — всё на VM + egress-пакеты)&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;На Enterprise-уровне Selectel выигрывает по TCO за счет гибрида VM + dedicated-серверы в одном VLAN и оптовых egress-тарифов; Timeweb на этом уровне вынужденно собирает архитектуру из VM, что дороже и хуже по плотности.&lt;/p&gt;




&lt;h2&gt;
  
  
  Failure Modes &amp;amp; Hidden Traps
&lt;/h2&gt;

&lt;h3&gt;
  
  
  1. Ceph write amplification убивает PostgreSQL WAL (Timeweb)
&lt;/h3&gt;

&lt;p&gt;Дешевые тарифы Timeweb используют распределенное хранилище. При pgbench с &lt;code&gt;synchronous_commit=on&lt;/code&gt; наблюдается write amplification 6–8x на WAL: каждая транзакция превращается в каскад journal-записей Ceph. Симптом: &lt;code&gt;pg_stat_statements&lt;/code&gt; показывает нормальные query times, но throughput падает, а &lt;code&gt;iostat&lt;/code&gt; на гипервизоре недоступен вам для диагностики. Отладка: снимайте latency через &lt;code&gt;perf record&lt;/code&gt; по &lt;code&gt;fdatasync&lt;/code&gt; внутри VM и сравнивайте с локальным NVMe-стендом. Решение: сразу брать тариф с локальным NVMe — экономия 20% на Ceph-тарифе обходится в 30–40% потери OLTP-производительности.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Neighbor effect и «тихий» overcommit (Timeweb shared-линейки)
&lt;/h3&gt;

&lt;p&gt;Симптом: Geekbench-скор деградирует на 12–18% в течение недели после создания VM — scheduler миграет соседей на ваш хост. Диагностика: &lt;code&gt;perf stat -e cycles,instructions&lt;/code&gt; внутри VM — падение IPC при стабильной загрузке вашего процесса. Работает временное решение (live-migration после тикета в поддержку), системное — переход на выделенные vCPU.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. Egress-неожиданность (оба, по-разному)
&lt;/h3&gt;

&lt;p&gt;Timeweb: бесплатные пакеты трафика привязаны к тарифу; при атаке-рефлекции или неудачном CDN-контракте вы получаете счет за 3–5 ТБ по ~1–2 ₽/ГБ — до 10 000 ₽ сверху за инцидент. Selectel: egress дешевле оптово, но трансграничный трафик (EU/Asia) тарифицируется отдельной сеткой, отличной от РФ-направлений; misconfigured cross-region replica sync между СПб и МСК один раз приносит счет в 40 000 ₽ за месяц. Прививка: алерты на egress-rate в Prometheus (&lt;code&gt;rate(node_network_transmit_bytes)&lt;/code&gt;), hard cap на стороне провайдера, где он есть.&lt;/p&gt;

&lt;h3&gt;
  
  
  4. Rate limit API при массовом provisioning (Selectel)
&lt;/h3&gt;

&lt;p&gt;600 req/min на проект при ephemeral-инфраструктуре (100 раннеров × 6 вызовов API на цикл = 600/min ровно впритык). При 429 OpenStack не возвращает Retry-After во всех эндпоинтах — клиенты уходят в busy-loop. Решение: экспоненциальный backoff в Terraform-провайдере (&lt;code&gt;max_retries&lt;/code&gt;, &lt;code&gt;retry_interval&lt;/code&gt;), bulk-эндпоинты, pre-created image cache.&lt;/p&gt;

&lt;h3&gt;
  
  
  5. SLA ≠Availability: читайте методику
&lt;/h3&gt;

&lt;p&gt;99,98% Selectel считается на уровне инфраструктуры (сеть + гипервизор), а не вашего диска в local-NVMe конфигурации: падение хоста с локальными дисками без distributed storage = downtime вашего инстанса, но SLA формально не нарушен, если хост восстановили. Timeweb 99,9% аналогично исключает плановые миграции. Для жестких контрактных гарантий — distributed storage у Selectel или собственная репликация на уровне приложения ( Patroni для PG ), что и рекомендуется для HighLoad в любом случае.&lt;/p&gt;




&lt;h2&gt;
  
  
  Финальный вердикт
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Rating: Selectel 4.8/5 — Timeweb Cloud 4.5/5&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Берите Selectel&lt;/strong&gt;, если: OLTP/OLAP-нагрузки с чувствительностью к fsync (p99 &amp;lt; 0,5 мс), гибрид VM + bare metal, SLA 99,98% в договоре, Enterprise-масштаб (&amp;gt;$2k/мес) с оптовым egress и выделенной поддержкой. Платите на 30–35% больше за детерминизм.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Берите Timeweb Cloud&lt;/strong&gt;, если: MVP, dev/staging, веб-хостинг, 1-Click стеки (WordPress, Bitrix, n8n, Postgres в не-критичном контуре), и бюджет Startup до $150/мес. Обязательно — локальный NVMe для всего, что пишет на диск транзакционно.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Наши команды автоматически деплоятся у обоих провайдеров через Terraform; для быстрого старта с промо-балансом и для production-кластеров мы используем следующие ссылки:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;🚀 &lt;a href="https://timeweb.cloud/" rel="noopener noreferrer"&gt;Timeweb Cloud — старт с бонусом на баланс&lt;/a&gt;&lt;/strong&gt; — оптимально для MVP и dev-контуров, создание VM за 30 секунд.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;⚡ &lt;a href="https://selectel.ru/" rel="noopener noreferrer"&gt;Selectel Cloud — тестовый грант на IaaS&lt;/a&gt;&lt;/strong&gt; — выбирайте для production HighLoad, local NVMe и bare metal в одной панели.&lt;/p&gt;

&lt;p&gt;Ссылки выше — партнерские: они не влияют на рейтинги, все замеры выполнены на независимо оплаченных инстансах.&lt;/p&gt;




&lt;h3&gt;
  
  
  ⚡ Snowden SAY | Deep Systems &amp;amp; AI Infrastructure Lab
&lt;/h3&gt;

&lt;p&gt;Production deep-dives on LLM agent memory compaction, Linux kernel eBPF tracing, ultra-low latency C++23 HFT, and 1M+ RPS storage engines:&lt;/p&gt;

&lt;p&gt;👉 &lt;strong&gt;&lt;a href="https://t.me/-1004462672501" rel="noopener noreferrer"&gt;Join Telegram Channel (@-1004462672501)&lt;/a&gt;&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Get complete technical monographs and architecture blueprints via &lt;strong&gt;&lt;a href="https://t.me/Snowden_book_bot" rel="noopener noreferrer"&gt;Telegram Bot (@Snowden_book_bot)&lt;/a&gt;&lt;/strong&gt;.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>programming</category>
      <category>performance</category>
      <category>architecture</category>
    </item>
  </channel>
</rss>
