DEV Community: Syed Anzar

Plane: The Open-Source Jira/Linear Alternative — What Actually Happens When You Self-Host a 61k-Star Project Management Platform

Syed Anzar — Tue, 03 Nov 2026 13:40:34 +0000

Plane: The Open-Source Jira/Linear Alternative — What Actually Happens When You Self-Host a 61k-Star Project Management Platform

You use Jira. Or Linear. Or ClickUp. Every day. Click "Create Issue," drag cards across a Kanban board, run a sprint retrospective. It works. Mostly.

But have you ever wondered what it takes to run one of these things yourself? Not the SaaS version — the actual software, on your own infrastructure, your own data, zero vendor lock-in.

Plane is a 61.9k-star open-source project management platform that replaces Jira, Linear, Monday, and ClickUp. Not a toy. A production-grade system with real-time collaboration, AI-powered workflows, a native MCP server for AI agents, and a microservice architecture running on PostgreSQL, Redis, RabbitMQ, and MinIO.

Here's what happens under the hood when you self-host it.

The Architecture: 10 Services, One Platform

Plane isn't a monolith. It's a distributed system disguised as a project management tool. Run docker-compose up and you're spinning up ten distinct services:

Service	Role	Tech
web	Main React frontend — work items, cycles, pages, analytics	React 19, React Router 7, MobX, Tailwind, Vite
api	Core Django REST API — all data operations	Django 5.2, DRF, Celery, Channels
worker	Async background jobs (imports, exports, notifications)	Celery + RabbitMQ
beat-worker	Scheduled tasks (cleanup, reports, reminders)	Celery Beat
migrator	Schema migrations on deploy (runs once, exits)	Django migrations
live	Real-time WebSocket server (cursors, presence, live updates)	Node.js, WebSockets
proxy	SSL termination, routing, cert management (Caddy)	Caddy
space	Public sharing — publish projects/views without login	React
admin	Instance administration (billing, licensing, SSO)	React
silo	Third-party integrations (GitHub, GitLab, Slack OAuth/webhooks)	Python

Plus five infrastructure dependencies:

PostgreSQL 15.7+ — primary data store (max_connections=1000)
Valkey 7.2 (Redis-compatible) — cache, sessions, real-time collab state
RabbitMQ 3.13 — message queue for all async work
MinIO/S3 (via pgsty/silo) — file uploads, attachments, exports
OpenSearch (optional) — full-text search indexing

This isn't "Django + React." This is a platform.

What Happens When You Create a Work Item

Click "New Issue" in the React frontend. Here's the path:

Frontend (MobX store) → optimistic UI update, instant feedback
REST API call → POST /api/v1/workspaces/{slug}/projects/{project_id}/work-items/
Django viewset → validation, permissions, business logic
PostgreSQL → INSERT into work_items with all relations (project, assignee, state, cycle, module)
Redis cache invalidation → affected queries purged
RabbitMQ message → work_item.created event published
Worker picks up → sends notifications, updates search index (OpenSearch), triggers webhooks (Silo)
Live service → WebSocket broadcast to all connected clients in that workspace → real-time UI update for collaborators

Zero polling. Zero manual refresh. The WebSocket connection stays open. When your teammate moves a card, you see it move instantly.

Real-Time Collaboration: The `live` Service

Most "real-time" tools fake it with long-polling or frequent AJAX. Plane runs a dedicated Node.js WebSocket server (apps/live).

┌────────────────────────────────────────────────────────────────────────────────────────────│
│  Browser A  │◄──────────────────────────────────────────────────────────────────────────────────────││   live svc  │
└──────────────────────────────────────────────────────────────────────────────────────────────│└────│
───────────────────────────────────────────────────────────────────────────────────────────────────────────────│ Redis Pub/Sub
┌─────────────────────────────────────────────────────────────────────────────────────────────────────│
│  Browser B  │◄──────────────────────────────────────────────────────────────────────────────────││   live svc  │
└──────────────────────────────────────────────────────────────────────────────────────│└────│

Presence: Who's online, where their cursor is, what they're viewing
Optimistic sync: Local changes broadcast immediately; server resolves conflicts
Scalability: Multiple live replicas behind Redis Pub/Sub — horizontal scaling works

The live service doesn't touch PostgreSQL. Pure real-time layer. Django handles persistence; live handles now.

Background Jobs: Celery + RabbitMQ Done Right

Import 10,000 issues from Jira. You don't wait. The API returns 202 Accepted immediately.

What happens next:

API enqueues jira_import_job to RabbitMQ
Worker (Celery) picks it up — runs in isolation, no request timeout
Chunks the import, processes batches, updates progress in Redis
Each created work item emits work_item.created → RabbitMQ → other workers → notifications, search indexing, webhooks
Beat worker runs nightly cleanup: expired sessions, stale locks, old audit logs

This is durable execution. Worker crashes mid-import? RabbitMQ requeues. The job completes. No lost work.

Self-Hosting: Data Sovereignty Without the DevOps Nightmare

Plane's Docker Compose file is 177 lines. One command brings up the entire stack:

# docker-compose.yml (simplified)
services:
  api:
    build: ./apps/api
    depends_on: [plane-db, plane-redis]
    env_file: ./apps/api/.env
  worker:
    build: ./apps/api
    command: ./bin/docker-entrypoint-worker.sh
    depends_on: [api, plane-db, plane-redis, plane-mq]
  beat-worker:
    command: ./bin/docker-entrypoint-beat.sh
  migrator:
    restart: "no"           # runs once, exits
    command: ./bin/docker-entrypoint-migrator.sh
  live:
    build: ./apps/live
  proxy:
    build: ./apps/proxy
    ports: ["80:80", "443:443"]
  plane-db:
    image: postgres:15.7-alpine
    command: postgres -c 'max_connections=1000'
  plane-redis:
    image: valkey/valkey:7.2.11-alpine
  plane-mq:
    image: rabbitmq:3.13.6-management-alpine
  plane-minio:
    image: docker.io/pgsty/silo:RELEASE.2026-09-16T00-00-00Z

Details that matter:

Caddy handles automatic Let's Encrypt certificates — set CERT_EMAIL and it works
Migrator runs restart: no — applies schema changes on deploy, then exits cleanly
Valkey (not Redis) — drop-in replacement, better memory efficiency, same protocol
pgsty/silo — MinIO fork optimized for single-node deployments, lighter resource usage
PostgreSQL max_connections=1000 — tuned for connection pooling under load

Air-gapped deployment supported. No internet required after initial pull. Helm charts for Kubernetes. Prime CLI for multi-instance management.

The Tech Stack: Battle-Tested, Not Trendy

Layer	Choice	Why It Matters
Backend	Django 5.2 LTS	Mature ORM, admin, auth, migrations, security built-in
Async	Celery 5.5 + RabbitMQ	Durable queues, retries, dead-letter, monitoring
Realtime	Node.js + WebSockets	Separate service = no blocking Django workers
Frontend	React 19 + React Router 7	Modern, fast, MobX for fine-grained reactivity
Database	PostgreSQL 15/16	ACID, JSONB, full-text search, proven at scale
Cache	Valkey 7.2	Redis API, better memory, active development
Storage	S3/MinIO	Object storage standard, any provider
Monorepo	Turborepo + pnpm	Shared types, incremental builds, cached tasks

No custom consensus algorithms. No exotic databases. Just solid engineering with the right tool for each job.

AI & Extensibility: Not an Afterthought

Plane has a native MCP (Model Context Protocol) server built in. AI agents can:

Create/update work items
Query projects, cycles, analytics
Run workflows
Access docs and pages

// MCP tool example
{
  "name": "create_work_item",
  "description": "Create a new work item in Plane",
  "parameters": {
    "type": "object",
    "properties": {
      "project_id": {"type": "string"},
      "name": {"type": "string"},
      "type": {"type": "string", "enum": ["issue", "task", "epic"]},
      "state_id": {"type": "string"}
    }
  }
}

Plus Slack @Plane mentions — type @Plane create issue "Fix login bug" in Project X and it happens. GitHub/GitLab two-way sync. Webhook framework for custom integrations.

Migration: You're Not Trapped

Jira → Plane importer: issues, sprints, epics, attachments, comments, users.
Linear → Plane importer: preserves cycles, labels, relations.
GitHub → Plane: bidirectional issue sync.

One-way door? No. Export your data anytime. PostgreSQL dump. S3 bucket. You own it.

What You're Signing Up For

Reality	Detail
Resource baseline	~4 GB RAM minimum (Postgres + Redis + RabbitMQ + 4 API workers + live + frontend)
Maintenance	`docker-compose pull && docker-compose up -d` for updates. Migrator handles schema.
Backups	`pg_dump` + S3 sync + Redis RDB + MinIO bucket versioning
Scaling	Add API workers, live replicas, RabbitMQ consumers. Postgres read replicas for heavy read workloads.
License	AGPL-3.0 — if you modify and expose publicly, share changes. Self-hosted internal use: no obligation.

Why This Matters for Developers

You spend years learning React, Django, PostgreSQL, Redis, RabbitMQ, WebSockets, Docker, CI/CD. Plane is the reference architecture for a modern full-stack application.

How to structure a Turborepo monorepo with shared TypeScript types across frontend/backend
How to separate real-time concerns from transactional API
How to design durable background job processing
How to run PostgreSQL at max_connections=1000 with connection pooling
How to do zero-downtime migrations with a dedicated migrator service
How to build a plugin/integration system (Silo) that doesn't couple to core

Studying Plane's codebase teaches you more about systems engineering than most tutorials.

TL;DR

Plane replaces Jira/Linear/ClickUp with a self-hosted, open-source platform that:

Runs on standard infrastructure (Postgres, Redis, RabbitMQ, S3)
Uses a microservice architecture (10 services, each with a clear responsibility)
Delivers true real-time collaboration via a dedicated WebSocket service
Handles durable async execution via Celery + RabbitMQ
Includes AI/agent readiness via native MCP server
Gives you full data ownership — AGPL-3.0, air-gapped ready, one-command deploy

Next time you drag a card in Jira, remember: there's a whole distributed system making that feel instant. Plane lets you own that system.

Repository: makeplane/plane (61.9k ⭐, AGPL-3.0)
Docs: developers.plane.so/self-hosting/overview
Try it: docker-compose up — 2 minutes to a running instance

How HTTP/3 and QUIC Actually Work: UDP, Streams, and Why 0-RTT Changes Everything

Syed Anzar — Mon, 02 Nov 2026 12:39:52 +0000

How HTTP/3 and QUIC Actually Work: UDP, Streams, and Why 0-RTT Changes Everything

You type https://example.com and hit Enter. Your browser opens a connection, negotiates TLS, sends an HTTP request, gets a response. That's the mental model most developers carry.

It's incomplete. Since 2022, there's a real chance that connection isn't TCP at all — it's QUIC. And QUIC doesn't just swap TCP for UDP. It restructures how transport, security, and application semantics interact at the packet level.

This article explains what actually happens when your browser speaks HTTP/3, from the wire format up to the stream multiplexing that makes head-of-line blocking a thing of the past.

The Short Version: What Changed

Layer	HTTP/2 (TCP + TLS)	HTTP/3 (QUIC)
Transport	TCP (reliable, ordered byte stream)	QUIC over UDP (reliable, ordered per stream)
Security	TLS 1.3 layered over TCP	TLS 1.3 integrated into QUIC
Multiplexing	HTTP/2 frames on one TCP stream	QUIC native streams (independent)
Handshake	TCP SYN → TLS ClientHello → ... (2–3 RTT)	QUIC Initial + TLS in 1 RTT (or 0-RTT resume)
Connection ID	4-tuple (src IP, src port, dst IP, dst port)	Opaque Connection ID (survives IP/port change)
Head-of-line blocking	Yes — one lost packet stalls all streams	No — loss only blocks the affected stream

The headline feature: QUIC moves reliability and ordering from the connection level to the stream level. A lost packet on stream 4 doesn't stall stream 0, 8, or 12.

The Wire Format: Packets, Frames, and Encryption Levels

QUIC doesn't send "data." It sends packets, each carrying frames. The frames belong to streams. The packets live inside UDP datagrams.

Four Encryption Levels = Four Packet Types

Every QUIC packet has an encryption level. This determines which frames it can carry:

Packet Type	Header	Encryption Level	Frames Allowed
Initial	Long	Initial keys (derived from client's random DCID)	`CRYPTO`, `ACK`, `PADDING`, `PING`, `CONNECTION_CLOSE`
0-RTT	Long	0-RTT keys (resumed session)	Everything Initial allows + `STREAM`, `MAX_DATA`, `NEW_CONNECTION_ID`, `DATAGRAM`...
Handshake	Long	Handshake keys	Everything 0-RTT allows + `NEW_TOKEN`, `PATH_RESPONSE`, `HANDSHAKE_DONE`
1-RTT	Short	1-RTT keys (established)	All frames — this is steady-state traffic

The long header (used during handshake) is self-describing: version, both Connection IDs with lengths, explicit packet type. The short header (1-RTT) strips all that — just a header byte, Destination CID, packet number, and two flag bits (spin bit, key phase). Header protection encrypts the packet number and reserved bits so on-wire they look like noise.

Frames: The Actual Payload

Frames are typed. The workhorse is STREAM — carries application data with a stream ID, offset, length, and FIN bit. Others:

ACK — acknowledges received packet numbers (with ECN support)
CRYPTO — carries TLS handshake messages (yes, TLS lives inside QUIC frames)
MAX_DATA / MAX_STREAM_DATA / MAX_STREAMS — flow control
NEW_CONNECTION_ID / RETIRE_CONNECTION_ID — connection migration
PATH_CHALLENGE / PATH_RESPONSE — path validation for migration
DATAGRAM (RFC 9221) — unreliable, unordered data for real-time media
CONNECTION_CLOSE — terminates connection (with error code)

Multiplexing happens at the frame level. One packet can carry a STREAM frame for stream 0, another STREAM for stream 4, and an ACK frame — all at once. There's no "this packet belongs to stream X." The frames inside decide.

Coalescing ≠ Multiplexing

Multiple QUIC packets (even from different packet number spaces) can be concatenated into one UDP datagram. This is coalescing — distinct from multiplexing. A single datagram can do both: coalesce an Initial + Handshake packet, each carrying frames from different streams.

Connection IDs: The Connection That Outlives Its Address

TCP identifies a connection by the 4-tuple. Change IP or port (Wi-Fi to cellular, NAT rebinding) and the connection dies.

QUIC uses Connection IDs (CIDs). Each endpoint chooses the CIDs it wants to receive and advertises them in the Source CID field of packets it sends. The peer copies that into the Destination CID of return packets.

Key insight: the Destination CID names the recipient, not the sender. On a client-to-server packet, the DCID is a value the server chose. The SCID is a value the client chose.

The client's very first Initial packet has no server CID yet, so it generates a random DCID (8+ bytes). This random value also seeds the Initial encryption keys. Once the server replies with its own SCID, the client adopts that as its new DCID and never uses the random one again.

Endpoints issue multiple CIDs over a connection's life (NEW_CONNECTION_ID frames). A migrating client switches to an unused CID, preventing passive observers from linking old and new paths by a constant CID.

Load balancers exploit this: they encode routing info in the CID they hand out, so later packets are steered to the right server without deep packet inspection.

The Handshake: 1-RTT by Default, 0-RTT on Resume

TCP + TLS 1.3: 2 RTT Minimum

Client                    Server
  SYN ──────────────────→
  ←────────────────── SYN+ACK
  ClientHello ─────────→
  ←──────── ServerHello, Certificate, Finished
  Finished ───────────→
  Application Data ────→

Two round trips before application data flows (TCP handshake + TLS handshake). TCP Fast Open and TLS early data exist but are hampered by middleboxes.

QUIC: 1 RTT (New Session)

Client                    Server
  Initial (CH) ────────→
  ←──────── Initial (SH, Cert, Finished)
  Handshake (Finished) →
  1-RTT (App Data) ────→

The QUIC Initial packet is the ClientHello. The server's Initial response carries ServerHello, Certificate, and Finished. The client's Handshake packet carries its Finished. Application data flows on 1-RTT packets immediately after.

0-RTT: Send Data Before the Server Replies

If the client has a session ticket from a previous connection (containing server transport parameters and TLS resumption PSK), it can send 0-RTT packets with application data alongside its Initial packet.

Client                    Server
  Initial + 0-RTT (App) →
  ←──────── Initial (SH, Cert, Finished)
  Handshake (Finished) →
  1-RTT (App Data) ────→

The catch: 0-RTT provides no replay protection. A captured 0-RTT packet can be resent by an attacker, and the server will process it again. HTTP/3 mitigates this via RFC 9221 — only idempotent requests (GET, HEAD) should use 0-RTT, and servers must implement anti-replay (single-use tokens, timestamp windows).

Streams: Independent, Ordered, Flow-Controlled

Each QUIC stream is an independent, reliable, ordered byte stream. Streams have:

Stream ID — 62-bit integer. Client-initiated bidirectional: 0, 4, 8... Server-initiated bidirectional: 1, 5, 9... Client-initiated unidirectional: 2, 6, 10... Server-initiated unidirectional: 3, 7, 11...
Offset — byte position within the stream (like TCP sequence number but per-stream)
FIN bit — marks end of stream

Flow control is per-stream AND connection-wide. MAX_STREAM_DATA limits bytes on a single stream; MAX_DATA limits total bytes across all streams. MAX_STREAMS limits concurrent stream count. This prevents a runaway stream from starving others.

No Head-of-Line Blocking

TCP delivers bytes in strict sequence order. If packet N is lost, packet N+1 sits in the receive buffer until N is retransmitted — even if N+1 belongs to a completely different logical request.

QUIC delivers per-stream. If stream 4 loses a packet, stream 0, 8, and 12 keep delivering data to the application. The transport layer handles retransmission for stream 4 independently.

# Conceptual: HTTP/3 request streams are independent
# Stream 0: GET /api/user → HEADERS + DATA frames
# Stream 4: GET /api/posts → HEADERS + DATA frames
# Stream 8: GET /api/comments → HEADERS + DATA frames

# If stream 4's packet is lost:
# - Stream 0 and 8 continue processing
# - Only stream 4's application callback waits

HTTP/3 Mapping: Requests = Bidirectional Streams

HTTP/3 maps each request-response pair to a single client-initiated bidirectional QUIC stream (stream 0, 4, 8...). The request goes on the stream; the response comes back on the same stream.

Two unidirectional control streams carry connection-level frames:

Client to Server: SETTINGS, GOAWAY, CANCEL_PUSH
Server to Client: SETTINGS, GOAWAY, PUSH_PROMISE, MAX_PUSH_ID

QPACK replaces HPACK. HPACK assumes in-order header delivery (true for HTTP/2 on TCP). QUIC doesn't guarantee cross-stream order. QPACK uses dedicated unidirectional streams for encoder/decoder state updates, while encoded header blocks reference static state without modifying it.

Stream Types in HTTP/3

Stream Type	Direction	Purpose
Request stream	Bidirectional (client-initiated)	HTTP request + response
Control stream (client)	Unidirectional	SETTINGS, GOAWAY, CANCEL_PUSH
Control stream (server)	Unidirectional	SETTINGS, GOAWAY, PUSH_PROMISE, MAX_PUSH_ID
Push stream	Unidirectional (server-initiated)	Server push responses

Connection Migration: Surviving Network Changes

When your phone switches from Wi-Fi to cellular, the IP and port change. TCP dies. QUIC survives.

Mechanism: the client sends packets from the new path with a new Destination CID (one the server issued via NEW_CONNECTION_ID). The server sees the CID, recognizes the connection, and validates the new path with PATH_CHALLENGE / PATH_RESPONSE frames (1 RTT). Once validated, traffic shifts to the new path.

Only the client can actively migrate in QUIC v1. Servers can indicate a preferred address during handshake (e.g., move from a shared anycast IP to a dedicated server IP).

Congestion control resets after migration — you're on a new path with unknown characteristics. This is why migration has a performance cost.

0-RTT in Practice: The Replay Problem

0-RTT is seductive — zero-round-trip resume. But it's a loaded gun.

Attacker captures client's 0-RTT packet (POST /api/transfer {to: attacker, amount: 1000})
Attacker replays it to server
Server processes it again → money transferred twice

HTTP/3 requires servers to:

Only accept idempotent methods in 0-RTT (GET, HEAD)
Implement anti-replay: single-use tokens, timestamp validation, or bloom filters
Reject 0-RTT if anti-replay state is unavailable

Developers using HTTP/3 libraries: don't assume 0-RTT is safe for mutations. Configure your client to only send safe methods in 0-RTT, or disable 0-RTT entirely for non-idempotent APIs.

Congestion Control: The Hidden Performance Variable

QUIC specifies reliability and flow control but not congestion control. Implementations choose: CUBIC, BBR, NewReno, etc.

This matters enormously. Cloudflare's QUIC uses CUBIC; their TCP uses BBR. In production benchmarks with 1% packet loss, Cloudflare's HTTP/3 lagged behind HTTP/2 because CUBIC collapses its congestion window aggressively on early loss, while BBR (rate-based) doesn't.

Google's QUIC uses BBR — their HTTP/3 matches or beats HTTP/2 under loss.

The lesson: QUIC performance doesn't equal protocol performance. It's implementation performance. When benchmarking, you're testing that specific QUIC stack's congestion control, not QUIC itself.

Real-World Performance: What the Data Shows

Scenario	HTTP/3 vs HTTP/2
Small payloads (<100 KB), no loss	HTTP/3 ~12% faster (1-RTT handshake advantage)
Large payloads (>1 MB), no loss	Roughly equivalent
Added loss (1%)	Mixed: Google HTTP/3 ≈ HTTP/2; Cloudflare HTTP/3 worse (CUBIC vs BBR)
Added latency (100 ms)	HTTP/3 better on Google; worse on Facebook (implementation quirks)
Page Load Time (multi-object)	HOL blocking removal has little measurable impact on Speed Index
Connection migration	Works but resets congestion control — brief throughput dip

Key findings from production studies (Cloudflare, Google, Facebook, APNIC, IMC 2021):

The 1-RTT handshake advantage dominates for short-lived connections (<1 MB)
For long-lived transfers, congestion control implementation matters more than protocol design
HOL blocking removal rarely moves the needle on real page loads — most pages don't have enough concurrent streams hitting loss simultaneously
Client implementation matters: Chrome's SSL verification adds ~20ms overhead vs command-line clients

What This Means for You

If You're a Backend Developer

Enable HTTP/3 (ALPN h3) — it's standardized, widely supported, and free latency reduction for new connections
Configure your QUIC stack's congestion control — BBRv2 or BBRv3 outperforms CUBIC under loss
Set MAX_STREAMS and flow control windows appropriately — defaults may limit parallelism
Implement 0-RTT anti-replay if you enable it — or disable 0-RTT for non-GET endpoints

If You're a Frontend Developer

Don't assume HTTP/3 is magic. The 1-RTT handshake helps first paint; HOL blocking removal helps only under loss with many concurrent resources
Resource prioritization still matters. HTTP/3 has no priority tree (HTTP/2 priorities were dropped). Use fetchpriority and preload instead
Connection pooling changes. QUIC connections are more expensive to create (crypto) but cheaper to maintain (no TCP state). Reuse aggressively

If You're Running Infrastructure

UDP traffic must pass. Corporate firewalls, load balancers, and DDoS mitigation often drop or rate-limit UDP. Test end-to-end
CID routing. If you terminate QUIC at a load balancer, encode backend routing in the CID — otherwise migration breaks
Monitor QUIC-specific metrics: handshake failure rate, 0-RTT acceptance rate, migration frequency, CID rotation rate

The Mental Model to Keep

┌─────────────────────────────────────────────────────────────┐
│                        UDP DATAGRAM                          │
│  ┌──────────────────┐  ┌──────────────────┐                 │
│  │   QUIC Packet    │  │   QUIC Packet    │  (coalesced)    │
│  │  (Initial, long) │  │ (Handshake, long)│                 │
│  └────────┬─────────┘  └────────┬─────────┘                 │
│           │                     │                            │
│  ┌────────▼─────────────────────▼────────────────────────┐  │
│  │                    FRAMES                               │  │
│  │  CRYPTO(frame)  ACK(frame)  PADDING(frame)              │  │
│  │  STREAM(id=0, off=0, data="GET /")  STREAM(id=4, ...)   │  │
│  └────────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────┘

QUIC is not "TCP over UDP." It's a new transport that integrates TLS, multiplexes at the frame level, identifies connections by CID instead of 4-tuple, and pushes reliability to the stream layer.

The next time you see h3 in your browser's network tab, you'll know exactly what's happening on the wire — and why it's not just TCP with a different header.

Trigger.dev: The Open-Source Durable Execution Platform That Replaces Cron, BullMQ, and Inngest for Long-Running TypeScript Jobs

Syed Anzar — Sun, 01 Nov 2026 04:59:47 +0000

Trigger.dev: The Open-Source Durable Execution Platform That Replaces Cron, BullMQ, and Inngest for Long-Running TypeScript Jobs

You've written the same background job pattern a dozen times. A user uploads a video. Your API returns 202 Accepted. Somewhere, a worker picks it up, transcodes it, uploads the result, sends an email. If the worker crashes halfway through, the user never gets notified. If the video takes 45 minutes, your serverless function times out at 10. So you split it into steps, add a queue, provision Redis, write retry logic, build a dashboard to see what's stuck. Congratulations - you've reinvented a job platform, badly.

This is where most TypeScript teams live. They know BullMQ. They've heard of Temporal. They've tried Inngest. But they keep hitting the same walls: timeouts, no visibility, retry storms that charge customers twice, infrastructure they didn't sign up to operate.

Trigger.dev solves this differently. It's not a queue library. It's not a workflow engine that forces you to rewrite code into deterministic activities. It's a durable execution platform that runs your plain async TypeScript functions on managed compute, checkpoints them mid-execution using Linux CRIU, and resumes them exactly where they left off - hours, days, or weeks later - with zero infrastructure for you to manage.

What Developers Get Wrong About Background Jobs

Myth: "I just need a queue."

A queue gives you push and pop. It doesn't give you:

A job that survives a deploy, a crash, or a Kubernetes node rotation
Automatic retries that don't redo work already done (idempotency keys + warm starts)
A 14-day wait.until() that costs zero compute while paused
Realtime progress streaming to your React frontend
Observability that shows the full trace of a 3-hour video pipeline

Myth: "Serverless functions are fine if I chain them."

Chaining serverless functions means:

Every step fits in the platform timeout (10s–15m)
You manually pass state between steps (serialization hell)
A failure at step 7 means re-running steps 1–6 unless you built your own checkpointing
Cold starts on every step
No single trace across the whole workflow

Myth: "Temporal is the only real durable execution."

Temporal is powerful. It also demands:

Deterministic workflow code (no Date.now(), no Math.random(), no direct fetch)
Splitting every side effect into Activities
Running your own workers (even on Temporal Cloud)
A week of ramp-up for new engineers
Versioning workflows with patching APIs when you change logic

Trigger.dev takes a different bet: your code should run as written. No DSL. No determinism rules. No replay. Just TypeScript.

How Trigger.dev Actually Works: Checkpoint-Resume, Not Event Sourcing

The core insight: a Linux process can be snapshotted and restored. CRIU (Checkpoint/Restore In Userspace) does exactly this - it freezes a process tree, serializes memory pages, CPU registers, open file descriptors, and kernel state to disk, then restores it later on the same or a different machine.

Trigger.dev wraps this in a TypeScript runtime. Here's the lifecycle:

1. Task Runs in an Isolated Container

// tasks/process-video.ts
import { task } from "@trigger.dev/sdk/v3";

export const processVideo = task({
 id: "process-video",
 run: async (payload: { videoUrl: string; userId: string }) => {
 // 1. Download (could take 10 minutes)
 const localPath = await downloadVideo(payload.videoUrl);

 // 2. Transcode with FFmpeg (could take 2 hours)
 const outputPath = await transcode(localPath);

 // 3. Upload result
 const cdnUrl = await uploadToCDN(outputPath);

 // 4. Notify user
 await sendEmail(payload.userId, cdnUrl);

 return { cdnUrl };
 },
});

This runs on Trigger.dev's managed compute (or your self-hosted workers). No timeout. The default machine is small-1x (0.5 vCPU, 0.5 GB RAM); you pick per task: machine: "large-2x" for the transcode step, micro for the email.

2. Wait = Checkpoint + Resource Release

When your code hits a wait point, the runtime checkpoints:

export const longWorkflow = task({
 id: "long-workflow",
 run: async (payload) => {
 const step1 = await doSomething(payload);

 // This pauses for 3 days. Zero compute billed.
 // Full process state (variables, call stack, open FDs) snapshotted via CRIU.
 await wait.for({ days: 3 });

 // Resumes HERE, with `step1` still in scope.
 const step2 = await doSomethingElse(step1);
 return step2;
 },
});

Wait points that trigger checkpoints:

wait.for({ minutes: 5 }) / wait.until(date) / wait.forToken(token)
triggerAndWait() / batchTriggerAndWait() (waiting on child tasks)
wait.forRequest() (human-in-the-loop approval)

Key detail: waits under 60 seconds don't checkpoint - the worker just blocks. This avoids overhead for short pauses.

3. Checkpoint = CRIU Snapshot → OCI Image → Registry

In Kubernetes mode (production), the coordinator:

Calls crictl checkpoint on the container → tar archive of process state
Wraps it with Buildah into an OCI image: buildah from scratch → add → config → commit → push
Stores the image reference in PostgreSQL; the blob lives in S3-compatible storage
Marks the run SUSPENDED, releases the worker, re-enqueues the run

On resume, the run is dequeued, the image is pulled, and the container is restored via CRIU. Execution continues from the instruction after the wait.

4. Warm Starts: Retries That Skip Completed Work

If a task crashes after 45 minutes of a 60-minute run, Trigger.dev can warm-start from the last checkpoint instead of restarting from zero.

export const unreliableTask = task({
 id: "unreliable-task",
 // Only checkpoint for retries if the run lasted > 30s
 retryWarmStartThresholdMs: 30_000,
 run: async (payload) => {
 const a = await step1(); // 10 min
 const b = await step2(a); // 20 min ← crashes here
 const c = await step3(b); // never reached
 return c;
 },
});

On retry, the runtime restores the checkpoint from after step2, reuses the cached result of step1 (via idempotency keys on triggerAndWait), and continues at step3. The 30 minutes of work isn't redone.

The Mental Model: One Concept - Tasks

Everything in Trigger.dev is a task. No workflows, activities, signals, or DSL.

// A task is just an async function with metadata
export const myTask = task({
 id: "my-task", // unique, used for triggering
 queue: { // optional concurrency control
 concurrencyLimit: 10,
 key: "per-user-${payload.userId}" // serialize per user
 },
 machine: "medium-1x", // CPU/RAM preset
 maxDuration: 3600, // optional cap (seconds)
 run: async (payload) => {
 // Your code. Plain TypeScript.
 // fetch, Date.now(), Math.random(), any npm package - all allowed.
 },
});

Triggering from Anywhere

// From your API route (Next.js, Express, Hono, Fastify...)
import { tasks } from "@trigger.dev/sdk/v3";

export async function POST(req: Request) {
 const handle = await tasks.trigger("process-video", {
 videoUrl: "https://...",
 userId: "user_123",
 });
 // handle.id, handle.publicAccessToken for realtime UI
 return Response.json({ runId: handle.id });
}

// From inside another task (fan-out, orchestration)
export const parent = task({
 id: "parent",
 run: async () => {
 // Fire-and-forget
 const handle = await childTask.trigger({ data: 1 });

 // Wait for result (checkpoints parent)
 const result = await childTask.triggerAndWait({ data: 2 });

 // Parallel fan-out with batch
 const handles = await batchTriggerAndWait([
 { payload: { i: 1 } },
 { payload: { i: 2 } },
 { payload: { i: 3 } },
 ]);
 },
});

Realtime: Stream Progress to Your UI

// React frontend
import { useRealtimeRun } from "@trigger.dev/react";

function VideoProgress({ runId, publicAccessToken }) {
 const { run } = useRealtimeRun(runId, publicAccessToken);

 return (
 <div>
 <p>Status: {run?.status}</p>
 <progress value={run?.metadata?.progress ?? 0} max="100" />
 {run?.output && <p>Done: {run.output.cdnUrl}</p>}
 </div>
 );
}

The run metadata updates live as your task progresses. No WebSocket server to build.

Comparison: Where Trigger.dev Wins (and Where It Doesn't)

Dimension	Trigger.dev	Inngest	Temporal	BullMQ
Model	Durable tasks (checkpoint-resume)	Durable steps (HTTP invoke + memoization)	Workflow replay (event sourcing)	At-least-once queue
Where code runs	Trigger.dev compute (Cloud) or your workers (self-host)	Your app (HTTP invoke)	Your workers	Your workers
Languages	TypeScript (Python via extension)	TS, Python, Go	Go, Java, TS, Python, .NET, more	Node.js (Python port exists)
Long-running jobs	✅ No timeout, checkpoint on wait	✅ Split into steps	✅ Continue-As-New	✅ If worker stays up
Determinism required	❌ No	❌ No	✅ Yes (workflow code)	N/A
Managed compute	✅ Cloud runs your tasks	❌ You run functions	❌ You run workers	❌ You run workers
Self-hosting	✅ Docker/K8s, same codebase	Possible, less common	✅ Heavy ops	It's a library
Flow control	Queues, concurrency keys	Best in class (throttle, debounce, priority)	Build in workflow code	Rate limits, priorities
Learning curve	Gentle	Gentle	Steep	Flat
Best for	AI pipelines, media jobs, long compute	Serverless TS apps, event-driven	Payments, sagas, regulated	Simple jobs on existing Redis

The fork is where compute runs.

Trigger.dev Cloud runs your tasks. You deploy tasks; Trigger.dev owns the queue, the execution environment, and observability.
Inngest orchestrates your serverless functions over HTTP. Your functions run on your infra (Vercel, Lambda, your VPS).
Temporal manages the server; you run workers that poll it.

If your jobs are minutes to hours of compute (video, AI, PDF, data pipelines), Trigger.dev's managed compute removes the timeout ceiling entirely. If your jobs are short steps triggered by events and you want zero new infrastructure, Inngest is simpler. If you need exactly-once workflow logic across polyglot services and will staff the ops model, Temporal is the standard.

Self-Hosting: No Lock-In, Same Codebase

Trigger.dev is Apache 2.0. The dashboard, orchestrator, workers, and CLI all live in the public monorepo. Self-hosting uses the identical runtime as Cloud.

# Docker Compose (dev/simple prod)
git clone https://github.com/triggerdotdev/trigger.dev
cd trigger.dev
docker compose up -d

# Kubernetes (Helm) - production scale
helm repo add trigger https://triggerdotdev.github.io/helm-charts
helm install trigger trigger/trigger \
 --set postgresql.enabled=true \
 --set redis.enabled=true \
 --set objectStorage.s3.bucket=my-bucket

You operate Postgres, Redis (or Valkey/Dragonfly), and the orchestrator. Cloud-only features you lose: warm starts, auto-scaling, dedicated support, managed checkpoints. Everything else - durable tasks, checkpoints, realtime, build extensions - works identically.

This matters for compliance. GovSignals runs Trigger.dev self-hosted at FedRAMP High. Your security team audits the exact code that runs your workloads. Start on Cloud, migrate to self-hosted (or vice versa) without changing a line of task code.

Build Extensions: FFmpeg, Playwright, Python - No Dockerfile

The hardest part of self-hosted workers: custom dependencies. FFmpeg for video. Playwright for scraping. Python for ML. Trigger.dev solves this at build time:

// trigger.config.ts
import { defineConfig } from "@trigger.dev/sdk/v3";
import { ffmpegExtension } from "@trigger.dev/build/extensions/ffmpeg";
import { playwrightExtension } from "@trigger.dev/build/extensions/playwright";
import { pythonExtension } from "@trigger.dev/build/extensions/python";

export default defineConfig({
 project: "my-proj",
 build: {
 extensions: [
 ffmpegExtension(), // adds ffmpeg, ffprobe
 playwrightExtension(), // adds Chromium + deps
 pythonExtension({ // run Python from TS
 requirements: ["pandas", "numpy", "openai"],
 }),
 // Or install any apt package:
 // aptGetExtension({ packages: ["libreoffice", "imagemagick"] }),
 ],
 },
});

One line in config. The CLI builds a Docker image with those binaries baked in. No Dockerfile to write, no base image to maintain, no RUN apt-get update debugging.

Real-World Patterns

AI Agent with Human-in-the-Loop

export const contentAgent = task({
 id: "content-agent",
 run: async ({ topic }) => {
 const outline = await llm.generate({ prompt: `Outline: ${topic}` });

 // Pause for human approval - days if needed, zero cost
 const approval = await wait.forToken("approval", {
 timeout: "7d",
 });

 if (!approval.approved) {
 return { status: "rejected" };
 }

 const draft = await llm.generate({ prompt: `Write: ${outline}` });
 const edited = await humanReview(draft); // another waitpoint
 return { article: edited };
 },
});

Scheduled Jobs That Survive Deploys

export const dailyReport = task({
 id: "daily-report",
 // Cron schedules are durable - attached to the task, not a separate cron process
 cron: { pattern: "0 6 * * *", timezone: "UTC" },
 run: async () => {
 const data = await fetchAnalytics();
 const pdf = await generatePDF(data);
 await uploadAndEmail(pdf);
 },
});

No cron process to monitor. No "did it run?" guesswork. The dashboard shows every run, its trace, and its output.

Multi-Tenant Queue Isolation

export const tenantJob = task({
 id: "tenant-job",
 queue: {
 // Each tenant gets their own lane; one noisy tenant doesn't starve others
 concurrencyLimit: 5,
 key: "tenant-${payload.tenantId}",
 },
 run: async ({ tenantId, ...payload }) => {
 // Process tenant's work
 },
});

Honest Gaps (as of v3)

No sandboxed code execution - coming, not shipped
No built-in agent memory across runs - you bring your own DB
TypeScript-first - Python via build extension only; no Go, Java, Rust SDKs yet
No time-based rate limiting - concurrency caps only
No offline local dev - needs connection to dev environment or Cloud
No visual workflow builder - code-first by design
No LLM cost caps per run - you implement budgeting

When to Adopt Trigger.dev

Start with Trigger.dev if:

You write TypeScript/Next.js and have jobs that exceed serverless timeouts
You want managed compute so you don't operate Redis + workers
You need realtime progress in your UI (AI generation, video processing)
You want self-hosting as an exit hatch (compliance, data residency)
You're building AI agents with human-in-the-loop, long waits, or tool calling

Stick with BullMQ if:

You already run Redis well, jobs are single-step, and you want zero vendor bill
You need raw queue primitives (priority, rate limiting, parent-child flows) and are happy operating workers

Pick Inngest if:

You deploy to Vercel/Netlify/Cloudflare and want functions to run in your app
You need best-in-class flow control (debounce, throttle, priority per key)
You want the smoothest local dev experience (HTTP invoke, no new processes)

Pick Temporal if:

Workflows touch money, healthcare, or regulated data where exactly-once matters
You need polyglot workers (Go, Java, .NET) and will invest in the learning curve
You want the most battle-tested durability model (Uber, Stripe, Coinbase scale)

The Bottom Line

Background jobs are infrastructure. Most teams don't want to build infrastructure - they want to ship features. Trigger.dev gives you durable execution as a platform, not a library you bolt onto a queue.

You write tasks. They run on managed compute. They checkpoint when they wait. They resume where they left off. You get observability, realtime, scheduling, and self-hosting for free. The code you write is the code that runs - no translation layer, no determinism tax.

GitHub: https://github.com/triggerdotdev/trigger.dev (16k+ stars, Apache 2.0)
Docs: https://trigger.dev/docs
Try it: npx trigger.dev@latest init - first deploy in ~5 minutes.

Stop managing workers. Start writing tasks that just work.

HyperSwitch: The Open-Source Payments Orchestrator That Replaces Your PSP Stack

Syed Anzar — Sat, 31 Oct 2026 12:49:09 +0000

HyperSwitch: The Open-Source Payments Orchestrator That Replaces Your PSP Stack

You integrate Stripe. Then Adyen for Europe. Then Braintree for PayPal. Then a local acquirer for Brazil. Each has its own SDK, webhook format, retry logic, and fee structure. Your codebase grows if (provider === 'stripe') branches like kudzu.

This is the problem HyperSwitch solves.

What Actually Is HyperSwitch?

HyperSwitch is an open-source payments orchestration platform — 45.6k GitHub stars, Apache 2.0, written in Rust, battle-tested at 300M daily transactions and $1T+ annual TPV by Juspay (India's largest payments processor). It sits between your application and 120+ payment processors, providing a single API that handles routing, retries, vaulting, reconciliation, and cost observability.

Think of it as the Linux of payments: you integrate once, then plug in whichever processors you need.

The Architecture That Makes It Work

Router + Scheduler: Two Services, One Job

HyperSwitch runs as two stateless Rust binaries:

Router — The hot path. Handles every payment request: authentication, FRM checks, 3DS decisions, connector selection via the Euclid routing engine, credential resolution from the vault, and connector dispatch through the Connector Abstraction Layer (CAL).

Scheduler — Background jobs. Producer batches scheduled tasks (card cleanup, API key expiry notices) into Redis queues; Consumer executes them. Horizontal scaling is trivial — any instance can handle any request.

All state lives in PostgreSQL. Redis handles caching, distributed locks, and the scheduler queue.

The Payment State Machine: Intent vs Attempt

This is where most orchestrators get it wrong. HyperSwitch separates two concepts:

payment_intent — The merchant's request. One per checkout session. Holds amount, currency, customer, capture method, setup_future_usage.
payment_attempt — A single authorization try against a specific PSP. Multiple attempts per intent (retries). Each has its own connector, connector_transaction_id, error_code, error_details.

When a retry happens, a new payment_attempt is created. The merchant-facing payment_id never changes — only the internal suffix (pay_abc_1, pay_abc_2) sent to the PSP changes.

Euclid: Routing That Isn't Just Round-Robin

The Euclid routing engine evaluates strategies in strict priority order:

Rule-based — Merchant-defined conditions: payment_method_type = visa AND country = DE AND amount > 10000 → use Adyen
Volume-based — Percentage split across eligible connectors
Cost-based — Select lowest MDR (merchant discount rate) among eligible
Default fallback — Ordered priority list from merchant config

After selection, eligibility analysis runs: Does this connector support the payment method + currency? Is it enabled for this profile? If any check fails, Euclid moves to the next candidate. Exhaust the list → payment fails without a single PSP call.

GSM: Normalizing 120+ Error Languages

Every PSP returns proprietary error codes. Stripe's card_declined ≠ Adyen's REFUSED ≠ Braintree's 2001. GSM (Global Status Monitoring) maps (connector, error_code, error_message) → standardized unified_code + retry flags:

Flag	Meaning
`retriable`	Any retry eligible
`step_up_possible`	3DS challenge could succeed
`clear_pan_possible`	Network token failed, try raw PAN
`alternate_network_possible`	Switch debit network (Visa → Maestro)

The retry decision flow is deterministic: GSM lookup → check flags in order (Step-Up → Clear PAN → Network → Cascading) → do_retry() creates new payment_attempt → re-enter Euclid.

The Vault: PCI Compliance Without the Pain

The Card Vault is a separate microservice. Sensitive data never touches your Router — or your application if you use the client-side SDK.

Tokenization flow (client-side): SDK → Vault /session_tokens → JWE-encrypted request → Vault decrypts, stores PAN, returns payment_method_id → Your backend sends payment_method_id to Router → Router calls Vault Locker API → Vault returns credential (PAN, network token, or proxy reference) → Router dispatches to PSP.

Key management: Custodian-based. Master key derived from multiple custodian shares (Shamir's Secret Sharing). Vault only activates when quorum of custodians unlock it. Zero data egress in self-hosted mode.

Reconciliation: Double-Entry for Payments

Merchants routing across PSPs get separate settlement reports — different formats, different schedules. HyperSwitch's reconciliation engine applies double-entry bookkeeping: every payment event generates balanced debit/credit ledger entries.

2-way: Hyperswitch ledger ↔ PSP settlement report
3-way: Hyperswitch ledger ↔ PSP settlement ↔ OMS order data

Auto-fetch pipelines pull reports from PSPs (SFTP, API, email), normalize via connector-specific rules, match on connector_transaction_id + amount + date with configurable tolerance windows, age unmatched entries, and surface exceptions in the Control Center.

What It Replaces

Paid Stack	HyperSwitch Module
Stripe + Adyen + Braintree integrations	Single API, 120+ connectors
Custom routing logic	Euclid engine (rules, volume, cost)
Manual retry code	GSM-driven smart retries (4 strategies)
PCI vault (TokenEx, VGS)	Built-in Card Vault + Network Tokens
Spreadsheet reconciliation	Automated 2/3-way engine
DataDog/New Relic for payments	Cost Observability dashboards

Self-Hosting: What You Actually Need

# docker-compose.yml (simplified)
services:
  router:
    image: juspaydotin/hyperswitch-router:latest
    environment:
      - DATABASE_URL=postgres://...
      - REDIS_URL=redis://...
      - LOCKER_URL=http://vault:8080
    ports: ["8080:8080"]

  vault:
    image: juspaydotin/hyperswitch-locker:latest
    environment:
      - DATABASE_URL=postgres://...
      - CUSTODIAN_KEYS=...
    ports: ["8081:8081"]

  scheduler:
    image: juspaydotin/hyperswitch-scheduler:latest
    environment:
      - DATABASE_URL=postgres://...
      - REDIS_URL=redis://...

  postgres: ...
  redis: ...

Deploy via Docker Compose, Kubernetes (Helm), or AWS CloudFormation. Control Center (React dashboard) runs separately for routing config, analytics, and exception management.

The Trade-offs

Not a drop-in Stripe replacement. You operate infrastructure. PCI-SSS validation requires the Encryption Service + custodian ceremony. Connector coverage is broad but not infinite — niche local methods may need custom connectors (template provided). The learning curve is real: routing rules, GSM mappings, vault activation.

But you get: Zero vendor lock-in. Full visibility into every hop. Routing logic in config, not code. No per-transaction markup. Apache 2.0 — fork it, extend it, audit it.

When to Adopt

Multiple PSPs already, or planning to add them
Team capable of running PostgreSQL + Redis + Rust services
Need routing/retries/vault/reconciliation as configurable infrastructure
Willing to invest in ops for long-term control

Start Here

# Quick local test
git clone https://github.com/juspay/hyperswitch
cd hyperswitch/docker
docker compose up -d
# Router at localhost:8080, Control Center at localhost:3000

Read the architecture deep-dive for the Router internals, Vault architecture for tokenization internals, and Reconciliation architecture for the ledger model.

The payments stack you'd build if you had 12 years and 300M daily transactions to refine it. Now you don't have to.

What Actually Happens When You Resolve a Domain Name

Syed Anzar — Sat, 31 Oct 2026 12:22:41 +0000

What Actually Happens When You Resolve a Domain Name

You type example.com in your browser. Milliseconds later, the page loads. What happened in between?

Most developers know DNS "turns names into IPs." Few can explain what actually happens when the cache is cold. Let's trace the full path—from your stub resolver to the root servers, through TLDs, to the authoritative answer—and the protocol details that make it work.

The Big Picture

DNS resolution is a distributed delegation chain. No single server knows everything. Each server knows only its zone and where to find the next level down.

Client → Stub Resolver → Recursive Resolver → Root → TLD → Authoritative → Answer

Your browser doesn't talk to root servers. It talks to a stub resolver (OS DNS client). That stub forwards to a recursive resolver (your ISP, 1.1.1.1, 8.8.8.8, or a local Unbound/BIND). The recursive resolver does the heavy lifting.

Step 0: The Cache Check

Before any network traffic, the recursive resolver checks its cache:

Exact match? Return immediately.
CNAME chain? Follow it, then return the final target.
Negative cache (NXDOMAIN/NODATA)? Return "doesn't exist" without querying upstream.
Partial match (NS records for a parent zone)? Start from there—this is the closest delegation point.

If the cache is cold (resolver just restarted), it has only root hints: a built-in list of 13 root server names (a.root-servers.net through m.root-servers.net) and their IPs.

Step 1: Priming — Getting the Real Root List

Root hints are hints. They can be stale. Before trusting them, the resolver sends a priming query (RFC 8109):

QNAME: . (root)
QTYPE: NS
RD: 0 or 1 (undefined for priming)
EDNS(0): YES, UDP payload ≥ 1024 bytes
DO bit: optionally set

Sent to one root hint address (selected randomly to spread load). The root responds with:

Answer section: NS RRset for . (the 13 root server names)
Additional section: A/AAAA records for those names (glue records)
AA bit = 1: authoritative answer

This primes the resolver with the current, authoritative root NS set and addresses. TTL is typically 6 days (518,400 seconds).

Step 2: The Descent — Iterative Queries

Now the resolver has a working root NS set. It begins iterative resolution (RD=0) for www.example.com:

Query to Root (`.`)

QNAME: www.example.com (or minimized: com — see QNAME minimization below)
QTYPE: A (or AAAA)
RD: 0

Root doesn't know example.com. It's only authoritative for .. It responds with a referral:

Authority section: NS records for .com (the TLD nameservers)
Additional section: Glue A/AAAA for those NS names (e.g., a.gtld-servers.net)

Query to TLD (`.com`)

Resolver picks one .com nameserver, sends:

QNAME: www.example.com (or minimized: example.com)
QTYPE: A
RD: 0

TLD server isn't authoritative for example.com either. Another referral:

Authority: NS records for example.com (your domain's authoritative nameservers)
Additional: Glue records if the NS names are inside example.com (e.g., ns1.example.com)

Query to Authoritative (`example.com`)

Resolver picks one authoritative NS for example.com:

QNAME: www.example.com
QTYPE: A
RD: 0

This server is authoritative. It responds with:

Answer section: The A record (e.g., 93.184.216.34)
AA bit = 1: "I own this data"
TTL: How long to cache (e.g., 3600 seconds)

Resolver caches the answer, the NS records, and glue—and returns it to the stub resolver.

QNAME Minimization: Privacy by Design (RFC 7816 / 9156)

Traditional resolvers send the full QNAME (www.example.com) to every server in the chain. The root doesn't need www.example.com—it only needs .com. The TLD only needs example.com.

QNAME minimization sends only what each server needs:

Server	Traditional Query	Minimized Query
Root	`www.example.com` A	`com` NS
TLD	`www.example.com` A	`example.com` NS
Auth	`www.example.com` A	`www.example.com` A

The resolver tracks zone cuts (delegation boundaries) in its cache. When it doesn't know where the cut is, it probes one label at a time (NS queries), caching NODATA responses to learn the boundary.

Why it matters: Root servers see only com, not your full browsing history. Reduces data exposure to authoritative operators. Also enables aggressive negative caching (RFC 8020): if .example doesn't exist, nothing under it exists either.

EDNS(0): Breaking the 512-Byte Limit (RFC 6891)

Original DNS: 512-byte UDP max. DNSSEC responses (with RRSIG, DNSKEY, NSEC) routinely exceed this.

EDNS(0) adds an OPT pseudo-record in the additional section:

NAME: . (root)
TYPE: OPT (41)
CLASS: Requestor's UDP payload size (e.g., 4096)
TTL: Extended RCODE (high 8 bits) + flags (DO bit in bit 15)
RDATA: Variable {option-code, option-length, option-data} tuples

Key fields:

UDP payload size: "I can receive this many bytes." Default 4096. Values < 512 treated as 512.
DO bit (DNSSEC OK): "Send me DNSSEC records."
Options: COOKIE (RFC 7873), ECS (RFC 7871), PADDING, etc.

If a responder can't fit the answer in the advertised size, it sets TC=1 (truncated). Resolver retries over TCP.

Middlebox warning: Firewalls/load balancers that strip OPT records or block >512-byte UDP break DNSSEC and large responses. RFC 6891 says they MUST NOT do this—but they do.

The Kaminsky Attack & Why Source Port Randomization Matters

In 2008, Dan Kaminsky showed how to poison a resolver's cache off-path.

The vulnerability: Resolvers accepted the first response matching:

Query name + QTYPE
16-bit Transaction ID (TXID)
Source/destination IP
Source/destination port (often fixed at 53)

Attacker floods resolver with spoofed responses, guessing TXID. With fixed source port, only 65,536 guesses needed—trivial.

The fix: Source port randomization (RFC 5452). Resolver picks a random ephemeral source port per query. Now attacker must guess TXID (16 bits) + source port (~15 bits) = ~31 bits of entropy. At 50,000 packets/sec, success probability drops from ~100% in seconds to ~64% in 24 hours with ~0.4 TB of traffic.

Remaining weaknesses:

NAT devices with predictable port allocation (sequential, port-preserving) reduce entropy.
ICMP side channels can leak source ports on some OS/resolver combos (CCS 2020).
Real fix: DNSSEC (cryptographic validation). Port randomization is a band-aid; DNSSEC is the cure.

DNSSEC: The Chain of Trust

DNSSEC signs zones with asymmetric crypto. Each zone has:

KSK (Key Signing Key): Signs the DNSKEY RRset. Published as DS record in parent zone.
ZSK (Zone Signing Key): Signs all other records in the zone.

Validation flow for www.example.com:

Resolver gets A record + RRSIG from example.com authoritative.
Resolver gets DNSKEY (ZSK) for example.com + RRSIG.
Resolver gets DS record for example.com from .com TLD + RRSIG.
Resolver gets DNSKEY (KSK) for .com from .com authoritative + RRSIG.
Resolver gets DS record for .com from root + RRSIG.
Resolver gets DNSKEY for root (trust anchor, pre-configured).

If every signature verifies up to the trust anchor → AD bit = 1 (Authenticated Data). If any break → SERVFAIL.

NSEC/NSEC3: Prove non-existence. NSEC lists "next secure name"—enables zone walking. NSEC3 hashes names (salted) to prevent enumeration.

Common Misconceptions

Myth	Reality
"My browser queries root servers"	Browser → stub → recursive resolver. Recursive does the descent.
"DNS is a single database"	Distributed delegation. Each zone owns its data.
"TTL is a guarantee"	TTL is a hint. Resolvers may cap it (e.g., 2-day max). Negative TTL from SOA MINIMUM.
"CNAME at apex works"	RFC 1912: CNAME at zone apex breaks NS/SOA. Use ALIAS/ANAME (non-standard) or flattening (Cloudflare).
"DNSSEC encrypts queries"	DNSSEC signs data (authenticity/integrity). It does NOT encrypt. Use DoH/DoT/DoQ for privacy.
"13 root servers = 13 machines"	13 identities (a–m.root-servers.net). Each is a global anycast constellation (hundreds of physical servers).

Practical Takeaways

Run a local validating resolver (Unbound, Knot Resolver). Cuts latency, enables DNSSEC validation, supports QNAME minimization.
Enable DNSSEC validation. It's the only real protection against cache poisoning.
Understand TTL caps. Don't rely on 60-second TTLs for fast failover—resolvers may ignore them.
Monitor your glue. If you rename NS hosts inside your zone, update glue at the registrar before the old records expire.
Use dig +trace to see the actual delegation chain. dig +dnssec to see validation.

The Mental Model

DNS isn't a lookup table. It's a tree of delegations, each signed (optionally), cached at every level, with privacy minimizations and protocol extensions layered on top. The next time you hit example.com, you're not "querying a database." You're walking a chain of trust from root to leaf—one referral at a time.

Tags: dns, networking, systems, security

Test Article

Syed Anzar — Sat, 31 Oct 2026 12:21:04 +0000

Test

Test content

SigNoz: The Open-Source Observability Platform That Actually Replaces Datadog

Syed Anzar — Fri, 30 Oct 2026 13:26:47 +0000

SigNoz: The Open-Source Observability Platform That Actually Replaces Datadog

You've probably heard "SigNoz is the open-source Datadog alternative" in every other dev post. But most explanations stop at "it uses OpenTelemetry and ClickHouse" -- as if that explains why it works.

It doesn't.

The real difference isn't the tech stack. It's how SigNoz solves the fundamental problem every observability platform faces: how to store and query logs, metrics, and traces together without melting your infrastructure or your budget.

The Problem Everyone Lives With

If you've used Datadog, New Relic, or any major SaaS observability platform, you know the drill:

Logs go to one system (Elasticsearch, Loki, CloudWatch)
Metrics go to another (Prometheus, Datadog metrics, InfluxDB)
Traces go to a third (Jaeger, Zipkin, Tempo)
Correlation happens in your head -- you switch tabs, copy trace IDs, pray the timestamps align

This isn't an accident. It's the natural consequence of how these systems evolved. Logs need full-text search. Metrics need time-series aggregation. Traces need span trees and latency percentiles. Three different query patterns. Three different storage engines.

Datadog solves this by charging you for three different products and gluing them together in the UI. You pay per host, per custom metric, per log ingested, per trace retained. The bill grows faster than your traffic.

SigNoz takes a different bet: one columnar database for everything.

Why ClickHouse Changes the Game

ClickHouse isn't just "fast." It's a column-oriented OLAP database built for analytical queries over massive datasets. Here's why that matters for observability:

One Query Language for All Signals

In SigNoz, a trace, a log line, and a metric live in the same database. You can write a single SQL query that joins spans with logs and aggregates metrics:

-- Find all errors in the payment service, show related logs, and calculate error rate
SELECT
    toDateTime(ts_bucket_start) as time,
    countIf(has_error = 1) as error_count,
    count() as total_spans,
    error_count / total_spans as error_rate
FROM signoz_index_v3
WHERE service_name = 'payment-service'
  AND ts_bucket_start >= now() - INTERVAL 1 HOUR
GROUP BY time
ORDER BY time DESC

Try doing that across Elasticsearch + Prometheus + Jaeger. You can't. You'd need application-level correlation, which means more code, more latency, more things to break.

High-Cardinality Without the Tax

This is the killer feature nobody talks about.

In Prometheus/Datadog, every unique label combination creates a new time series. Add request_id, user_id, session_id to your metrics and you've created a cardinality explosion. Your bill explodes. Your query latency explodes. You're told "don't do that."

In ClickHouse, high-cardinality attributes live in typed Map columns. They're compressed with ZSTD. They don't create new time series. You can filter by user_id across billions of spans. It just works.

SigNoz takes this further: hot attribute keys get promoted to materialized columns automatically. The system adapts to your query patterns.

Compression That Actually Matters

ClickHouse uses per-column codecs tuned for observability data:

Data Type	Codec	Why It Works
Timestamps	`DoubleDelta`	Monotonic increasing = tiny deltas
Strings (labels, trace IDs)	`ZSTD`	Dictionary + entropy coding
Metric values (latencies, counts)	`Gorilla`	XOR-based floating-point compression
Integers (durations, counts)	`T64`	Variable-byte encoding

Typical observability datasets compress 10-20x. That's not marketing -- it's columnar storage doing what it was designed for.

The Architecture: Three Planes, Not Five Microservices

SigNoz used to ship five separate containers: query-service, frontend, alertmanager, otel-collector, ClickHouse. That was v0.75 and earlier.

Since v0.76 (March 2025), it's three components:

+---------------------------------------------------------------+
|                    SigNoz Binary (single Go binary)           |
|  +----------+ +----------+ +----------+ +------------------+  |
|  |  React   | |  API     | |  Ruler   | |  Alertmanager    |  |
|  |  UI      | |  Server  | |  (alerts)| |  (notifications) |  |
|  +----------+ +----------+ +----------+ +------------------+  |
|                        |                                     |
|                  +-----+-----+                               |
|                  |  OpAMP    | <-- Manages collector config  |
|                  |  Server   |                               |
|                  +-----------+                               |
+---------------------------------------------------------------+
                              |
                              v
+---------------------------------------------------------------+
|              SigNoz OTel Collector (ingestion plane)          |
|  OTLP gRPC/HTTP * Jaeger * Zipkin * Prometheus * Kafka       |
|  Custom processors: span metrics, span mapper, LLM pricing   |
+---------------------------------------------------------------+
                              |
                              v
+---------------------------------------------------------------+
|                      ClickHouse (+ Keeper/ZooKeeper)          |
|  signoz_traces * signoz_logs * signoz_metrics * signoz_meter |
|  Resource fingerprint tables for fast filtering              |
+---------------------------------------------------------------+

The Ingestion Plane: SigNoz OTel Collector

This isn't a vanilla OpenTelemetry Collector. It's a curated distribution with SigNoz-specific components:

Receivers: OTLP (gRPC 4317, HTTP 4318), Jaeger, Zipkin, Prometheus, Kafka, OpenCensus, HTTP logs (Heroku, JSON)

Processors that matter:

signozspanmetrics/delta -- computes RED metrics (Rate, Errors, Duration) from spans automatically. No separate metrics pipeline needed.
signozspanmapper -- normalizes framework-specific attributes (LangChain, LlamaIndex, etc.) into standard gen_ai.* semantic conventions
signozllmpricing -- multiplies token counts by per-model prices, writes cost attributes per span
batch -- 50k spans / 5 seconds (ClickHouse best practice: large, infrequent inserts)

Exporters: ClickHouse (traces, logs, metrics, meter), metadata exporter (drives query builder autocomplete)

Connector: signozmeter -- counts ingested volume per service/environment/host, powers usage dashboards

Extension: OpAMP health check + managed config. The collector connects to the OpAMP server inside the SigNoz binary, receives its config remotely, and applies it without redeployment. Log pipelines, span mapper rules, LLM pricing -- all editable in the UI.

The Storage Plane: ClickHouse Schema Design

This is where most observability platforms struggle. SigNoz's schema is worth understanding because it explains the query performance.

Traces (signoz_index_v3) and Logs (logs_v2) share the same layout:

Main table:           Resource table (companion):
+-----------------+   +---------------------------+
| ts_bucket_start |   | resource_fingerprint (PK) |
| resource_fp ----+-->| labels (string map)       |
| trace_id        |   | seen_at_ts_bucket_start   |
| span_id         |   +---------------------------+
| duration_nano   |
| has_error       |
| ...attributes...|
+-----------------+

Resource attributes (service.name, k8s.namespace.name, deployment.environment) are hashed into a resource_fingerprint. The fingerprint is stored once per time bucket in the companion table. The main table only carries the fingerprint + timestamp bucket.

Query execution:

Filter resource table by attributes (e.g., service.name = 'payment-service')
Get matching fingerprints
Scan main table granules: resource_fingerprint GLOBAL IN (...) + ts_bucket_start range

This is why SigNoz docs tell you to always filter on resource attributes first. Without a resource filter, you scan all granules. With it, you scan only the relevant ones.

Metrics (signoz_metrics_v2) use a different layout optimized for time-series: metric name + labels as columns, value + timestamp. PromQL and ClickHouse SQL both work.

TTL strategy: Daily partitions, ttl_only_drop_parts = 1. Retention drops whole parts (15 days for logs/traces, 30 days for metrics) -- no row-level deletes, no merge overhead.

What You Actually Get: Feature Walkthrough

APM That Works Out of the Box

Instrument your app with OpenTelemetry (auto-instrumentation exists for Java, Python, Node, Go, .NET, PHP, Ruby). Send to SigNoz OTel Collector. You get:

Service map with latency/error/throughput per service
Top endpoints with p50/p90/p99 latency, error rate, Apdex
Database calls, external calls, message queue spans -- all correlated
No dashboard building required. The APM views are generated from span data.

Logs That Correlate With Traces

Click a trace span -> see all logs with the same trace_id. Click a log -> jump to its trace. This works because both share the resource fingerprint + time bucket layout. No separate correlation engine needed.

Metrics From Spans (No Separate Pipeline)

The signozspanmetrics processor computes delta-temporality histograms and call counts per service/operation every 60 seconds. Dimensions: service.namespace, deployment.environment, service.version. Your APM dashboards are powered by span-derived metrics -- you don't need a separate Prometheus scrape setup for RED metrics.

LLM/AI Observability Built In

Since v0.143, the collector includes:

signozspanmapper -- normalizes input.value, gen_ai.prompt -> standard gen_ai.input.messages / gen_ai.output.messages
signozllmpricing -- reads gen_ai.usage.*_tokens, multiplies by model prices (pushed via OpAMP), writes signoz.gen_ai.usage.*.cost

You get token usage, latency, cost per request, model breakdown -- all in the same traces view.

Alerting That Doesn't Require a Separate Stack

The ruler (inside the SigNoz binary) evaluates alert rules on a schedule. Queries run against ClickHouse. Alertmanager handles deduplication, grouping, notification routing (Slack, PagerDuty, OpsGenie, webhook, email). Threshold, anomaly, and Apdex-based alerts all supported.

Dashboards: Query Builder, PromQL, or Raw ClickHouse SQL

Three query modes:

Query Builder -- visual, no-code, generates ClickHouse SQL
PromQL -- for Prometheus migrants, runs via ClickHouse's PromQL-compatible engine
ClickHouse SQL -- full power for complex joins, cross-signal correlation

Dashboard panels can mix all three. Variables, templating, drill-downs -- standard fare.

Deployment: From Laptop to Production

Local (Docker Compose)

git clone https://github.com/SigNoz/signoz.git
cd signoz/deploy/docker/clickhouse-setup
./install.sh
# UI at http://localhost:8080

Bare Metal (systemd via Foundry)

# Foundry = declarative CLI for self-hosted SigNoz
# One config provisions infra, installs backend, sets up OTel collection
foundry apply -f config.yaml

Supports Docker, Swarm, systemd, Kubernetes, ECS, bare metal. Single binary + ClickHouse + Keeper.

Kubernetes (Helm)

helm repo add signoz https://charts.signoz.io
helm install signoz signoz/signoz -n platform

Production: replicated ClickHouse (3+ shards), 3-node Keeper ensemble, PostgreSQL metastore, multiple SigNoz replicas behind ingress.

SigNoz Cloud

Fully managed. 30-day free trial, usage-based pricing starting at $49/month. SOC 2 Type II, HIPAA, RBAC, SSO, BYOC option.

The Trade-offs (Be Honest)

SigNoz isn't magic. Here's what you're signing up for:

Advantage	Cost
No vendor lock-in -- OpenTelemetry native, your instrumentation is portable	You operate ClickHouse -- need fast local disks (NVMe), CPU for merges, Keeper/ZooKeeper quorum
Predictable pricing -- no per-host, per-custom-metric, per-user-seat fees	Schema migrations -- upgrades require running the migrator job, occasional breaking changes
Single database for all signals -- cross-signal correlation in SQL	Operational complexity -- more moving parts than a SaaS dashboard
High-cardinality friendly -- filter by user_id, request_id freely	Metrics cardinality still costs -- unbounded labels on metrics create time series
Self-hostable on bare metal -- no Docker/K8s required	Foundry learning curve -- new tool, replaces raw Compose files since v0.130

The AGPL collector matters: The OTel Collector distribution is AGPL-3.0. If you modify and redistribute it, you must open-source your changes. Running it unmodified? No issue. Building a managed service on top? Talk to a lawyer.

When SigNoz Makes Sense

You're already on OpenTelemetry or willing to adopt it (vendor-free instrumentation)
You want correlated logs/metrics/traces without stitching three SaaS products
High-cardinality attributes are part of your debugging workflow
You have infrastructure capacity (or budget for managed Cloud) to run ClickHouse
You prefer SQL/PromQL over proprietary query languages

When to Stick With Datadog/New Relic

You only need infrastructure monitoring (host maps, cloud integration depth)
You don't want to operate a database -- SaaS wins on operational simplicity
You need Log Patterns / anomaly detection on logs -- SigNoz doesn't have this yet (roadmap)
Your team is small and observability isn't a core competency

Quick Start: Instrument a Go Service in 5 Minutes

// main.go
package main

import (
    "context"
    "net/http"
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
    "go.opentelemetry.io/otel/sdk/resource"
    "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.26.0"
)

func initTracer() (*trace.TracerProvider, error) {
    exporter, err := otlptracehttp.New(context.Background(),
        otlptracehttp.WithEndpoint("localhost:4318"),
        otlptracehttp.WithInsecure(),
    )
    if err != nil {
        return nil, err
    }

    tp := trace.NewTracerProvider(
        trace.WithBatcher(exporter),
        trace.WithResource(resource.NewWithAttributes(
            semconv.SchemaURL,
            semconv.ServiceName("go-demo"),
            semconv.DeploymentEnvironment("development"),
        )),
    )
    otel.SetTracerProvider(tp)
    return tp, nil
}

func main() {
    tp, _ := initTracer()
    defer tp.Shutdown(context.Background())

    http.HandleFunc("/hello", func(w http.ResponseWriter, r *http.Request) {
        ctx, span := otel.Tracer("go-demo").Start(r.Context(), "hello-handler")
        defer span.End()

        // Simulate work
        span.AddEvent("processing")
        w.Write([]byte("Hello, SigNoz!"))
    })

    http.ListenAndServe(":8080", nil)
}

Run SigNoz locally (./install.sh in the docker compose dir), run this service, hit /hello a few times. Open http://localhost:8080 -> Traces -> go-demo. You'll see the span, latency, and if you add logs, they'll correlate automatically.

The Bigger Picture

SigNoz represents a shift in how observability is built: open standards + columnar storage + unified query layer instead of proprietary agents + fragmented storage + per-signal pricing.

The industry is moving toward OpenTelemetry anyway. Datadog, New Relic, Grafana, Splunk -- all support OTLP ingestion now. But they still charge you for the storage and query layer they built around it.

SigNoz says: the storage and query layer should be open too.

Repository: github.com/SigNoz/signoz (32k+ stars)
Docs: signoz.io/docs
Slack: signoz.io/slack
Try Cloud: signoz.io/teams (30-day free trial)

Tags: opensource, observability, opentelemetry, clickhouse, datadog-alternative, selfhosted

TigerBeetle: What Actually Happens When You Need a Database That Can't Lose Money

Syed Anzar — Fri, 30 Oct 2026 11:16:52 +0000

TigerBeetle: What Actually Happens When You Need a Database That Can't Lose Money

You use PostgreSQL. You use MySQL. You've probably heard of Redis, Cassandra, maybe even CockroachDB. But there's a database that financial infrastructure companies quietly rely on for the hot path of money movement—and almost no application developers know it exists.

TigerBeetle.

It's not a general-purpose database. It doesn't do JOINs. It doesn't have a query planner. It doesn't support arbitrary schemas. What it does is one thing: financial transactions at a million transfers per second on a single core, with durability guarantees that survive disk corruption, network partitions, and power loss.

Here's what actually happens under the hood.

The Problem: OLTP Isn't OLGP

Most developers think of databases as "places to store data." But there's a fundamental split:

OLGP (Online General Purpose) — PostgreSQL, MySQL, SQLite. Flexible schemas, ad-hoc queries, ACID transactions, indexes for everything. Great for "what's this user's order history?"

OLTP (Online Transaction Processing) — TigerBeetle. Rigid schema (double-entry bookkeeping), no ad-hoc queries, deterministic execution, hardware-aware storage. Great for "move $42.37 from account A to account B, right now, and never lose it."

The world is becoming more transactional. India's UPI processed 16.9 billion payments in January 2025 alone. Serverless APIs bill by the millisecond. Smart meters trade energy by the kilowatt-hour every 15 minutes. General-purpose databases weren't built for this write-heavy, high-contention reality.

TigerBeetle was.

Double-Entry Isn't Just Accounting—It's a Data Model

Every transfer in TigerBeetle has:

debit_account_id — where value comes from
credit_account_id — where value goes to
amount — unsigned 128-bit integer (no floating point, ever)
ledger — which asset type (USD, BTC, kilowatt-hours, loyalty points)
code — business event type (refund, payout, fee, etc.)
user_data_128 — your idempotency key (critical for exactly-once)

That's it. No NULLs. No variable-length strings. Every transfer is exactly 128 bytes, cache-line aligned. The entire schema is baked into the binary.

Why double-entry? Because credits == debits is an invariant the database can enforce without application logic. If the sum ever diverges, something is corrupted. This isn't just accounting hygiene—it's a correctness check that runs at wire speed.

Single-Threaded by Design (And Faster for It)

"Wait, single-threaded? In 2026?"

Yes. And here's why.

Financial workloads are contention-heavy. A few accounts (the "hot" ones) get hammered—think a payment processor's settlement account, or a popular creator's payout wallet. Transfers between hot accounts must serialize. No amount of sharding or parallelism fixes this; Amdahl's Law sees to that.

TigerBeetle leans into it: one thread, one event loop, zero locks, zero mutexes, zero GC pauses (it's written in Zig, no runtime). The commit function that executes a batch of transfers is a tight CPU loop:

for (transfers) |transfer| {
    debit = accounts[transfer.debit_account_id];
    credit = accounts[transfer.credit_account_id];

    if (debit.balance < transfer.amount) {
        result = .insufficient_funds;
        continue;
    }

    debit.balance -= transfer.amount;
    credit.balance += transfer.amount;
    result = .success;
}

Two hash-map lookups, a bounds check, two integer additions. That's the "speed of light" for this problem. TigerBeetle gets within striking distance by eliminating everything else: no network round-trips to application code, no lock acquisition, no page cache indirection.

The Prefetch Trick: IO Parallelism Without Thread Parallelism

Here's the clever part. The commit function doesn't read from disk. It only touches in-memory state.

Before commit runs, a separate prefetch phase analyzes the batch, predicts which accounts will be touched, and issues parallel async IO to load them into memory. While commit executes sequentially on one core, the storage subsystem is doing parallel reads across all available IO queues (io_uring on Linux).

When commit starts, the data is already hot. Zero stall cycles.

Consensus: Viewstamped Replication, Not Raft

TigerBeetle replicates across 6 replicas using a Viewstamped Replication derivative (VSR)—the same lineage as Raft, but with a critical difference: the log is the source of truth, not the state machine.

Each replica stores a single file: the "data file." It contains:

A hash-chained Write-Ahead Log (WAL) of "prepares" (batches of ~8,000 transfers)
A forest of LSM trees for the derived state (account balances, transfer history)

The primary accepts requests, orders them, creates prepares, appends to WAL, replicates to backups. Quorum (4 of 6) = committed. Replicas apply prepares in sequence number order. Deterministic state machine = identical state everywhere.

But here's what makes it unusual: the data file uses copy-on-write blocks with checksums at every level. Blocks are 0.5 MiB. Each block references children by (offset, checksum). The superblock (root) is atomically flushed to form checkpoints.

If a replica crashes, it reads the last checkpoint, replays the WAL suffix, and arrives at the exact same state. Determinism guarantees it.

Durability That Survives Silent Corruption

This is where TigerBeetle diverges from almost every other database.

Most databases trust fsync(). If fsync() returns success, the data is "safe." But fsync() can lie—silent disk corruption, controller bugs, helical corruption (where a write lands on the wrong track). Jepsen found TigerBeetle survives all of this.

How? Logical AND physical determinism.

Every block has a checksum. Every block references children by checksum. The hash chain from superblock to leaf mathematically proves the data wasn't corrupted. If a replica reads a block and the checksum mismatches, it doesn't panic—it fetches the same block from another replica (which has the identical hash chain) and repairs itself transparently.

The system doesn't distinguish between "local disk" and "remote replica" for checksummed data. It's all just "sources of blocks with verified checksums." This is why TigerBeetle can guarantee durability even when storage lies.

The Architecture Around It: Gateways, Not Direct Access

You don't connect your frontend to TigerBeetle. The architecture looks like:

[Mobile App] → [API Gateway] → [TigerBeetle Cluster]
                    ↓
            [PostgreSQL (metadata)]

The gateway:

Authenticates/authorizes
Generates idempotency keys (128-bit, client-persisted)
Batches individual transfers into large prepares
Maps string enums ("refund", "USD") to TigerBeetle's integer codes
Caches ledger/account metadata (which never changes—append-only)

TigerBeetle only sees integer IDs. It never does string parsing, never talks to untrusted clients, never does auth. It just processes transfers.

What You Don't Get (And Why That's the Point)

Feature	TigerBeetle
SQL	❌
JOINs	❌
Secondary indexes (arbitrary)	❌
Arbitrary schemas	❌
Interactive transactions	❌
Ad-hoc analytics	❌
Horizontal sharding	❌ (6 replicas = HA, not scale-out)

If you need to query "all transfers for user X in March," you do that in PostgreSQL. TigerBeetle stores the transfer, PostgreSQL stores the metadata linking transfer IDs to user IDs.

This separation is intentional. The hot path (recording money movement) stays fast and simple. The control plane (who owns what, what's the currency name) stays flexible.

When You Actually Need This

Payment processors (millions of txns/sec)
Marketplace payouts (high contention on creator wallets)
Ledger services (immutable audit trail required)
Real-time billing (per-second serverless metering)
Loyalty/points systems (high volume, low value, strict balances)

When you don't need it:

User profiles, product catalogs, content management
Anything with ad-hoc query patterns
Low-write, read-heavy workloads
Prototyping (operational complexity of 6-replica cluster)

The Mental Model Shift

Stop thinking "database = general purpose store." Start thinking:

Hot path = specialized OLTP database. Cold path = general purpose OLGP database.

TigerBeetle doesn't replace PostgreSQL. It sits beside it, handling the one thing PostgreSQL struggles with: high-contention, write-heavy, durability-critical financial transfers at scale.

And it does that one thing so well that a single core handles a million transfers per second—while your PostgreSQL cluster chokes on lock contention for the same workload.

TigerBeetle is open source (Apache 2.0), written in Zig, and production-ready. The architecture docs are unusually readable for a systems project. The single-page docs get you running in minutes.

How Function Calling Actually Works: Grammar-Constrained Decoding Under the Hood

Syed Anzar — Thu, 29 Oct 2026 07:34:51 +0000

How Function Calling Actually Works: Grammar-Constrained Decoding Under the Hood

You send a function definition with a JSON schema. The model responds with a perfect tool call — valid JSON, correct types, enum values from the allowed list, required fields present. No parsing errors. No retries.

It looks like the model "understands" the schema. It doesn't.

What actually happens: your JSON schema gets compiled into a context-free grammar. That grammar drives a pushdown automaton at inference time. At every decoding step, the automaton computes which vocabulary tokens would keep the output valid. Those tokens keep their logits. Every other token gets its logit set to negative infinity. The model samples from what remains.

The model still decides which tool to call and what arguments to pass. But the braces, commas, quotes, property names, type constraints, and enum restrictions? Those are mechanically guaranteed. The model literally cannot produce invalid structure.

The Problem: Free Text Doesn't Guarantee Structure

An LLM trained on next-token prediction assigns probabilities to every token in its vocabulary. Nothing in the training objective forces the sequence to form valid JSON, match a regex, or satisfy a context-free grammar.

The failure modes are familiar:

Missing or extra braces — the model loses track of nesting depth
Wrong type — an integer field gets a quoted string
Hallucinated enum values — the model invents a value outside the allowed set
Truncated output — the model emits EOS mid-object
Key name drift — user_id becomes userId under paraphrase pressure

Post-hoc repair (parsing with fallback, asking the model to fix its own output) adds latency and is unreliable. The only robust solution: prevent invalid token sequences from being sampled in the first place.

The Key Insight: Local Validity Is Enough

You don't need to forbid bad sequences globally. You only need to forbid the single next token that, given what has been emitted so far, would make the sequence irrecoverable.

This is exactly what an automaton state transition tells you.

From JSON Schema to Context-Free Grammar

When you send a function tool definition, the provider's infrastructure converts the JSON schema into a context-free grammar (CFG). The conversion is mechanical:

JSON Schema	Grammar Rule
`"type": "string"`	Matches any valid JSON string
`"type": "number"`	Matches JSON number literals
`"type": "object"` with `"properties"`	Matches object with exactly those keys and value types
`"enum": [...]`	Alternation listing only allowed literals
`"required": [...]`	Restricts which property combinations are accepted
Nested objects/arrays	Recursive grammar rules

For a get_weather function:

{
  "type": "object",
  "properties": {
    "city": { "type": "string" },
    "unit": { "enum": ["celsius", "fahrenheit"] }
  },
  "required": ["city"]
}

The resulting grammar (Lark-like notation):

start: "{" ws pair_city ws maybe_unit "}"
pair_city: "\"city\"" ws ":" ws STRING
maybe_unit: ("," ws pair_unit)?
pair_unit: "\"unit\"" ws ":" ws UNIT_ENUM
UNIT_ENUM: "\"celsius\"" | "\"fahrenheit\""
STRING: "\"" /[^"\\]*/ "\""
ws: /\s*/

This grammar accepts {"city": "Tokyo"} and {"city": "Tokyo", "unit": "celsius"} but rejects {"city": 42}, {"unit": "kelvin"}, and {"temperature": 22}.

Finite-State Machines vs Pushdown Automata

Regular expressions (finite-state machines) handle flat patterns — phone numbers, UUIDs, ISO 8601 dates. They cannot handle arbitrary nesting.

JSON with arbitrarily nested objects and arrays requires a context-free grammar, which corresponds to a pushdown automaton — a finite automaton plus a stack. The stack tracks nesting depth: every { pushes, every } pops. When the stack is empty and you're in an accepting state, the JSON is complete and valid.

Logit Masking: The Enforcement Mechanism

At each decoding step:

The model computes raw logits ℓ ∈ ℝ^|V| (one per vocabulary token)
The grammar engine inspects tokens generated so far, determines valid next tokens given current automaton state
Invalid tokens get logit = -∞
Softmax, temperature, top-k, top-p proceed over remaining valid tokens
Token is sampled from constrained distribution
Automaton advances with the chosen token

ℓ̃_v = ℓ_v          if valid[q_t][v] = 1
      = -∞           otherwise

After softmax, disallowed tokens have probability zero. The sampler operates on the masked distribution exactly as it would on an unconstrained one. Temperature scaling, top-p, top-k all compose cleanly.

The model's relative preferences over allowed tokens are preserved. If the model thought "Tokyo" was twice as likely as "Osaka" before constraining, it's still twice as likely after. The grammar only removes options that would violate the schema.

Token-Level Masks: Why Character Simulation Matters

Tokenizers operate over multi-character tokens, not single characters. A token like "weather" (single token) or "42" (single token in many tokenizers) consumes multiple characters at once.

The mask must simulate each token's full character span through the automaton, not just check the first character.

Example: Grammar requires exactly 4 digits ([0-9]{4}).

At state q0 (0 digits seen): token "27" (2 chars) is allowed → lands at q2
At state q3 (3 digits seen): token "27" is forbidden → second char would overshoot into dead state

A token's legality flips with the state. This is why production systems (XGrammar, llguidance, Outlines) pre-compute token-level masks by simulating each token's full decoded string through the automaton.

Inside XGrammar: Compiled Grammar + Matcher

XGrammar (used in vLLM, SGLang) implements this at production speed:

Grammar — describes output structure (JSON Schema → CFG)
GrammarCompiler — compiles grammar to CompiledGrammar (pre-computes token masks, builds transition tables)
GrammarMatcher — stateful, per-request object:
- accept_token(token_id) — advances internal state
- fill_next_token_bitmask() — produces 32-bit compressed bitmask for current state

The bitmask is a compressed bitset (int32 array). For 128K vocabulary: ~16 KB per request. The matcher only runs on CPU; the bitmask is copied to GPU where a single kernel applies it to logits before sampling.

Adaptive token mask caching: requests with identical grammar/tokenizer state share masks. The cache key is the stack top node — validity of future tokens is determined by the current parser state.

The Function Calling Pipeline End-to-End

1. Schema Conversion
   JSON Schema → Context-Free Grammar (cached)

2. Prompt Construction
   Tool definitions serialized into model's prompt format
   (Harmony tokens, XML-wrapped definitions, etc.)

3. Free Generation
   Model generates conversational text unconstrained
   Until it emits tool-call trigger token (e.g., `<|tool_call|>`)

4. Constrained Generation
   Grammar engine activates
   At each step: compute valid tokens → mask logits → sample → advance automaton
   Continues until complete valid JSON object produced

5. Parsing
   Output (guaranteed structurally valid) parsed to extract function name + args

6. Execution
   Tool call returned to developer or executed server-side
   Result flows back to model, generation continues

Why This Matters

Structural correctness is a hard guarantee. The model's training only needs to handle the semantic side: deciding which tool to call and what argument values to pass. The scaffolding — braces, commas, quotes, property names, type constraints — is enforced mechanically at every token.

This is why function calls almost never return malformed JSON. Not because the model is that reliable. Because the sampling process won't let it be unreliable.

What Developers Should Know

Prompt the structure anyway. Constrained decoding only affects sampling. You want the model's underlying distribution to already align with the structure. Describe the expected output in the prompt.
First request with a new schema is slower. Grammar compilation and caching happen on first use. Subsequent requests reuse the compiled grammar.
Not all JSON Schema features are enforceable. Recursive schemas, unbounded nesting, and some validation keywords may not be fully enforced by the decoder. Check your provider's supported subset.
API-based services can't do this. OpenAI, Anthropic APIs don't expose logit access. Constrained decoding requires access to the decoding loop (local models, vLLM, SGLang, self-hosted).

The Takeaway

Function calling isn't magic. It's not prompt engineering. It's a pipeline where a JSON schema compiles to a formal grammar, and that grammar constrains the model's sampling so it can only produce valid output.

The model does the hard part: understanding intent, choosing the right tool, selecting the right values. The grammar engine does the mechanical part: guaranteeing every brace, comma, quote, and type constraint is correct.

Next time you see a perfect tool call, remember: the model didn't "learn JSON." The decoder was simply forbidden from getting it wrong.

DuckDB: What Actually Happens Under the Hood — Vectorized Execution, Push-Based Pipelines, and Morsel-Driven Parallelism

Syed Anzar — Wed, 28 Oct 2026 13:00:14 +0000

DuckDB: What Actually Happens Under the Hood — Vectorized Execution, Push-Based Pipelines, and Morsel-Driven Parallelism

You know DuckDB as "SQLite for analytics." You've probably run SELECT * FROM 'data.parquet' and watched it scan millions of rows in milliseconds. But do you know why it's that fast?

Most developers treat DuckDB as a black box: embed it, query it, get results. But the speed comes from three deliberate architectural choices that fundamentally differ from how PostgreSQL, SQLite, or even Spark execute queries.

Let's open the box.

The Three Pillars

Traditional DBs (PostgreSQL, SQLite)	DuckDB
Row-at-a-time (Volcano iterator)	Vectorized: 2,048 rows per batch
Pull-based: parent asks child for data	Push-based: child pushes to parent
Centralized executor decides parallelism	Morsel-driven: each operator self-parallelizes

These aren't optimizations. They're a different execution model.

Vectorized Execution: Why 2,048 Rows?

DuckDB doesn't process rows. It processes vectors — contiguous arrays of 2,048 values per column.

// Traditional row-at-a-time (PostgreSQL, SQLite)
for (row in table) {
    value = row.column;
    result = fn(value);
    output.append(result);
}
// 1 function call per row, terrible cache usage, no SIMD

// DuckDB vectorized
for (vector in table) {
    values = vector.column;        // 2048 values in contiguous memory
    results = fn_simd(values);     // 1 function call, SIMD-friendly
    output.append(results);
}

Why 2,048?

Fits in L1/L2 cache (~16 KB for int64)
Power of 2 -> efficient bit operations
Large enough to amortize function call overhead
Small enough to avoid memory pressure

Each vector lives inside a DataChunk — the fundamental unit of data flow:

DataChunk:
  Column 0: [int1, int2, ..., int2048]   // 2048 integers
  Column 1: [str1, str2, ..., str2048]   // 2048 strings
  Column 2: [ts1,  ts2,  ..., ts2048]    // 2048 timestamps
  Count: 2048  (valid rows <= capacity)

The count can be less than 2048 — that's how filters work. A WHERE clause doesn't remove rows; it just reduces the count field. The underlying memory stays allocated.

Push-Based Execution: Data Flows Downhill

Traditional databases use the Volcano iterator model (pull-based):

// Parent pulls from child
while (chunk = child.GetNext()) {
    process(chunk);
    parent_push(chunk);  // if parent exists
}

Problem: the parent controls the pace. The child can't push multiple chunks ahead. You get pipeline bubbles — the parent waits for the child, the child waits for I/O, nobody overlaps work.

DuckDB flips it. Children push to parents:

// Child pushes to parent
void Child::Execute(DataChunk &input) {
    process_data();
    parent.Execute(output_chunk);  // push, don't wait
}

What this enables

True pipelining — A chunk flows through Filter -> Project -> Aggregate in one pass. No materialization between operators.
Cache locality — Data stays hot in L1/L2 as it moves through the pipeline. The same 2048-value array gets transformed in place.
Natural parallelism — Multiple pipelines run independently on different cores. No central scheduler bottleneck.
Backpressure for free — If a parent's buffer fills, the child's Execute blocks. I/O pauses propagate naturally.

Morsel-Driven Parallelism: No Central Scheduler

Most parallel databases have a central executor that hands out work. DuckDB doesn't.

Instead, each operator decides its own parallelism based on the data it sees.

idx_t PhysicalOperator::EstimatedThreadCount() const {
    if (children.empty()) {
        // Table scan: parallelism = row groups / 2
        return max(cardinality / (ROW_GROUP_SIZE * 2), 1);
    }
    if (type == UNION) {
        // Sum children's threads
        return sum(child.EstimatedThreadCount());
    }
    // Other operators: max of children
    return max(child.EstimatedThreadCount());
}

A table with 1M rows (8 row groups of ~122K each) on 4 cores:

Thread 1: row groups 0, 1
Thread 2: row groups 2, 3
Thread 3: row groups 4, 5
Thread 4: row groups 6, 7
-> All partial results combined

No synchronization during processing. Threads only sync at pipeline boundaries (HashJoin build, HashAggregate, Order).

Pipeline breakers

Some operators must materialize and break the pipeline:

Operator	Why it breaks
HashJoin (build side)	Must build complete hash table before probing
HashAggregate	Must see all rows before finalizing groups
Order (Sort)	Must collect all rows before sorting

DuckDB organizes operators into pipelines separated by these breakers.

Query: SELECT country, COUNT(*) FROM users WHERE age > 18 GROUP BY country ORDER BY count DESC LIMIT 10

Pipeline 1: TableScan(users) -> Filter(age>18) -> HashAggregate(GROUP BY country)
  -> Parallel: 4 threads scan different row groups, each builds partial hash table

Pipeline 2: HashAggregate (combine) -> Order(count DESC)
  -> Single-threaded: merge partial aggregates, sort

Pipeline 3: Order -> Limit(10)
  -> Early termination after 10 rows

The Optimizer: Stratified and Subquery-Aware

Before execution, DuckDB runs a stratified optimizer — multiple passes, each with a specific goal:

Logical rewrites — predicate pushdown, projection pushdown, join reordering, subquery unnesting
Physical planning — choose operators, partition joins, decide parallelism
Adaptive execution — runtime statistics feed back into filter ordering, join strategy

Predicate pushdown (logical)

-- Original
SELECT name FROM (SELECT * FROM users) WHERE age > 30;

-- Optimized: filter pushed into table scan
TableScan(users, FILTER age > 30) -> Project(name)

DuckDB reads only the age and name columns. The other 48 columns never leave disk.

Projection pushdown

-- Only 'name' and 'age' columns read from storage
SELECT name FROM users WHERE age > 30;

Columnar storage + projection pushdown = massive I/O reduction.

Join ordering

-- Small table (countries) as build side, large table (users) as probe side
SELECT * FROM users
JOIN countries ON users.country_id = countries.id;

DuckDB estimates cardinalities and puts the smaller table on the build side automatically.

Adaptive Execution: Runtime Feedback

DuckDB doesn't just trust the optimizer. It adapts during execution.

Adaptive filtering

WHERE expensive_function(x) > 10 AND cheap_column = 'value'

The executor tracks selectivity. If cheap_column = 'value' eliminates 99% of rows, it evaluates that first next time — even if the optimizer ordered it differently.

Adaptive hash joins

Data size	Strategy
Small	In-memory hash table
Medium	Partitioned hash table (radix)
Large	Disk-based partitioning with spillage

The same operator switches strategies mid-query based on actual row counts.

What This Means in Practice

Query Parquet files directly — no ETL

import duckdb

# Zero-copy: DuckDB reads Parquet column chunks directly
# No import, no copy, no schema inference step
result = duckdb.query("""
    SELECT station_id, AVG(temperature)
    FROM 's3://weather-data/2024/*.parquet'
    WHERE year = 2024
    GROUP BY station_id
""").df()

The httpfs extension streams row groups. Predicate pushdown skips entire row groups via Parquet statistics.

Larger-than-memory analytics

DuckDB 0.9+ (Sep 2023) added out-of-core support for all operators:

Hash aggregate: spills to disk when groups exceed memory
Hash join: partitions build/probe sides to disk
Sort: external merge sort
Window functions: segment tree with disk spill

-- 50 GB data, 16 GB RAM — works
SELECT customer_id, COUNT(*)
FROM 's3://huge-dataset/*.parquet'
GROUP BY customer_id;

When NOT to Use DuckDB

Use case	Better choice
High-concurrency OLTP (many writers)	PostgreSQL, MySQL
Mobile/embedded with tiny footprint	SQLite
Real-time replication, read replicas	PostgreSQL + streaming replication
Complex transactions, foreign keys	PostgreSQL
Vector search (semantic similarity)	pgvector, sqlite-vec, or dedicated vector DB

DuckDB is single-writer, multiple-reader. It's not a general-purpose RDBMS replacement.

Mental Model Summary

Concept	What it means for you
Vector (2048 rows)	CPU processes batches, not rows. SIMD, cache-friendly.
DataChunk	Columns travel together. Filter = reduce count, not delete memory.
Push-based	Data flows downhill. Pipelines fuse operators. No iterator overhead.
Morsel	~122K rows. Unit of parallelism. Threads steal morsels dynamically.
Pipeline breaker	HashJoin build, HashAggregate, Sort. Synchronization points.
Adaptive	Runtime stats override static plans. Filter reordering, join spill.

Peek Under the Hood Yourself

-- See the physical plan with parallelism
EXPLAIN ANALYZE
SELECT country, COUNT(*)
FROM users
WHERE age > 18
GROUP BY country;

-- Enable detailed profiling
PRAGMA enable_profiling;
PRAGMA profiling_output = 'profile.json';
SELECT ...;
-- Open profile.json: operator timings, memory, thread counts, actual vs estimated rows

The profile shows exactly which pipelines ran in parallel, where data spilled, and whether the optimizer's estimates matched reality.

The Takeaway

DuckDB isn't "SQLite for analytics" because of its API. It's "SQLite for analytics" because it threw away the Volcano iterator model and built an execution engine from first principles for analytical workloads:

Columnar storage -> read only what you need
Vectorized (2048) -> SIMD, cache, amortized overhead
Push-based -> fused pipelines, no materialization
Morsel-driven -> self-parallelizing operators, no central bottleneck
Adaptive -> runtime reality beats static estimates

Next time you run a query and get results in milliseconds on a 10 GB Parquet file, you'll know exactly which thread processed which morsel, which pipeline it flowed through, and why it didn't need a server.

That's not magic. That's mechanical sympathy.

Test Article

Syed Anzar — Wed, 28 Oct 2026 12:56:41 +0000

Test

This is a test article.

How RAG Actually Works: Chunking, Embeddings, Retrieval, and Reranking Under the Hood

Syed Anzar — Wed, 28 Oct 2026 07:14:50 +0000

How RAG Actually Works: Chunking, Embeddings, Retrieval, and Reranking Under the Hood

RAG feels simple on paper. Split documents into chunks, embed them, fetch the closest ones at query time. Done.

That description hides half a dozen hard sub-problems. Split too coarsely and you retrieve paragraphs that bury the relevant sentence in noise. Split too finely and no single chunk carries enough context to be useful. Use only dense retrieval and you miss documents where an exact product code or rare proper noun is the critical signal. Use only keyword search and you miss paraphrases. Retrieve top-k blindly and the final context window fills with redundant or marginally relevant chunks.

Most tutorials stop at "embed, store, retrieve." This article goes deeper — what actually happens at each stage, where the trade-offs live, and what production systems do differently.

The Pipeline at a Glance

INGESTION (offline, whenever documents change)
sources → parse/clean → deduplicate → chunk → embed (dense) + index terms (BM25) → store vectors, text, metadata

QUERY (online, per request)
question → (optional) rewrite/expand → hybrid retrieval (BM25 + dense, top 50–200) → rerank with cross-encoder (keep top 5–10) → select for relevance/diversity within token budget → build prompt with numbered sources → generate with citation instructions → check citations → return answer with links

Two pipelines meet at the index. The offline path runs when documents change. The online path runs per request. Keeping them aligned is where most RAG systems break.

1. Chunking: The Hardest Part Nobody Talks About

Every embedding model has a maximum token length — 512 tokens for older bi-encoders, 4,096–8,192 for modern ones (text-embedding-3-large, nomic-embed-text-v1.5). But even if your model supports 8k tokens, shoving an entire chapter into a single vector is a mistake: the embedding averages the semantics of every sentence, so a query about one narrow fact scores poorly against a dense multi-topic chunk that happens to contain that fact.

Chunking controls a fundamental quality–recall trade-off:

Too large chunks — high recall (the fact is somewhere in there), but low precision (the LLM gets noise and may hallucinate or ignore the signal)
Too small chunks — high precision, but context loss: "It increased by 12%" is meaningless without its referent

Getting chunking wrong is probably the single most common cause of poor RAG performance in production.

Fixed-Length Chunking with Overlap

The simplest approach: split every chunk_size tokens, with an overlap of tokens shared between adjacent chunks so sentence boundaries don't cut off context.

def fixed_chunk(text: str, chunk_size: int = 512, overlap: int = 64, tokenizer=None) -> Iterator[str]:
    if tokenizer is None:
        tokens = text.split()
        decode = lambda ids: " ".join(ids)
    else:
        tokens = tokenizer.encode(text)
        decode = tokenizer.decode

    step = chunk_size - overlap
    if step <= 0:
        raise ValueError("chunk_size must be strictly greater than overlap")

    start = 0
    while start < len(tokens):
        end = min(start + chunk_size, len(tokens))
        yield decode(tokens[start:end])
        if end == len(tokens):
            break
        start += step

Typical production values: chunk_size=256–512 tokens with overlap=32–64. The overlap parameter is crucial — without it, a sentence split across two chunks loses its grammatical antecedent in both halves.

Semantic Chunking

Fixed-length chunking is agnostic to content structure. Semantic chunking exploits embedding similarity to find natural topic breaks:

Split text into sentences
Embed each sentence
Compute cosine similarity between consecutive sentence embeddings
Mark a chunk boundary wherever similarity drops below a threshold (or is a local minimum)

def semantic_chunk(sentences: list[str], embed_fn, threshold: float = 0.75, min_sentences: int = 3) -> list[str]:
    embeddings = embed_fn(sentences)  # shape (N, D)
    sims = [cosine_similarity(embeddings[i], embeddings[i+1]) for i in range(len(sentences)-1)]

    chunks = []
    current = []
    for i, sent in enumerate(sentences):
        current.append(sent)
        if i < len(sims):
            is_boundary = sims[i] < threshold and len(current) >= min_sentences
            if is_boundary:
                chunks.append(" ".join(current))
                current = []
    if current:
        chunks.append(" ".join(current))
    return chunks

Semantic chunking produces more coherent chunks but is slower (requires an embedding call) and more sensitive to the threshold. A common tuning strategy: run the chunker on a held-out set, visualize the similarity curve, and set the threshold at the 15th percentile of observed similarities.

Structure-Aware Chunking

Real documents have structure: headings, paragraphs, bullet lists, code blocks. Respecting that structure almost always outperforms purely statistical methods.

# The production equivalent: LangChain's MarkdownHeaderTextSplitter
from langchain_text_splitters import MarkdownHeaderTextSplitter

md_splitter = MarkdownHeaderTextSplitter(
    headers_to_split_on=[("#", "h1"), ("##", "h2"), ("###", "h3")]
)
chunks = md_splitter.split_text(markdown_document)
# Each chunk carries heading metadata for downstream filtering

The libraries you actually ship. Nobody writes chunkers from scratch twice — you write them once to understand the trade-off, then use maintained splitters. The two dominant ones are LangChain (langchain_text_splitters) and LlamaIndex (llama_index.core.node_parser), and both express exactly the strategies above.

Late Chunking (The Context-Preserving Trick)

Late chunking, introduced by Günther et al. (2024) and popularized with jina-embeddings-v2, inverts the usual order:

Pass the entire document through the transformer encoder to produce contextualized token embeddings
Mean-pool over the token spans corresponding to each logical chunk

The attention mechanism lets every token see its full document context before the pooling boundary is applied. "It increased by 12%" gets an embedding that encodes what "it" refers to, because self-attention already saw the preceding paragraph.

# late_chunking.py — illustrative
from transformers import AutoTokenizer, AutoModel
import torch

def late_chunk_document(document: str, chunk_boundaries: list[tuple[int, int]], model_name: str = "jinaai/jina-embeddings-v2-base-en") -> list[LateChunk]:
    tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
    model = AutoModel.from_pretrained(model_name, trust_remote_code=True).eval()

    encoding = tokenizer(document, return_tensors="pt", return_offsets_mapping=True, truncation=True, max_length=8192)
    offset_mapping = encoding.pop("offset_mapping")[0]
    input_ids = encoding["input_ids"]
    attention_mask = encoding["attention_mask"]

    with torch.no_grad():
        token_embeddings = model(input_ids=input_ids, attention_mask=attention_mask).last_hidden_state[0]

    chunks = []
    for start_char, end_char in chunk_boundaries:
        token_mask = (offset_mapping[:, 0] >= start_char) & (offset_mapping[:, 1] <= end_char) & (offset_mapping[:, 1] > offset_mapping[:, 0])
        span_embeddings = token_embeddings[token_mask]
        if span_embeddings.shape[0] == 0:
            continue
        chunk_embedding = span_embeddings.mean(dim=0)
        chunk_embedding = chunk_embedding / (chunk_embedding.norm() + 1e-9)
        chunks.append(LateChunk(text=document[start_char:end_char], embedding=chunk_embedding, start_char=start_char, end_char=end_char))
    return chunks

Constraint: the full document must fit within the model's context window (~8k tokens). For longer documents, apply late chunking within sliding windows.

Late chunking has a generative cousin: contextual retrieval (Anthropic, 2024) asks an LLM to write a one-sentence situating preamble for each chunk and prepends it before embedding. Late chunking is cheap (one encoder pass, no LLM) but needs a long-context encoder; contextual retrieval works with any encoder but costs one LLM call per chunk at index time. They compose.

2. Hybrid Search: BM25 + Dense Retrieval

Dense retrieval excels at semantic similarity but struggles with exact-match recall. If a user asks about "CVE-2023-44487" or a specific product SKU, the query embedding may not land near the document containing that exact string — because the model generalizes. BM25 handles these cases natively.

BM25 score for query q against document d:

BM25(q,d) = Σ IDF(t) * (f(t,d) * (k1+1)) / (f(t,d) + k1 * (1 - b + b * |d|/avgdl))

where f(t,d) is term frequency, |d| is document length, avgdl is average document length, k1≈1.2–2.0, b≈0.75.

Use a real BM25 engine. A naive Python implementation rescores the entire corpus on every query — O(N⋅|q|) — fine for teaching, hopeless beyond a few thousand chunks. Real lexical retrieval uses an inverted index: for each query term, walk only the posting list of documents that actually contain it.

Open-source ladder:

bm25s — sparse-matrix BM25 in Python; right default for single-process index up to millions of short chunks
Pyserini — Python bindings over Lucene; research-grade path
Elasticsearch / OpenSearch — distributed inverted index when corpus outgrows one machine; both ship native hybrid retrieval and built-in RRF
Most vector databases now do the sparse leg too: Qdrant, Weaviate, Milvus all accept sparse vectors alongside dense and expose server-side RRF fusion

Reciprocal Rank Fusion (RRF)

Given a BM25 ranking and a dense ranking, how to combine them? Cormack et al. (2009) introduced RRF:

RRF(d) = Σ 1 / (k + rank_r(d))

where k = 60 (smoothing constant). Documents not in a ranking are omitted from that term. Highest combined RRF score wins.

RRF does not require score normalization — it only uses rank ordinals, making it immune to the scale mismatch between BM25 scores (0–20) and cosine similarities (0.5–1.0).

def reciprocal_rank_fusion(rankings: list[list[tuple[int, float]]], k: int = 60) -> list[tuple[int, float]]:
    rrf_scores = defaultdict(float)
    for ranked_list in rankings:
        for rank, (doc_idx, _score) in enumerate(ranked_list, start=1):
            rrf_scores[doc_idx] += 1.0 / (k + rank)
    return sorted(rrf_scores.items(), key=lambda x: -x[1])

Worked example: BM25 ranks [D2, D1, D3, D4, D5], dense ranks [D1, D3, D2, D5, D4]. With k=60:

D1: 1/62 + 1/61 = 0.0325 → rank 1 (wins despite BM25 rank 2)
D2: 1/61 + 1/63 = 0.0323 → rank 2
D3: 1/63 + 1/62 = 0.0320 → rank 3

D1 wins in fused ranking because its dense rank of 1 contributes a large term. The k=60 constant prevents any single top-1 ranking from dominating completely.

Learned Sparse Retrieval (SPLADE)

BM25 and dense retrieval fail in opposite directions. Learned sparse retrieval keeps the representation sparse (a vector over the tokenizer's vocabulary, lives in an inverted index) but learns the weights with a transformer instead of counting term frequencies.

SPLADE (Formal et al., SIGIR 2021) runs text through a masked-language-model head, producing a logit for every vocabulary term at every position. Then saturate and max-pool over positions:

w_j = max_i log(1 + ReLU(w_ij))

The log(1+·) is a saturation function borrowed from BM25's term-frequency damping. Sparsity is trained in with a FLOPS regularizer — typical passage keeps ~100 non-zero terms.

Payoff: term expansion. A passage about "myocardial infarction" gets non-zero weight on heart and attack even though those strings never appear, so a lexical index can now match a paraphrase.

# sentence-transformers v5+ added SparseEncoder
from sentence_transformers import SparseEncoder

model = SparseEncoder("naver/splade-v3")
doc_emb = model.encode_document(corpus)       # sparse tensors, |V|-dimensional
query_emb = model.encode_query("http2 rapid reset attack")
scores = model.similarity(query_emb, doc_emb) # sparse dot product

# Inspect *why* a document scored
print(model.decode(doc_emb[0], top_k=8))  # e.g. [('http', 2.1), ('reset', 1.9), ...]

SPLADE is a first-stage retriever — drop-in replacement for (or third leg alongside) BM25. Fuse its ranking with dense ranking using RRF, then rerank with cross-encoder. Costs are real: encoding requires a transformer forward pass at both index and query time, and expansion terms lengthen posting lists. The honest 2026 summary: BM25 + strong dense encoder remains the pragmatic default; learned sparse is the upgrade when exact-match recall matters and queries are paraphrase-heavy, or when you need interpretability without giving up semantic matching.

3. Cross-Encoder Rerankers: The Precision Stage

The retrieval stage (BM25 or dense) must be fast — processing millions of documents in tens of milliseconds. Speed requires a bi-encoder architecture: query and document embedded independently, similarity = cheap dot product. Downside: query and document tokens never "see" each other during encoding, so nuanced relevance signals (multi-hop, contrastive relevance) can be missed.

A cross-encoder receives the concatenation [query; document] as a single sequence and produces a scalar relevance score. Every query token attends to every document token, enabling fine-grained relevance reasoning at the cost of O(N) forward passes (one per candidate). This makes cross-encoders unsuitable for first-stage retrieval but ideal for reranking a small shortlist of 20–100 candidates.

There's a well-defined middle: late interaction (ColBERT) keeps one vector per token rather than one per document, precomputes all document token vectors offline like a bi-encoder, and scores with a MaxSim operator recovering much of the cross-encoder's token-level matching. Costs ~10–100x more index storage in exchange for cross-encoder-like quality at first-stage speed.

Two-Stage Pipeline

Query → Hybrid Retrieval (BM25 + dense) → top-100 candidates
          fast, cheap recall
      → Cross-Encoder Reranker → scores all 100 [query; passage] pairs
          slower, expensive, precise scoring
      → top-5 chunks → LLM Context Window

Only the 5 winning chunks reach the LLM context window, keeping prompt costs low while the cross-encoder's joint query-passage attention ensures relevance quality.

from sentence_transformers import CrossEncoder
import numpy as np

def load_reranker(model_name: str = "cross-encoder/ms-marco-MiniLM-L-6-v2") -> CrossEncoder:
    return CrossEncoder(model_name, max_length=512)

def rerank(query: str, candidates: list[str], model: CrossEncoder, top_k: int = 5, batch_size: int = 32) -> list[tuple[str, float]]:
    pairs = [(query, passage) for passage in candidates]
    scores = model.predict(pairs, batch_size=batch_size)
    ranked = sorted(zip(candidates, scores), key=lambda x: -x[1])
    return ranked[:top_k]

# Latency numbers (A100):
# cross-encoder/ms-marco-MiniLM-L-6-v2, 100 candidates, avg 200 tokens: ~60ms
# On CPU (Intel Xeon): ~400ms for same 100 candidates

Practitioner tip: Two APIs, one job. Classical BERT-style rerankers (ms-marco-*, bge-reranker-*, jina-reranker-*) use CrossEncoder. The 2025–2026 generation of LLM-initialized rerankers — Qwen3-Reranker, mxbai-rerank-v2 — read relevance from a causal LM as the logit of a "yes"/"no" token given a prompt with instruction, query, and document. They buy prompt conditioning ("rank by whether the passage contains a numeric answer") and long context, and cost a full decoder forward pass per pair. Load with AutoModelForCausalLM per model card, or use a wrapper: rerankers (AnswerDotAI) gives one Reranker(...).rank(query, docs) call over cross-encoders, ColBERT, LLM rerankers, and hosted APIs alike. In production, serve the reranker off the request path — HuggingFace Text Embeddings Inference hosts reranker models behind a /rerank endpoint with continuous batching.

Training Your Own Reranker

If you have domain-specific relevance labels (click logs, human annotations, distillation from a more powerful model), fine-tune a cross-encoder with binary cross-entropy on (query, positive, negative) triples, or listwise ranking loss (LambdaLoss, ListNet).

from datasets import Dataset
from sentence_transformers.cross_encoder import CrossEncoder, CrossEncoderTrainer, CrossEncoderTrainingArguments
from sentence_transformers.cross_encoder.losses import BinaryCrossEntropyLoss

train_dataset = Dataset.from_dict({
    "query": ["What is RRF?", "What is RRF?"],
    "passage": [
        "Reciprocal rank fusion combines ranked lists using 1/(k+rank).",
        "The capital of France is Paris.",
    ],
    "label": [1.0, 0.0],  # 1 = relevant, 0 = hard negative
})

model = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2", num_labels=1)
loss = BinaryCrossEntropyLoss(model)

args = CrossEncoderTrainingArguments(
    output_dir="./my-domain-reranker",
    num_train_epochs=1,  # 1 epoch usually enough for domain adaptation
    per_device_train_batch_size=16,
    learning_rate=2e-5,
    warmup_ratio=0.1,
    fp16=True,
)

trainer = CrossEncoderTrainer(model=model, args=args, train_dataset=train_dataset, loss=loss)
trainer.train()
model.save_pretrained("./my-domain-reranker/final")

The data matters more than the loss. Pointwise BCE on random negatives teaches almost nothing — random negatives are trivially separable. Mine hard negatives: run your existing first-stage retriever, take documents ranked 10–100 that are not labelled relevant, and use those as zeros — they are exactly the confusions the reranker exists to fix. Usual ratio: 4–8 hard negatives per positive. With graded relevance or teacher scores, prefer listwise loss (LambdaLoss, ListNet) over BCE: it optimizes the ordering of a candidate list, which is what nDCG actually measures. Distilling a large reranker's scores into a MiniLM-sized student with MarginMSELoss is the standard way to get most quality at a fraction of latency.

4. Query Rewriting and HyDE

The query the user types is rarely optimal for retrieval — short, colloquial, or implicit.

Multi-Query Expansion

Generate several paraphrases of the original query, retrieve for each, and union the result sets (deduplicating by document ID). Dramatically improves recall for queries that can be stated multiple ways.

REWRITE_PROMPT = """You are a retrieval expert. Given a user question, produce {n} alternative phrasings that together cover the semantic space of the question. Return a JSON list of strings.

User question: {question}

Alternative phrasings (JSON list):"""

def expand_query(question: str, llm_call: callable, n_variants: int = 3) -> list[str]:
    prompt = REWRITE_PROMPT.format(question=question, n=n_variants)
    response = llm_call(prompt)
    # extract JSON list from LLM output...
    return [question] + variants[:n_variants]  # include original

HyDE — Hypothetical Document Embeddings

Instead of embedding the (short, vague) query, ask the LLM to generate a hypothetical document that would answer the query, then embed that document and retrieve using its embedding.

The intuition: a hypothetical answer document and a real answer document inhabit closer regions of embedding space than the query and the answer document, because they share vocabulary, entity mentions, and syntactic structure.

HYDE_PROMPT = """Write a short, factual paragraph (2-4 sentences) that directly answers the following question. Do not say "I don't know" — write the best answer you can, even if uncertain.

Question: {question}

Answer:"""

def hyde_embed(question: str, llm_call: callable, embed_fn: callable, n_hypotheses: int = 1) -> np.ndarray:
    hypotheses = []
    prompt = HYDE_PROMPT.format(question=question)
    for _ in range(n_hypotheses):
        hypotheses.append(llm_call(prompt).strip())
    embeddings = embed_fn(hypotheses)
    mean_embedding = embeddings.mean(axis=0)
    return mean_embedding / (np.linalg.norm(mean_embedding) + 1e-9)

HyDE hallucination risk: The LLM's hypothetical document will contain plausible-sounding but possibly incorrect facts. This is fine for retrieval (you only use the embedding, not the text), but be careful not to accidentally include the hypothesis in the LLM context. Always retrieve from the real corpus using the HyDE embedding, then pass the retrieved real documents to the generator.

Use HyDE when: corpus uses formal/technical language differing from query style; queries are short and ambiguous; you can afford 1–2 extra LLM calls per query (~200ms latency budget). Avoid when: queries are already long and specific; LLM hallucinations could mislead retrieval; low-latency SLA (<50ms) prevents extra LLM calls.

5. Metadata Filtering and Parent-Child Retrieval

Purely semantic retrieval treats all documents as equally eligible. In practice, queries have hard constraints: "only Q4 2024", "only 'legal' category", "only product model XYZ-500". Metadata filtering applies these constraints before or during ANN search, drastically reducing the candidate pool and improving precision.

Pre-filtering vs Post-filtering

Pre-filter (filter before ANN): restrict index to documents matching metadata predicate, then search that subset. Exact, but gives up prebuilt ANN graph over surviving set.
Post-filter (retrieve top-k, then filter): fast, but may need to retrieve large k to guarantee filtered results cover top-k meaningful hits.

Modern vector databases (Qdrant, Pinecone, Weaviate, Milvus) support hybrid pre/post-filtering with HNSW graph filterable attributes.

# Qdrant example
from qdrant_client import QdrantClient
from qdrant_client.models import Filter, FieldCondition, MatchValue, Range
import numpy as np

def filtered_search(client, collection_name, query_vector, filters: dict, top_k=10):
    conditions = []
    if "category" in filters:
        conditions.append(FieldCondition(key="category", match=MatchValue(value=filters["category"])))
    if "date_after" in filters or "date_before" in filters:
        conditions.append(FieldCondition(key="timestamp", range=Range(gte=filters.get("date_after"), lte=filters.get("date_before"))))

    qdrant_filter = Filter(must=conditions) if conditions else None
    response = client.query_points(
        collection_name=collection_name,
        query=query_vector.tolist(),
        query_filter=qdrant_filter,
        limit=top_k,
        with_payload=True,
    )
    return [hit.payload for hit in response.points]

Parent-Child Document Retrieval

Index fine-grained child chunks (single sentences or 128-token windows) for high-precision retrieval, but when a child chunk is retrieved, return its parent document (full paragraph or section) as LLM context. Gives the LLM surrounding context while keeping the retrieval signal sharp.

@dataclass
class ChildChunk:
    id: str
    parent_id: str
    text: str
    embedding: np.ndarray

@dataclass
class ParentDocument:
    id: str
    text: str
    metadata: dict

def parent_child_retrieve(query_embedding, child_index, parent_store: dict, top_k=5) -> list[ParentDocument]:
    child_hits = child_index.search(query_embedding, top_k=top_k * 3)  # over-fetch for unique parents
    seen_parent_ids = set()
    parents = []
    for child in child_hits:
        pid = child.parent_id
        if pid not in seen_parent_ids:
            seen_parent_ids.add(pid)
            if pid in parent_store:
                parents.append(parent_store[pid])
        if len(parents) >= top_k:
            break
    return parents

Both frameworks ship this: LangChain's ParentDocumentRetriever, LlamaIndex's SentenceWindowNodeParser (retrieve a sentence, return window of ±k sentences) and AutoMergingRetriever (retrieve leaf chunks over hierarchy, substitute parent when enough siblings hit — the behavior worth stealing if you build your own).

One thing this doesn't fix: redundancy. Five distinct chunks from five near-identical documents will still all survive into the prompt. Standard remedy: Maximal Marginal Relevance (MMR) — greedily trade query relevance against novelty relative to already-selected chunks — applied after reranking and before prompt assembly.

6. A Production RAG Pipeline

@dataclass
class RAGConfig:
    chunk_size: int = 512
    chunk_overlap: int = 64
    chunking_strategy: str = "fixed"  # "fixed" | "semantic" | "structural"
    bm25_top_k: int = 40
    dense_top_k: int = 40
    rrf_k: int = 60
    use_hyde: bool = False
    n_query_variants: int = 1
    use_reranker: bool = True
    reranker_model: str = "cross-encoder/ms-marco-MiniLM-L-6-v2"
    rerank_candidates: int = 50
    reranker_top_k: int = 5
    metadata_filter: dict = field(default_factory=dict)

def run_rag_pipeline(query: str, bm25_index, dense_index, embed_fn, llm_fn, reranker, config: RAGConfig) -> RAGResult:
    # Step 1: Query representation
    if config.use_hyde:
        query_embedding = hyde_embed(query, llm_fn, embed_fn)
    else:
        query_embedding = embed_fn([query])[0]

    # Step 2: Hybrid retrieval
    bm25_results = bm25_index.search(query, top_k=config.bm25_top_k)
    dense_results = dense_index.search(query_embedding, top_k=config.dense_top_k)
    fused = reciprocal_rank_fusion([bm25_results, dense_results], k=config.rrf_k)
    fused_chunks = [(all_docs[idx].text, score) for idx, score in fused]
    candidate_texts = [t for t, _ in fused_chunks]
    rrf_by_text = dict(fused_chunks)

    # Step 3: Reranking
    if config.use_reranker and reranker is not None:
        reranked = rerank(query, candidate_texts[:config.rerank_candidates], model=reranker, top_k=config.reranker_top_k)
        final_chunks = [t for t, _ in reranked]
        reranker_scores = [float(s) for _, s in reranked]
    else:
        final_chunks = candidate_texts[:config.reranker_top_k]
        reranker_scores = []

    return RAGResult(query=query, retrieved_chunks=final_chunks, rrf_scores=[rrf_by_text[t] for t in final_chunks], reranker_scores=reranker_scores)

Choosing Configuration for Your Domain

Domain Characteristic	Recommended Setting
Precise entity lookup (medical codes, SKUs, legal citations)	BM25 weight high; hybrid with RRF; add metadata filter on document type
Conversational QA over prose	Dense retrieval dominant; enable HyDE; semantic chunking
Long documents with clear section structure	Structural chunking + parent-child retrieval
Multilingual corpus	Multilingual encoder + reranker (e.g., `BAAI/bge-m3` + `BAAI/bge-reranker-v2-m3`); BM25 needs per-language tokenization
Paraphrase-heavy queries over technical text	Add SPLADE as third ranker in RRF fusion
Low-latency SLA (<100ms)	Skip HyDE; use MiniLM reranker or skip reranking; pre-filter metadata
High-stakes accuracy requirement	Enable all stages: HyDE + hybrid + full reranker; re-evaluate every 30 days

7. Diagnostics and Iteration

A RAG pipeline has many knobs. Easy to tune one component in isolation while unknowingly degrading another. Rigorous approach: evaluate each stage separately with stage-specific metrics, then measure end-to-end quality.

from typing import NamedTuple

class RetrievalMetrics(NamedTuple):
    recall_at_k: float
    mrr: float
    ndcg_at_k: float
    latency_p50_ms: float
    latency_p99_ms: float

def compute_recall_at_k(relevant_ids: list[set[str]], retrieved_ids: list[list[str]], k: int) -> float:
    recalls = [len(rel & set(ret[:k])) / len(rel) for rel, ret in zip(relevant_ids, retrieved_ids) if rel]
    return sum(recalls) / max(len(recalls), 1)

def compute_mrr(relevant_ids: list[set[str]], retrieved_ids: list[list[str]]) -> float:
    rrs = []
    for rel, ret in zip(relevant_ids, retrieved_ids):
        for rank, doc_id in enumerate(ret, start=1):
            if doc_id in rel:
                rrs.append(1.0 / rank)
                break
        else:
            rrs.append(0.0)
    return sum(rrs) / len(rrs)

Key metrics per stage:

Retrieval: Recall@k, MRR, nDCG@k, latency p50/p99
Reranking: nDCG@k on reranked list, latency per candidate
Generation: Faithfulness, Answer Correctness, Context Relevancy (RAGAs framework)
End-to-end: Token-level F1, EM score on QA benchmarks

Always evaluate on your domain. The table in Section 6.1 gives starting points, not guarantees.

Summary: What Actually Matters

Chunking is the foundation. Get it wrong and nothing downstream fixes it. Start with fixed-size recursive splitting (LangChain's RecursiveCharacterTextSplitter) with chunk_size=512, overlap=64. Upgrade to structure-aware or semantic chunking when you have evidence it helps.
Hybrid retrieval is the default. Pure dense misses exact matches. Pure BM25 misses paraphrases. RRF fuses them without score normalization headaches. Add SPLADE as a third leg only when paraphrase-heavy queries over technical text justify the cost.
Reranking is where precision lives. Bi-encoders cast a wide net; cross-encoders score with joint attention. The two-stage funnel (100 → 5) is the cost-conscious architecture. Load the reranker once at process startup — constructing it per query costs hundreds of milliseconds.
Query rewriting is cheap recall. Multi-query expansion and HyDE cost 1–2 LLM calls but often recover documents the raw query misses. HyDE turns query→document matching into document→document matching.
Metadata filtering is free precision. Hard constraints (date, category, product) applied at the vector DB level eliminate irrelevant candidates before they consume retrieval or reranking budget.
Parent-child retrieval solves the context gap. Retrieve small chunks for sharp signals, return large contexts for generation. Deduplicate parents, then apply MMR for diversity.
Measure each stage independently. Recall@k for retrieval, nDCG@k for reranking, faithfulness/correctness for generation. Optimizing one metric while ignoring others is how you get a system that retrieves perfectly but hallucinates in the answer.

DEV Community: Syed Anzar

Plane: The Open-Source Jira/Linear Alternative — What Actually Happens When You Self-Host a 61k-Star Project Management Platform

Plane: The Open-Source Jira/Linear Alternative — What Actually Happens When You Self-Host a 61k-Star Project Management Platform

The Architecture: 10 Services, One Platform

What Happens When You Create a Work Item

Real-Time Collaboration: The live Service

Background Jobs: Celery + RabbitMQ Done Right

Self-Hosting: Data Sovereignty Without the DevOps Nightmare

The Tech Stack: Battle-Tested, Not Trendy

AI & Extensibility: Not an Afterthought

Migration: You're Not Trapped

What You're Signing Up For

Why This Matters for Developers

TL;DR

How HTTP/3 and QUIC Actually Work: UDP, Streams, and Why 0-RTT Changes Everything

How HTTP/3 and QUIC Actually Work: UDP, Streams, and Why 0-RTT Changes Everything

The Short Version: What Changed

The Wire Format: Packets, Frames, and Encryption Levels

Four Encryption Levels = Four Packet Types

Frames: The Actual Payload

Coalescing ≠ Multiplexing

Connection IDs: The Connection That Outlives Its Address

The Handshake: 1-RTT by Default, 0-RTT on Resume

TCP + TLS 1.3: 2 RTT Minimum

QUIC: 1 RTT (New Session)

0-RTT: Send Data Before the Server Replies

Streams: Independent, Ordered, Flow-Controlled

No Head-of-Line Blocking

HTTP/3 Mapping: Requests = Bidirectional Streams

Stream Types in HTTP/3

Connection Migration: Surviving Network Changes

0-RTT in Practice: The Replay Problem

Congestion Control: The Hidden Performance Variable

Real-World Performance: What the Data Shows

What This Means for You

If You're a Backend Developer

If You're a Frontend Developer

If You're Running Infrastructure

The Mental Model to Keep

Further Reading

Trigger.dev: The Open-Source Durable Execution Platform That Replaces Cron, BullMQ, and Inngest for Long-Running TypeScript Jobs

Trigger.dev: The Open-Source Durable Execution Platform That Replaces Cron, BullMQ, and Inngest for Long-Running TypeScript Jobs

What Developers Get Wrong About Background Jobs

How Trigger.dev Actually Works: Checkpoint-Resume, Not Event Sourcing

1. Task Runs in an Isolated Container

2. Wait = Checkpoint + Resource Release

3. Checkpoint = CRIU Snapshot → OCI Image → Registry

4. Warm Starts: Retries That Skip Completed Work

The Mental Model: One Concept - Tasks

Triggering from Anywhere

Realtime: Stream Progress to Your UI

Comparison: Where Trigger.dev Wins (and Where It Doesn't)

Self-Hosting: No Lock-In, Same Codebase

Build Extensions: FFmpeg, Playwright, Python - No Dockerfile

Real-World Patterns

AI Agent with Human-in-the-Loop

Scheduled Jobs That Survive Deploys

Multi-Tenant Queue Isolation

Honest Gaps (as of v3)

When to Adopt Trigger.dev

The Bottom Line

HyperSwitch: The Open-Source Payments Orchestrator That Replaces Your PSP Stack

HyperSwitch: The Open-Source Payments Orchestrator That Replaces Your PSP Stack

What Actually Is HyperSwitch?

The Architecture That Makes It Work

Router + Scheduler: Two Services, One Job

The Payment State Machine: Intent vs Attempt

Euclid: Routing That Isn't Just Round-Robin

GSM: Normalizing 120+ Error Languages

The Vault: PCI Compliance Without the Pain

Reconciliation: Double-Entry for Payments

What It Replaces

Self-Hosting: What You Actually Need

The Trade-offs

When to Adopt

Start Here

What Actually Happens When You Resolve a Domain Name

What Actually Happens When You Resolve a Domain Name

The Big Picture

Step 0: The Cache Check

Real-Time Collaboration: The `live` Service

Query to Root (`.`)

Query to TLD (`.com`)

Query to Authoritative (`example.com`)