DEV Community: Mubashir Taj

How YouTube Handles a Billion Hours of Video

Mubashir Taj — Wed, 11 Nov 2026 13:14:10 +0000

Every day, people watch more than a billion hours of video on YouTube. That number is hard to picture, so here is a smaller one: more than 500 hours of video are uploaded every minute. The system has to accept that flood, convert every upload into a dozen playable formats, store it forever, and deliver it instantly to a phone on a slow connection in one country and a television on fiber in another, all while deciding which video to recommend next. The interesting part is not any single piece of that. It is how the pieces fit together so that no one part becomes a bottleneck that takes the rest down with it.

Start with the upload, because everything downstream depends on a decision made in the first seconds. When a creator uploads a video, YouTube does not store the file as it arrived and serve it directly. The uploaded file might be a 4K master of several gigabytes, recorded with settings no browser can play efficiently. The system first stores that original in a durable object store, as the source of truth, and then hands it to a transcoding pipeline. Transcoding converts the original into many versions: different resolutions from 144p to 4K, different bitrates, and different codecs. A phone on a weak connection gets a small 360p stream. A television gets 4K. The viewer never sees this. They press play and the player picks the version that fits their connection right now, and switches versions mid playback if the connection changes.

Transcoding at YouTube's scale cannot be a single queue with a single pool of workers. The pipeline is a distributed job system. An upload creates a job, the job is split into chunks, and those chunks are processed in parallel across many machines. A short video might transcode in seconds. A two hour upload in 4K takes longer, and the system shows the creator a progress state rather than blocking on completion. The important design choice is that upload, transcode, and publish are separate stages with a queue between each pair. If transcoding slows down, uploads keep working, and videos already transcoded keep publishing. A failure in one stage does not cascade into the others.

Storage is the next problem, and it is bigger than it looks. YouTube stores exabytes of video, and the copies multiply fast: one original, plus a dozen transcoded versions, plus thumbnails, plus the metadata that describes them all. No single database and no single filesystem holds that. The storage layer is a distributed object store built for this shape: huge files, written once, read millions of times, rarely deleted. Files are split into chunks, and each chunk is replicated across machines in different racks and different data centers. A disk failure, a rack failure, even a data center failure does not lose a video, because another copy is elsewhere. Hot videos, the ones millions of people are watching right now, are also copied into caching layers closer to viewers, so the origin storage is not hit for every play.

Delivery is where the system earns its reputation, and the mechanism is a content delivery network of unusual scale. YouTube runs its own global edge network, with caching servers in thousands of locations. When you press play, the video does not stream from a central server. It streams from an edge cache near you, and if that cache does not have the segment yet, it fetches it from a parent cache, which fetches it from origin. Video is delivered in small segments, typically a few seconds each, over HTTP. That segmentation is what makes adaptive bitrate work. The player downloads one segment at a time, measures how fast it arrived, and picks the quality of the next segment accordingly. A dip in your connection means the next segment is lower quality, not a frozen video. The player also buffers a few segments ahead, so a brief stall in the network is invisible.

Metadata is a separate system from video storage, and that separation is deliberate. The title, description, tags, view count, comments, and the channel it belongs to live in databases optimized for reads and writes of small records, not in the object store with the video bytes. This matters because the read patterns are completely different. Video bytes are read sequentially, in large chunks, by a small number of viewers per file at a time. Metadata is read constantly, by everyone, for search, recommendations, and the watch page. Mixing the two workloads in one store would make both slower.

The view count is a good example of how the system trades exactness for scale. A video with a hundred million views cannot update a single counter row on every play. That would be a write hotspot, one row that every viewer in the world tries to update at once. Instead, views are counted in shards and aggregated. The number you see is close to exact, and exactness within a small margin is fine for a view count. The system applies the same thinking to likes and comments: aggregate, approximate where the product allows it, and reserve strong consistency for the things that genuinely need it, like whether a video is public, private, or deleted.

Search and recommendations deserve their own mention, because they are the reason people stay on the site. Search is a separate indexing pipeline. When a video is published, its title, description, tags, and captions are indexed, along with signals like watch time and engagement. A search query hits that index, not the storage layer, and returns ranked results in milliseconds. Recommendations work differently. They are built from models that predict what a viewer is likely to watch next, based on watch history, what similar viewers watched, and the video's own performance with audiences like this one. The models are trained offline on enormous datasets and served from a separate system that can answer a ranking request in tens of milliseconds. The watch page you see is the output of that ranking, rendered from metadata, not a query against the video store.

The database story behind all of this is worth a closer look, because YouTube is one of the best known users of a pattern called sharding with a twist. The system uses a globally distributed database that can scale horizontally across data centers while still offering strong consistency where it matters. A video's metadata lives on a shard determined by its id, and a read for that video goes to the shard that owns it. That keeps any single database from becoming a bottleneck. The lesson for smaller systems is not to copy the database. It is to notice the pattern: identify the unit of sharding early, keep related data together on one shard where you can, and never design a feature that requires a cross shard transaction on the hot path.

Live streaming adds a different set of pressures. A live video cannot be transcoded in advance, because it does not exist yet. The ingest server receives the stream, transcodes it in near real time into several quality levels, and pushes the segments to the edge as they are produced. Latency becomes a product decision. The default mode buffers a few seconds for stability. A low latency mode trades some of that buffer for a delay closer to two seconds, which matters for live chat and interaction. The tradeoff is explicit, and the system lets the creator choose, because there is no single right answer.

There is a temptation to read all of this and conclude that YouTube's architecture is out of reach, and in its full form it is. But the patterns inside it are ordinary engineering, applied with discipline. Separate the hot path from the cold path. Put a queue between stages so one slow stage does not block the rest. Store the original and derive the rest. Deliver from the edge, in small segments, and let the client adapt. Shard by a stable key, and aggregate counters instead of updating one row. Keep metadata and bulk data in different stores, because they have different access patterns. These are decisions any team can make at any scale, and they are what make a system survive growth instead of collapsing under it.

If you are building something that handles media, start with the smallest version of this shape. Accept an upload, store the original, and transcode it asynchronously with a job queue. Serve the result through a CDN, even a small one, and let the player adapt to the connection. Put the metadata in a database and the bytes in object storage, and keep them separate from the start. That separation is the decision that is hardest to retrofit later, and it is the one YouTube's design gets right first. The billion hours are impressive. The architecture that survives them is the lesson.

Monolith vs Microservices: When Splitting Helps and When It Hurts

Mubashir Taj — Tue, 10 Nov 2026 13:15:06 +0000

Few architecture debates generate as much confident bad advice as monolith versus microservices. Ask a room of developers and you will hear two certainties that contradict each other: that a monolith is technical debt waiting to happen, and that microservices are needless complexity. Both claims are wrong in the same way. They treat the question as a matter of taste, when it is actually a matter of fit. The right architecture depends on the team, the domain, and the stage of the product. The same split that saves one organization will sink another.

Start with what each shape actually is, because the words get used loosely. A monolith is a single deployable unit that contains the whole application: the web layer, the business logic, and usually the data access, all in one codebase and one process. A microservices architecture splits the application into many small services, each independently deployable, each owning its own data, communicating over the network. The difference that matters most is not size. It is the boundary between the pieces. In a monolith, a call between two modules is a function call: fast, type checked, and easy to trace. In microservices, that same call crosses a network, which means latency, serialization, partial failure, and versioning. Everything else follows from that one fact.

A monolith's strengths are real and often underrated. Development is simple. A new developer clones one repository, runs one command, and has the whole system on their machine. Refactoring is cheap, because the compiler or the test suite sees the whole codebase at once. Renaming a field touches every caller in one commit. Transactions are straightforward, because one database can guarantee that two related writes either both happen or neither does. Debugging is easier too: a stack trace shows the full path through the code, and there is no network hop to suspect. Deployment is one artifact, one version, one rollback. For a small team building a product whose shape is still changing, that simplicity is worth a great deal.

The monolith's weaknesses are also real, and they appear on a schedule. As the codebase grows, the build gets slower, the test suite gets slower, and the deploy becomes scarier, because every change ships with every other change. Teams start stepping on each other: two features in the same release, one of them buggy, block the other from going out. Scaling gets blunt. If one endpoint needs ten times the capacity and the rest need none, you scale the whole application anyway, because it is one unit. And the technology choice is one choice for everything. The language and framework picked on day one are the language and framework for every feature that follows.

Microservices fix those specific problems, and it is worth naming them precisely rather than praising the pattern in general. Independent deployment is the big one. A team can ship its service without coordinating a release train with five other teams. Independent scaling follows: the hot service gets more instances while the quiet ones stay small. Fault isolation is real too. If the recommendation service crashes, the checkout service keeps working, provided the two are genuinely independent and checkout does not block on recommendations. And technology can vary by service, so the team that needs a fast data pipeline can use one stack while the team that needs a simple CRUD API uses another.

But every one of those benefits has a price, and the price is paid in distributed systems problems. The network is not reliable, so every call between services needs a timeout, a retry policy, and a plan for what happens when the other service is down. Data consistency gets harder in a specific way. There is no single database transaction across two services, so a business operation that used to be one atomic write becomes a saga: a sequence of steps with compensating actions when a later step fails. That is not a small change. It is a different way of thinking about correctness, and it leaks into every feature that touches more than one service.

Observability gets harder too. A request that used to appear in one log file now crosses five services, each with its own logs, and reconstructing what happened requires a correlation id passed through every hop and a tracing system that collects it. Testing gets harder in the same direction. A unit test is still easy, but an integration test now needs several services running, with realistic data, in the right versions. Teams that split too early discover they have traded one hard problem, a big codebase, for several harder problems, distributed transactions, network failure modes, and cross service debugging, before they had any of the benefits that justify the trade.

So when does splitting actually help? There are a few honest signals. The first is team size and independence. When several teams work on the same codebase and their releases keep colliding, splitting along team boundaries can help, because each team gets its own deploy pipeline and its own release cadence. The second is genuinely different scaling needs. A video processing pipeline and a user profile API have nothing in common in their load profile, and scaling them together wastes money. The third is fault isolation that the business actually needs. If one feature going down must not take the rest of the system with it, and the product genuinely depends on that guarantee, the isolation is worth the cost. The fourth is regulatory or security boundaries, such as payment data that must live in a separate system with separate access controls.

And when does splitting hurt? Almost always when the product is young. A startup with three developers and an unproven product does not have a scaling problem. It has a learning problem, and microservices slow down learning, because every change to a boundary becomes a negotiation between services. Splitting hurts when the domain boundaries are not yet clear. If you cannot draw the lines between your services with confidence, you will draw them wrong, and a wrong service boundary is far more expensive than a wrong module boundary, because moving code between services means changing APIs, data ownership, and deployment pipelines. It hurts when the team lacks operational maturity. If you do not have good logging, tracing, alerting, and deployment automation, microservices will multiply the operational surface faster than the team can manage it.

The practical middle path, and the one most teams should take, is the modular monolith. Keep one deployable application, but organize it into strict modules with clear boundaries, each module owning its own data access and exposing a narrow interface to the others. No module reaches into another module's tables. No module imports another module's internals. The boundaries are enforced by code review and by structure, not by the network. Then, if one module later needs to become a service, the extraction is a mechanical job: the interface already exists, the data ownership is already clear, and the hard design work is already done. If it never needs to become a service, you have lost nothing.

The Go example below shows the shape. Two modules, orders and inventory, live in the same binary. They communicate through interfaces defined in a shared package, and neither one touches the other's storage. The day inventory needs to become a service, the implementation of the Inventory interface is replaced with an HTTP client, and the orders module does not change.

package domain

type Order struct {
    ID     string
    ItemID string
    Qty    int
}

type Inventory interface {
    Reserve(itemID string, qty int) error
    Release(itemID string, qty int) error
}

type Orders struct {
    inv Inventory
}

func NewOrders(inv Inventory) *Orders {
    return &Orders{inv: inv}
}

func (o *Orders) Place(itemID string, qty int) (*Order, error) {
    if err := o.inv.Reserve(itemID, qty); err != nil {
        return nil, err
    }
    return &Order{ID: "ord_1", ItemID: itemID, Qty: qty}, nil
}

package inventory

import "errors"

var errOutOfStock = errors.New("out of stock")

type Store struct {
    stock map[string]int
}

func NewStore() *Store {
    return &Store{stock: map[string]int{"widget": 100}}
}

func (s *Store) Reserve(itemID string, qty int) error {
    if s.stock[itemID] < qty {
        return errOutOfStock
    }
    s.stock[itemID] -= qty
    return nil
}

Notice what this buys you. The orders module never sees the stock map. The only contract between the modules is the interface. Tests for orders use a fake inventory, and tests for inventory use no orders code at all. That discipline is most of what microservices promise, without any of the network.

There is also a decision framework worth applying before any split. Ask four questions, and answer them with evidence, not enthusiasm. First, which teams are actually blocked by shared releases, and how often? If the answer is rarely, there is no deployment problem to solve. Second, which parts of the system have genuinely different scaling or availability needs? If the answer is none yet, there is no scaling problem to solve. Third, can you draw the service boundaries from the domain, with each service owning data that no other service needs to write? If the answer is no, the boundaries will be wrong. Fourth, does the team have the operational tooling to run many services: tracing, centralized logging, automated deploys, and alerts that a tired person can act on at 3 in the morning? If the answer is no, fix that first, because splitting without it is how you build a system nobody can debug.

A final warning about the most common failure mode. Teams split a monolith because it feels slow and messy, but the mess was never about the deployment shape. It was about unclear module boundaries, shared mutable state, and code that any part of the system could touch. Splitting that codebase into services does not fix the mess. It distributes it across a network, where it becomes harder to see and harder to fix. The order of work matters: clean up the boundaries inside the monolith first. If the monolith is still slow after that, the split will be easy, because the boundaries are already drawn. If it is fast enough after that, you have saved yourself a distributed system you did not need.

The question was never monolith or microservices. It was: what problem are you solving, and is a network boundary the cheapest way to solve it? Answer that honestly, and the architecture mostly picks itself.

How Cloudflare Absorbs a DDoS Attack

Mubashir Taj — Mon, 09 Nov 2026 13:14:11 +0000

A distributed denial of service attack is not a clever exploit. It is a flood. The attacker does not need to find a vulnerability in your code. They need to send more traffic than your infrastructure can handle, and then your real users cannot get in. The interesting engineering question is not how to stop the flood at its source, because you cannot. The question is how to build a system that stands in the flood and keeps serving legitimate requests anyway. Cloudflare is the most studied answer to that question, and its design is worth understanding even if you never run a network of its size.

The first idea is anycast, and it is the foundation everything else rests on. In a normal setup, your domain resolves to an IP address that lives on one server in one data center. Every request, legitimate or malicious, goes to the same place. Cloudflare announces the same IP address from more than 300 cities at once. When a user in Karachi requests your site, BGP routing sends them to the nearest Cloudflare data center. When a botnet in another region fires a flood, that traffic is also spread across the nearest data centers to the bots sending it. The attack is not concentrated on one target. It is diluted across a global network before any filtering even begins.

This matters because a DDoS attack is usually a volume problem before it is a filtering problem. A single data center with a 100 gigabit link dies when a 200 gigabit flood arrives, no matter how good its filters are, because the pipe itself is full. A network with hundreds of data centers, each with its own capacity, can take the same flood in pieces. No single location sees enough of it to saturate. The architecture turns one impossible problem into hundreds of manageable ones.

The second idea is that filtering happens at the edge, as close to the attacker as possible. Cloudflare's network runs its own software stack on every server, and that stack inspects traffic in layers. At layer 3 and 4, the network layer, the system deals with the bluntest attacks: SYN floods, UDP floods, and ICMP floods. These attacks try to exhaust connection tables and bandwidth with raw packets. The mitigation here is mostly statistical and structural. A SYN flood is answered with SYN cookies, a technique that lets the server validate a connection attempt without allocating memory for it first. UDP floods aimed at random ports are dropped at the edge because there is no service listening there. The goal at this layer is to discard the obviously invalid traffic with as little work as possible, so the expensive analysis is reserved for traffic that looks almost legitimate.

Layer 7 is where the attack gets interesting, and where the filtering gets smarter. A layer 7 attack is an HTTP flood: millions of requests that look like normal GET requests, hitting expensive endpoints, perhaps with rotating user agents and residential IP addresses. You cannot drop these by IP address alone, because the addresses look real and keep changing. Cloudflare handles this with a combination of rate limiting, behavioral analysis, and challenges.

Rate limiting is the simplest tool and still one of the most effective. A data center tracks how many requests each client makes in a window, and when a client exceeds a threshold, further requests are blocked or slowed. The threshold is set per endpoint, because a login endpoint and a static image have very different legitimate rates. A single global limit would either block real users or let the attack through. The configuration below is the kind of rule a Cloudflare customer writes for a sensitive endpoint, and the same shape works in any reverse proxy.

{
  "description": "Limit login attempts per IP",
  "match": {
    "request": {
      "url": { "path": { "contains": "/login" } }
    }
  },
  "action": "block",
  "ratelimit": {
    "characteristics": ["ip"],
    "period": 60,
    "requests_per_period": 10,
    "mitigation_timeout": 300
  }
}

Behavioral analysis adds a second layer. The system builds a picture of what normal traffic looks like for a particular site: which countries it comes from, which user agents are common, how fast a human can click through pages. Traffic that deviates sharply from that picture gets a higher risk score. A sudden burst of requests from a region that never visits your site, all using the same unusual user agent, is suspicious even if no single request breaks a rule. The score decides the response: let it through, slow it down, or present a challenge.

The challenge is the third tool, and it is deliberately annoying to bots and cheap for humans. A managed challenge might ask the browser to run a small JavaScript proof of work, or to prove it can execute code the way a real browser does. A simple script making raw HTTP requests cannot pass. A headless browser can, but it is far more expensive to run at scale, and that cost changes the attacker's economics. This is the deeper principle behind all of it: you do not need to make an attack impossible. You need to make it expensive enough that it stops being worth doing.

A fourth idea is capacity itself, and Cloudflare's approach here is unusual. Most mitigation services charge by the volume of attack traffic they absorb, which means a big attack becomes a big bill for the victim. Cloudflare offers unmetered mitigation, absorbing attacks of any size without charging extra. That is only possible because the network is so large that even a record breaking attack is a small fraction of total capacity. The business model and the architecture reinforce each other: the bigger the network, the cheaper each attack is to absorb, and the cheaper absorption is, the more customers join, and the bigger the network gets.

There is also a product called Magic Transit, which extends the same protection to infrastructure that does not sit behind Cloudflare's proxy. A company announces its own IP ranges through Cloudflare's network, traffic to those addresses is routed through Cloudflare data centers, cleaned, and then forwarded to the origin over a tunnel. The origin never sees the attack traffic at all. This is how a service that is not a website, such as a game server or an API running on its own infrastructure, gets the same edge filtering.

All of this raises a fair question for a team that does not run a global network: what can you actually borrow from this design? More than it seems. The principles scale down well.

First, put a filtering layer in front of your origin. That can be a CDN, a load balancer with rate limiting, or a simple reverse proxy. The origin should never be directly reachable from the public internet. If attackers can find your origin IP, they can bypass every layer you built. Keep the origin IP private, accept traffic only from your proxy, and rotate the address if it ever leaks.

Second, rate limit by endpoint and by client, not by server. The nginx configuration below is a workable starting point for a small service. It defines a zone that tracks requests per IP, then applies a limit to a specific location. A client that exceeds ten requests per second gets a 503 response instead of reaching your application.

http {
    limit_req_zone $binary_remote_addr zone=login:10m rate=10r/s;

    server {
        location /login {
            limit_req zone=login burst=5 nodelay;
            proxy_pass http://app_backend;
        }
    }
}

Third, make your expensive endpoints cheap to reject. A login endpoint that runs a password hash on every request is a gift to an attacker, because each request costs you far more than it costs them. Validate the cheap things first: is the request well formed, is the rate acceptable, does the client look like a browser. Only then spend CPU on the expensive check. The same ordering principle applies in code. Here is a token bucket in Go that you can drop in front of a handler, so the limit is enforced before the handler does any real work.

package main

import (
    "net/http"
    "sync"
    "time"
)

type limiter struct {
    mu       sync.Mutex
    tokens   float64
    lastSeen time.Time
}

var buckets sync.Map

func allow(ip string) bool {
    now := time.Now()
    v, _ := buckets.LoadOrStore(ip, &limiter{tokens: 10, lastSeen: now})
    l := v.(*limiter)
    l.mu.Lock()
    defer l.mu.Unlock()
    elapsed := now.Sub(l.lastSeen).Seconds()
    l.tokens += elapsed * 2
    if l.tokens > 10 {
        l.tokens = 10
    }
    l.lastSeen = now
    if l.tokens < 1 {
        return false
    }
    l.tokens--
    return true
}

func rateLimited(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        if !allow(r.RemoteAddr) {
            http.Error(w, "Too many requests", http.StatusTooManyRequests)
            return
        }
        next.ServeHTTP(w, r)
    })
}

Fourth, plan for the attack to succeed partially. No filter is perfect. Have a runbook: how do you raise rate limits, how do you block a region temporarily, who do you call at your provider, and what does your status page say. An attack that finds you without a plan costs you hours of improvisation at the worst possible moment.

The lesson of Cloudflare's design is not that you need 300 data centers. It is that denial of service is a capacity and economics problem, and the defense is to push filtering as far from your origin as possible, make each layer of attack expensive to sustain, and never let the attacker's cheapest move reach your most expensive code. Build the filtering at the edge, rate limit by endpoint, reject cheaply, and keep the origin hidden. The flood will still come. The question is whether it reaches you.

How Discord Handles Millions of Concurrent Voice Users

Mubashir Taj — Sun, 08 Nov 2026 14:33:04 +0000

Voice chat is unforgiving. A web page can load half a second late and nobody notices. A voice call that stalls for half a second is immediately obvious, and a call that drops packets for several seconds is unusable. Discord has to deliver that experience to millions of people at the same time, across thousands of servers, in dozens of regions, on connections ranging from fiber to congested mobile networks. The architecture that makes it work is not one clever trick. It is a set of deliberate tradeoffs: use UDP instead of TCP, forward audio instead of mixing it centrally wherever possible, keep voice state separate from chat state, and push failure handling to the edge where it can be recovered quickly.

To understand the design, it helps to separate three problems that are often confused. First, how does a client find and join a voice channel? Second, how does audio move between participants with low latency? Third, how does the system stay stable when a server fails, a region degrades, or a single channel suddenly grows from ten people to ten thousand listeners? Each problem has its own component, and each component can scale and fail independently.

Discord splits its real time work across two kinds of gateway. The main gateway handles presence, messages, typing indicators, and the general state of a guild. The voice gateway handles one thing only: voice sessions. When you join a voice channel, the client receives a voice server endpoint, a session identifier, and a token. It then opens a separate WebSocket connection to that voice gateway and negotiates a UDP connection for the actual media. This separation matters more than it first appears. Chat traffic is bursty and can tolerate short delays. Voice traffic is continuous and cannot. If both shared the same connection and the same backpressure logic, a spike in message traffic could stall voice signaling. By isolating voice, Discord protects the latency sensitive path from everything else happening in the app.

Once the session is established, audio flows over UDP. TCP guarantees delivery and ordering, which sounds desirable until you think about what it costs in a live call. If a TCP packet is lost, the protocol pauses delivery of everything after it until the missing packet is retransmitted. In a file download that is correct behavior. In a voice call it produces exactly the wrong result: old audio arrives late, new audio is held back, and the conversation stutters. UDP makes no such promise. Packets can arrive out of order or not at all, and the application decides what to do about it. For voice, the right decision is usually to move on. A twenty millisecond slice of audio that arrives late is worse than useless, because the conversation has already moved past it. Dropping it and playing the next slice keeps the call feeling live.

The audio itself is encoded with Opus, a codec designed for interactive speech. Opus adapts its bitrate to the available bandwidth, handles both speech and music reasonably well, and includes packet loss concealment, which means the decoder can synthesize a plausible continuation when a packet goes missing. That last property is easy to underestimate. At low loss rates, good concealment makes a lossy network sound almost clean. Discord also relies on jitter buffers on the receiving side. Packets on the internet do not arrive at perfectly even intervals, so the client holds a small buffer, typically tens of milliseconds, and plays audio out of it at a steady rate. A larger buffer absorbs more jitter but adds latency. A smaller buffer keeps latency low but risks gaps. Tuning that tradeoff, and adapting it as network conditions change, is a large part of what makes a call feel stable.

The forwarding model is the next key decision. In a small call, one naive approach is a full mesh: every participant sends audio directly to every other participant. That works for three or four people and collapses quickly after that, because each client must upload one stream per peer and manage one connection per peer. Discord instead uses voice servers as selective forwarding units. Each participant sends a single upstream stream to the voice server, and the server forwards the relevant streams to each listener. It does not decode and mix all audio into one combined stream, which would cost far more CPU and would remove the client's ability to control individual volumes and mute states. Forwarding keeps the server cheap relative to mixing, and it keeps per user controls intact, since each incoming stream remains separate on the receiving client.

That model also explains why voice servers are regional. Latency is dominated by physical distance and routing, so placing the voice server close to the participants matters directly. Discord assigns a voice region to each guild, and the client connects to a server in that region. When participants are spread across continents, no single region is ideal for everyone, and the practical answer is to pick the region that minimizes the worst case for the active speakers, not to chase a theoretical optimum. Region selection is therefore a configuration choice with real user impact, and large communities change it deliberately when their audience shifts.

Scale brings a different set of problems. A voice server can handle many concurrent sessions, but it cannot handle an unbounded number. Discord shards voice load across a fleet of servers and treats each voice session as movable state with a clear owner. If a voice server process crashes, the sessions on it must be reestablished, and the client must be able to reconnect without the user pressing anything. This is where the separation between signaling and media pays off again. The voice gateway session can be reestablished, a new UDP endpoint can be issued, and the client can resume sending audio, often quickly enough that users perceive a brief glitch rather than a dropped call. Designing for fast reconnection is more valuable than pretending crashes will not happen, because at Discord's scale they happen constantly.

Encryption and authentication are part of the voice path, not an afterthought. Voice packets are encrypted, and the session token issued during join prevents arbitrary clients from injecting audio into a channel. The encryption scheme is designed for real time use: it adds minimal overhead per packet and does not require a handshake per packet. That matters because voice sends roughly fifty packets per second per speaker. Any per packet cost, whether CPU, bytes, or round trips, multiplies immediately. System design for real time media is often a story about per packet costs, and the winning designs are the ones that keep that cost close to zero.

There are lessons here that transfer to much smaller systems. The first is to isolate latency sensitive traffic from everything else. If your application mixes chat, notifications, and voice on one connection, one slow consumer can stall the rest. The second is to choose the transport that matches the semantics you actually need. TCP is the right default for data that must arrive exactly once in order. UDP is the right default for data where timeliness beats completeness. The third is to prefer forwarding over mixing when clients need individual control, and to design reconnection as a normal path rather than an error path. Users forgive a brief glitch. They do not forgive a call that dies and forces them to rejoin manually.

Discord's voice system works at scale because it respects the physics of real time communication. It keeps the media path short, simple, and close to the user. It accepts packet loss instead of fighting it with retransmission. It forwards rather than mixes, isolates voice from chat, and treats reconnection and measurement as core features. Any team building voice, whether for a game, a classroom, or a workplace tool, will face the same constraints at a smaller scale. The answers will differ in detail, but the questions will be the same: how fast must this arrive, what happens when it does not, and how quickly can a broken session come back on its own.

// Voice forwarding is a fanout problem, not a mixing problem.
// Each speaker sends one stream. The server copies it to listeners
// without decoding every stream into a single mixed output.
type VoiceSession struct {
    SpeakerID string
    Listeners map[string]bool
}

func (s *VoiceSession) Forward(packet []byte) {
    for listener := range s.Listeners {
        // In a real server this write is non blocking and
        // drops the packet for a slow listener instead of
        // stalling every other listener.
        sendTo(listener, packet)
    }
}

func sendTo(listener string, packet []byte) {
    // UDP send to the listener endpoint.
    _ = listener
    _ = packet
}

WebSockets vs Polling vs Server-Sent Events: Picking Realtime Transport

Mubashir Taj — Sat, 07 Nov 2026 13:13:54 +0000

Realtime features show up everywhere now: live dashboards, chat, collaborative editing, notifications, stock tickers, delivery tracking. The transport you pick for those features shapes your server architecture, your scaling story, and your debugging experience for years. The three serious options on the web platform are polling, Server-Sent Events, and WebSockets. They are not interchangeable, and the right choice depends on the direction of traffic, the latency you need, and how much infrastructure complexity you are willing to carry.

Start with the shape of the traffic

Before comparing protocols, describe the traffic. Four questions settle most decisions:

Who talks first, and how often? If the server pushes updates on its own schedule, you need a server to client channel. If the client mostly asks and the server answers, plain requests may be enough.
Does the client also send messages, or only receive? A chat app is bidirectional. A price ticker is server to client only.
How fresh must the data be? A notification that arrives a few seconds late is fine. A collaborative cursor that lags by seconds is unusable.
How many concurrent connections will you hold? Ten thousand idle connections and ten thousand connections sending messages every second are very different scaling problems.

With those answers in hand, the comparison becomes concrete rather than a matter of taste.

Polling: simple, and more expensive than it looks

Polling means the client asks the server on a timer: every few seconds it sends a request, and the server answers with the latest state. Short polling uses a fixed interval. Long polling holds the request open until new data arrives or a timeout fires, then the client immediately opens another request.

Polling wins on simplicity. It is ordinary HTTP, so it works through every proxy, load balancer, and firewall without special configuration. It is stateless on the server, which means any instance can answer any request, horizontal scaling is trivial, and a crashed server loses nothing. Debugging is easy because every interaction is a normal request you can log, replay, and inspect in browser developer tools.

The cost is paid in latency and wasted work. With a five second interval, the average update arrives two and a half seconds late, and the worst case is five seconds. Shorten the interval to one second and the latency improves, but now every client sends sixty requests a minute, most of which return "nothing new". At scale that is real load: request parsing, authentication, database checks, and response serialization, multiplied by every connected client, almost all of it thrown away. Long polling removes most of the empty responses, but it holds a connection open per client, which reintroduces the connection management problem, and it still pays the cost of a new request and re-authentication on every event.

Polling is the right choice when updates are infrequent, latency tolerance is measured in seconds, and simplicity matters more than efficiency. A dashboard that refreshes every thirty seconds, a job status page, or an admin panel are all fine with polling. It is the wrong choice for chat or live collaboration, where the request volume and the latency both hurt.

Server-Sent Events: the underused middle ground

Server-Sent Events, or SSE, give you a one way stream from server to client over a plain HTTP connection. The client opens a request, the server keeps it open, and sends events as text lines whenever there is something new. The browser API is small: an EventSource object, a message handler, and automatic reconnection with a last event ID so the server can replay missed events.

SSE keeps the operational advantages of HTTP. It works through proxies and load balancers with far less special configuration than WebSockets, it flows over the same TLS and authentication setup as the rest of your API, and each event is a simple text frame that is easy to log. The browser handles reconnection for you, which removes a whole class of client bugs. On the server, SSE still holds one connection per client, so you need an event loop or async runtime that can hold many idle connections cheaply, but you do not need a separate WebSocket infrastructure.

The limits are real and worth stating plainly. SSE is server to client only. If the client needs to send data, it does so with separate HTTP requests, which is fine for many designs but awkward for true bidirectional interaction. Browsers cap the number of simultaneous SSE connections per origin, historically around six, which matters if a single page opens several streams. Binary data must be encoded as text, so SSE is a poor fit for binary payloads. And some proxies buffer responses, which can delay events unless buffering is disabled for the stream endpoint.

Here is the shape of an SSE endpoint in a Node.js server, to show how little machinery it needs:

app.get("/events", (req, res) => {
  res.writeHead(200, {
    "Content-Type": "text/event-stream",
    "Cache-Control": "no-cache",
    Connection: "keep-alive",
  });
  const timer = setInterval(() => {
    res.write(`data: ${JSON.stringify({ time: Date.now() })}\n\n`);
  }, 1000);
  req.on("close", () => clearInterval(timer));
});

SSE is the right choice for server to client streams where the client rarely talks back: live feeds, notifications, progress updates for a long running job, sports scores, price tickers. If you are reaching for WebSockets for a feed that never takes client input, SSE is almost certainly simpler and sufficient.

WebSockets: full duplex, full responsibility

WebSockets start as an HTTP request that upgrades to a persistent, bidirectional, message based connection. After the upgrade, either side can send messages at any time, with low overhead per message and no HTTP request and response cycle per event. That is the lowest latency option of the three, and the only one designed for genuinely bidirectional traffic.

That power comes with operational responsibility. A WebSocket connection is stateful: the server holding it must deliver messages to that client. Scaling to multiple instances therefore needs a routing mechanism, usually a publish and subscribe backbone such as Redis, so an event arriving at instance A can reach a client connected to instance B. Rolling deploys must drain connections gracefully, and both sides need ping and pong handling to detect dead connections, plus client reconnection logic with backoff, because unlike SSE the browser will not reconnect for you.

The client side is straightforward in modern browsers:

const socket = new WebSocket("wss://example.com/chat");
socket.addEventListener("open", () => {
  socket.send(JSON.stringify({ type: "join", room: "general" }));
});
socket.addEventListener("message", (event) => {
  const payload = JSON.parse(event.data);
  renderMessage(payload);
});

WebSockets are the right choice for chat, multiplayer interaction, collaborative editing, and any feature where both sides send messages frequently and latency matters. They are the wrong choice when traffic is one way, because you pay the stateful infrastructure cost without using the bidirectional capability.

A decision guide you can actually use

The choice can be reduced to a short decision path:

If updates are infrequent and a few seconds of delay is acceptable, use polling. Keep it until the request volume or the latency requirement forces a change.
If the server pushes updates and the client rarely sends anything back, use SSE. It gives you streaming with HTTP operational simplicity.
If both sides send messages, or latency must be well under a second in both directions, use WebSockets, and budget for the connection routing and reconnection work that comes with them.

Two edge cases deserve a note. If your traffic must pass through corporate proxies that interfere with long lived connections, polling may be the only option that works reliably, and reliability beats elegance. If you need ordering guarantees across reconnections, none of these transports gives you that for free: you need application level sequence numbers and a replay mechanism, whether you build it on SSE last event IDs or on your own WebSocket protocol.

Scaling notes that apply to all three

Whatever you choose, three scaling concerns recur. First, authentication: a long lived connection is authenticated once at setup, so token expiry mid connection needs a plan, such as revalidating on a timer or issuing connection scoped credentials. Second, backpressure: a slow client must not let the server accumulate an unbounded queue of unsent events, so you need a policy for dropping, coalescing, or disconnecting slow consumers. Third, observability: count connections, message rates, and reconnect rates per instance, because realtime failures often show up as a slow rise in reconnects long before users report anything.

Realtime transport is not a fashion choice. Polling is simple and stateless, SSE is the efficient one way stream that many teams overlook, and WebSockets are the full duplex tool with a real operational cost. Match the transport to the direction and urgency of your traffic, and you will get a system that is as simple as the feature allows and no simpler.

How Coding Agents Actually Work: Tools, Context, and Loops

Mubashir Taj — Fri, 06 Nov 2026 13:13:41 +0000

A coding agent looks like magic from the outside. You describe a feature, and a few minutes later there is a diff, a test run, and a summary of what changed. Underneath that experience there is no magic, only a loop: a language model reads a task, picks a tool, observes the result, and decides what to do next. Everything interesting about coding agents follows from the details of that loop: which tools are available, what context the model can see, how it plans, and how it checks its own work. Once you understand those pieces, you can predict where agents will succeed, where they will fail, and how to use them well.

The loop is the whole architecture

At its core, an agent is a program wrapped around a language model. The program gives the model a goal, a set of tools, and a transcript of everything that has happened so far. The model produces either a piece of text or a tool call. If it calls a tool, the program runs it and appends the result to the transcript. Then the model is asked again, with the longer transcript. This repeats until the model declares the task done, hits a step limit, or the program stops it for another reason.

That loop explains both the power and the failure modes. The model is not executing code directly and it is not browsing your repository by intuition. It only knows what has been placed in its context or returned by a tool. If a tool result is truncated, the model is working with a partial picture. If the transcript grows too long, older details get summarized or dropped, and the agent can forget a constraint you stated at the start. Every agent design decision is, in some form, a decision about managing that transcript.

A typical loop for a coding task runs through recognisable stages, though the model may revisit them in any order:

Explore the repository: list files, read the relevant modules, and find existing patterns for the kind of change requested.
Form a plan: decide which files change, what the new behaviour should be, and how it will be verified.
Edit code in small steps, running the compiler or tests after each meaningful change.
Observe failures, read the error output, and fix the cause rather than the symptom.
Summarise the change for review, including what was tested and what remains uncertain.

Steps 3 and 4 are where agents earn their keep or waste your time. An agent that edits and never runs anything is just autocomplete with confidence. An agent that runs the test suite and reads the failure output is doing something closer to engineering.

Tools are the agent's hands

A language model on its own can only produce text. Tools are what let it act. The standard toolset for a coding agent includes file reading, file editing, shell command execution, and search across the codebase. More capable setups add tools for running specific test commands, querying a database schema, fetching documentation, or opening a pull request.

Tool design matters more than most people expect. A read tool returning an entire 2,000 line file floods the context, while a search tool returning ranked snippets finds the right function with a fraction of the tokens. An edit tool applying a precise replacement is far safer than one rewriting whole files, which can silently drop behaviour the model never saw. Good harnesses also sandbox shell execution, since a model running arbitrary commands can delete files or leak secrets, so treat an agent's shell access as scoped, logged, and reversible where possible.

Here is a simplified sketch of the loop in pseudocode, to make the structure concrete:

transcript = [system_prompt, user_task]
for step in range(max_steps):
    action = model.generate(transcript)
    if action.is_final_answer():
        break
    result = run_tool(action.tool, action.arguments)
    transcript.append(action)
    transcript.append(result)

Nothing in that sketch is exotic. The sophistication lives in the system prompt, the tool definitions, and the rules for when to stop. A harness might stop after repeated failures on the same test, ask the user a clarifying question, or require the model to explain its plan before editing. Those guardrails are product decisions, and they are what separate a research demo from a tool you trust on a real codebase.

Context is the scarce resource

Every model has a context window, a limit on how much text it can consider at once. That window holds the system prompt, the task, the tool definitions, and the entire transcript of actions and observations. Repository exploration is therefore a budgeting exercise. Reading every file is impossible on a large codebase, so the agent must choose what to read, and that choice determines the quality of the result.

Agents manage context in a few standard ways. They retrieve files by relevance instead of reading whole directories. They summarise long tool outputs, keeping the error lines and dropping the rest. They compact the transcript when it grows too long, replacing early exploration steps with a summary. And they use persistent memory, such as a project notes file, to carry conventions across sessions without re-reading the whole repository each time.

This is why the way you write code affects how well agents work on it. Small focused modules, clear naming, and tests that document expected behaviour are all context efficient: the agent can find the right file quickly and understand it without reading half the repository. A codebase with duplicated logic, vague names, and behaviour hidden in configuration is expensive for an agent to explore, and the agent will make worse decisions because it literally cannot see enough of the system at once. The old advice to write readable code has gained a new justification: readability is now also agent legibility.

Planning, and why agents sometimes wander

Early agents planned very little: they reacted to each tool result and hoped the task converged. Modern agents often plan explicitly, writing a short plan before editing, breaking the task into steps, and revisiting the plan when a step fails. Planning helps on multi file changes, where the order of edits matters and a single wrong assumption can cascade.

Even with planning, agents wander. The common patterns are worth recognising. One is the rabbit hole: the agent finds a plausible but wrong root cause and keeps patching around it instead of stepping back. Another is scope creep: asked to fix a bug, the agent also refactors neighbouring code, renames things, and upgrades a dependency, producing a diff that is hard to review. A third is the confident wrong fix: the agent changes a test expectation to match its broken implementation, so the suite passes while the behaviour is still wrong. That last one is a review problem as much as an agent problem, and it is why diffs produced by agents deserve the same scrutiny as diffs from any other author, perhaps more.

The practical countermeasures are simple. Keep tasks small and independently verifiable. Require the agent to run the relevant tests and report the output. Review the diff, not just the summary, and check whether tests were weakened rather than fixed. And keep the agent away from tasks where a wrong change is expensive and hard to detect, such as security sensitive code or data migrations, unless a human reviews every line.

Verification is what makes an agent trustworthy

The difference between a demo and a dependable tool is verification. A serious coding agent setup treats the model's output as a draft until something independent checks it: the compiler, the type checker, the test suite, a linter, or a second review pass. The strongest setups run tests in a clean environment, not in the agent's own shell session, so a passing test means something.

This also shapes how you should evaluate agent tools. Do not judge them by the best demo output. Judge them by the failure rate on your codebase, the size and reviewability of the diffs they produce, and how often their summaries match what the diff actually does. A tool that produces a correct change half the time but always shows you exactly what it changed is more useful than a tool that is right slightly more often but hides its work.

Using agents well is not about writing longer prompts. It is about decomposition: breaking work into pieces with clear acceptance criteria, giving the agent a fast feedback loop through tests, and reviewing output critically. The agent amplifies existing engineering habits, good and bad alike.

Coding agents are not a replacement for understanding your system. They are a force multiplier on top of that understanding. The loop is simple, the tools are the hands, context is the budget, and verification is the trust mechanism. Keep those four ideas in mind, and the next time an agent surprises you, whether pleasantly or not, you will usually be able to say exactly which part of the loop produced the surprise.

Kubernetes Networking: Services, Ingress, and CNI Explained

Mubashir Taj — Thu, 05 Nov 2026 13:13:48 +0000

Kubernetes networking looks simple from the outside. You apply a Deployment, expose it with a Service, add an Ingress rule, and traffic flows. The confusion starts the first time traffic does not flow, because the system that moves packets is not one component. It is several layers that each solve a different problem: pod to pod communication, stable access to a changing set of pods, external entry into the cluster, and the plugin that wires the whole thing to the host network. Understanding each layer separately makes debugging much faster when something breaks.

Every pod gets an IP, and that changes everything

In Kubernetes, every pod receives its own IP address from a flat network space, and every pod can reach every other pod directly, without network address translation in between. A container does not need to know which node its peer runs on, and two pods can both listen on port 8080 without conflict, because their IPs differ.

The consequence is that pod IPs are the real unit of networking, while everything else is a layer of indirection on top. The problem is that pod IPs are unstable. Pods are created and destroyed by Deployments, autoscalers, and node failures. A client that hardcodes a pod IP will break the first time that pod is replaced. That instability is exactly what Services exist to hide.

Services give you a stable address for an unstable set of pods

A Service is a stable virtual IP, called the ClusterIP, plus a DNS name, backed by a set of pods selected by labels. When a pod dies and a replacement starts with a new IP, the Service updates its endpoint list automatically. Clients keep calling the same address and never notice the churn underneath.

There are four main Service types, and each builds on the previous one:

ClusterIP is the default. It exposes the Service on an internal IP reachable only from inside the cluster. This is the right choice for internal traffic, such as a backend calling a database or one microservice calling another.
NodePort opens the same Service on a fixed port on every node, so traffic to any node IP on that port is forwarded to the Service. It is useful for quick testing, but it is a blunt tool for production, since it exposes a high numbered port on every node and gives you no routing based on hostname or path.
LoadBalancer builds on NodePort and asks the cloud provider to provision an external load balancer that forwards to the NodePorts. This is the standard way to expose a single service to the internet on a cloud provider, but each Service gets its own load balancer, which becomes expensive and hard to manage when you have dozens of public services.
ExternalName is a special case that maps a Service name to an external DNS name instead of to pods. It is handy when code inside the cluster needs to call an outside dependency through the same service discovery mechanism as everything else.

A headless Service, created by setting clusterIP to None, skips the virtual IP and returns pod IPs directly through DNS, which matters for stateful workloads where a client must reach a specific replica.

Here is a minimal Service definition for an API deployment:

apiVersion: v1
kind: Service
metadata:
  name: api
spec:
  selector:
    app: api
  ports:
    - port: 80
      targetPort: 8080
  type: ClusterIP

The selector is the whole mechanism. If the labels on your pods do not match the selector, the Service will have no endpoints and traffic will go nowhere, one of the most common causes of a Service that looks correct but never answers.

kube-proxy is where the virtual IP becomes real

A ClusterIP corresponds to no network interface, and nothing in the kernel knows how to route to it by default. That job belongs to kube-proxy, which runs on every node and programs the packet filtering rules that make Service IPs work.

In iptables mode, kube-proxy writes rules redirecting Service traffic to a backing pod IP chosen at random. IPVS mode uses the kernel IPVS subsystem instead, which scales better and supports more balancing algorithms. Either way, the Service IP is maintained by rules on every node, updated whenever the endpoint list changes.

This explains a failure mode that confuses people. If kube-proxy rules are stale on one node, traffic from pods on that node can fail while traffic from other nodes works. Checking endpoints with kubectl get endpoints separates a selector problem from a proxy problem: if the endpoints are present and correct, the selector is fine and the fault lies further down the stack.

Ingress is the front door, not just another Service

A LoadBalancer Service gives each public service its own external address. Ingress solves the problem differently. You run one Ingress controller, backed by a single LoadBalancer Service, and it routes incoming HTTP and HTTPS traffic to the right internal Service based on hostname and path. One public IP can serve many applications, with TLS termination handled in one place.

An Ingress resource is a set of routing rules, not a running process. The controller watches those rules and configures a reverse proxy, usually NGINX or HAProxy, to implement them. If the resource exists but traffic returns a 404, the controller may not be installed, may watch a different ingress class, or may have failed to reload.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx
  rules:
    - host: app.example.com
      http:
        paths:
          - path: /api
            pathType: Prefix
            backend:
              service:
                name: api
                port:
                  number: 80
          - path: /
            pathType: Prefix
            backend:
              service:
                name: frontend
                port:
                  number: 80

A few practical points follow from this design. First, Ingress only handles HTTP and HTTPS; raw TCP or UDP from outside needs a LoadBalancer Service or a controller extension. Second, path matching has sharp edges: Prefix matching behaves differently from exact matching, and rewrite annotations change the path the backend sees. Third, TLS comes from a Secret referenced by the Ingress resource, and certificate rotation should be automated with a tool like cert-manager rather than done by hand.

CNI is the plugin that builds the pod network

So far we have assumed pods can reach each other. That assumption is implemented by the Container Network Interface, or CNI, plugin chosen when the cluster was built. When the kubelet starts a pod, it calls the CNI plugin to attach the pod to the network: allocate an IP, set up a virtual interface, and configure routing so that the pod IP is reachable from other nodes.

Different CNI plugins make different tradeoffs, and the choice is hard to change later. Flannel is simple and reliable but offers no network policy enforcement. Calico adds rich network policy support, letting you restrict which pods can talk to which. Cilium uses eBPF in the kernel for high performance networking, observability, and policy, and it is increasingly the default on managed platforms.

The practical implication is that two clusters running the same manifests can behave differently. A NetworkPolicy that works on Calico or Cilium does nothing on a cluster whose CNI ignores policy, so verify that your CNI actually enforces policy before relying on it, and test a deny by default posture in staging first.

DNS ties it together

Service discovery is DNS based. CoreDNS answers names like api.default.svc.cluster.local, with short forms like api working within the same namespace. DNS failures look like application bugs, so before blaming application code, check whether resolution works from inside a pod and whether CoreDNS itself is healthy.

A debugging order that saves time

When traffic fails, work from the top of the stack down, and verify each layer before moving on:

Confirm the pods are running and ready, and that the application listens on the expected port inside the container, not just on localhost.
Check the Service endpoints. Empty endpoints mean a selector or label problem, not a network problem.
Test pod to pod connectivity directly by pod IP. If that fails, the CNI or node routing is suspect.
Test the Service ClusterIP from a pod on the same node and from a pod on another node. A difference between the two points at kube-proxy on one node.
Test DNS resolution of the Service name from inside a pod.
For external traffic, check the Ingress controller logs and confirm the controller is running and watching the correct ingress class, then check the Service and endpoints behind it.

Most incidents resolve at one of these steps, and the order matters because each step rules out an entire layer.

Kubernetes networking is several systems cooperating: the CNI builds the pod network, Services provide stable addresses, kube-proxy implements them in the kernel, DNS makes them discoverable, and Ingress provides a single managed front door. Learn each layer on its own terms, and the system becomes something you can reason about and fix.

How Stripe Handles Payments at Scale Without Losing a Cent

Mubashir Taj — Wed, 04 Nov 2026 13:15:58 +0000

Payments are the hardest problem in backend engineering, not because the code is complicated, but because the failure modes are unforgiving. A social feed can drop a post and recover. A payment system that charges a customer twice, loses a successful charge, or records a refund that never happened has created a real financial discrepancy, and that discrepancy will be discovered by a customer, an accountant, or a bank, in that order of unpleasantness. Stripe processes payments at enormous scale across many currencies, payment methods, and regulatory regimes, and it does so without losing track of money. The techniques it relies on are not secret, and they are available to any team building payment features. They are worth studying because they generalize to any system where correctness matters more than speed.

The central idea is idempotency. In ordinary APIs, sending the same request twice usually creates two effects: two rows, two emails, two charges. In a payment API, that is unacceptable, because networks are unreliable and clients will retry. A client that sends a charge request and never receives a response cannot know whether the charge succeeded. The safe behavior is to retry, but retrying must not create a second charge. Stripe solves this with idempotency keys. The client generates a unique key for each logical operation and sends it with the request. The server stores the result of the first execution against that key, and if the same key arrives again, it returns the stored result instead of executing again. The retry becomes safe, which changes the entire failure calculus. Clients can retry aggressively on timeouts, because the server guarantees that a retry of the same operation will not duplicate its effect.

Idempotency keys only work if they are used correctly, and the details matter. The key must be unique per logical operation, not per HTTP attempt, and it must be generated by the client before the first attempt, so that retries carry the same key. A key that is generated fresh on every retry is useless. The server must also store the result durably and atomically with the operation itself, so that a crash between executing the charge and recording the result cannot leave the system in a state where the charge happened but the key record did not. Stripe's API makes this explicit by accepting an idempotency key header on mutating requests, and by scoping keys to an account, which prevents collisions between unrelated operations. The lesson for any payment system is to design idempotency in from the start. Retrofitting it after duplicate charges appear in production is painful, because the duplicates are already in the ledger.

The second idea is the ledger. Stripe does not treat a payment as a single row that gets updated in place. It treats money movement as a sequence of immutable entries that sum to a balance. A charge creates entries, a refund creates offsetting entries, a payout creates entries, and the current state of an account is derived from those entries rather than stored as a single mutable number. This double entry style approach has a powerful property: errors become visible. If entries do not sum correctly, reconciliation catches it. If a row is updated in place and the update is lost, nothing in the data reveals the loss. If an entry is missing from a ledger, the totals do not reconcile, and the discrepancy points directly at the gap. Financial systems have used ledgers for centuries for exactly this reason, and Stripe's architecture follows that tradition in database form.

State machines are the third idea, and they govern how a payment moves through its lifecycle. A payment intent in Stripe moves through states such as requires payment method, requires confirmation, processing, succeeded, and canceled, and only certain transitions are legal. The API enforces those transitions, so a client cannot refund a payment that never succeeded, or capture a payment that was already canceled. That enforcement matters at scale because it turns a large class of bugs into rejected requests instead of corrupt data. Developers building their own payment flows should model the same discipline: define the states, define the allowed transitions, enforce them in one place, and never let two different code paths update payment status with ad hoc rules. When status can only change through defined transitions, auditing becomes possible, because every change has a reason and a previous state.

Underneath all of this is the database problem. Payments require strong consistency for the operations that move money, and that means transactions, carefully scoped. A charge that debits one account and credits another must succeed or fail as a unit. A refund that reduces a balance and creates a refund record must do both or neither. Stripe's systems use transactional stores for these critical paths, and they accept the cost of that consistency, because the alternative, eventual consistency for money movement, creates windows where the system reports a state that is not true. That does not mean every part of a payment platform needs the strongest consistency level. Read models, dashboards, and analytics can be eventually consistent. The ledger itself, and the transitions that change it, cannot.

Retries and webhooks form the next layer. Payment processing involves third parties: banks, card networks, and wallets, each with their own latency and failure modes. A charge may be authorized but not yet captured, a bank may respond slowly, or a webhook delivery may fail. Stripe handles this by treating its own system as the source of truth and treating external events as inputs that must be reconciled, not as facts that can be trusted blindly. Webhook handlers must themselves be idempotent, because webhooks are delivered at least once and may arrive duplicated or out of order. A webhook handler that credits an account on every delivery will double credit on a duplicate delivery. A handler that checks the event identifier against already processed events, and ignores duplicates, will not. That pattern, store the event id, check before applying, apply transactionally, appears everywhere in payment systems, and skipping it is one of the most common causes of real money bugs.

Consider a simplified example. A service receives a webhook for a successful payment, and it must mark an order as paid exactly once, even if the webhook arrives three times. The correct structure is a table of processed event identifiers, checked inside the same transaction that updates the order. If the event was already processed, the handler returns success without changing anything. If it was not, the handler records the event and updates the order together, so a crash cannot leave one without the other.

# Webhook handling must be idempotent: the same event delivered
# twice must not apply its effect twice.
def handle_payment_succeeded(event_id, order_id, db):
    with db.transaction():
        already = db.query(
            "SELECT 1 FROM processed_events WHERE event_id = %s",
            (event_id,),
        )
        if already:
            return "already processed"
        db.execute(
            "INSERT INTO processed_events (event_id) VALUES (%s)",
            (event_id,),
        )
        db.execute(
            "UPDATE orders SET status = 'paid' WHERE id = %s",
            (order_id,),
        )
    return "ok"

Reconciliation is the safety net that catches what idempotency and transactions miss. Stripe reconciles its internal ledger against bank statements, card network reports, and payout records on a regular cycle. Any discrepancy becomes an investigation item, not a silent drift. Smaller teams should adopt a version of the same discipline: a daily job that compares the sum of charges, refunds, and payouts in the application database against the payment provider's reports, and alerts on any mismatch. Reconciliation is not glamorous, and it is tempting to skip it when volumes are small. That is exactly when it is cheapest to build, and exactly when it prevents the small discrepancies that become large ones as volume grows.

None of this requires Stripe's scale to be useful. A small application that sells a single product can implement idempotency keys, a ledger style record of transactions, a payment state machine, idempotent webhook handling, and daily reconciliation, and each of those pieces will prevent a real class of bug. The scale changes the volume, not the principles. Money must be accounted for exactly, retries must be safe, state transitions must be constrained, and external events must be verified before they change internal state. Systems that honor those principles can grow from hundreds of transactions to millions without redesigning their core. Systems that ignore them will eventually face a discrepancy that no amount of manual correction can fully explain, because the data that would explain it was never recorded in the first place. That is what it means, in practice, to handle payments without losing a cent: not that failures never happen, but that every failure leaves a trace, and every trace can be reconciled.

Kafka vs RabbitMQ: Picking the Right Message Broker

Mubashir Taj — Tue, 03 Nov 2026 13:14:31 +0000

Every backend team eventually faces the same question: which message broker should we use? The question sounds simple, but it hides a deeper one, because Kafka and RabbitMQ are not two versions of the same tool. They are built on different models of what messaging is for. RabbitMQ is a message queue in the traditional sense. A producer sends a message, a broker routes it to a queue, a consumer takes it off the queue, and once it is acknowledged the message is gone. Kafka is a distributed commit log. A producer appends records to a topic, those records stay there for a configured retention period, and any number of consumers can read them independently, at their own pace, replaying from earlier offsets whenever they need to. Choosing between them is therefore less about which is better in the abstract and more about which model matches the workload.

Start with RabbitMQ's model, because it is the older and more intuitive one. A producer publishes to an exchange, the exchange routes the message to one or more queues according to bindings and routing keys, and consumers pull from those queues. Different exchange types give different routing behavior. A direct exchange routes by exact key, a topic exchange routes by pattern, a fanout exchange broadcasts to every bound queue, and a headers exchange routes by message attributes. That routing flexibility is RabbitMQ's strength. It is excellent for work distribution, where many workers share a queue and each message should be processed by exactly one worker, and for task queues where a job must be delivered once and then removed. It is also excellent for complex routing rules that would be awkward to express in a log based system.

Delivery semantics in RabbitMQ are built around acknowledgements. A consumer receives a message, processes it, and then acknowledges it. Until that acknowledgement arrives, the broker holds the message and can redeliver it to another consumer if the first one fails. That gives at least once delivery in practice, and with careful design it gives effective once processing when the consumer's work is idempotent. Dead letter exchanges handle the failure case cleanly: a message that is rejected or expires can be routed to a separate queue for inspection instead of being lost or retried forever. For teams building background job systems, notification dispatch, or request reply patterns, this model maps directly onto the problem.

Kafka's model is different from the ground up. A topic is divided into partitions, and each partition is an ordered, append only log. Producers write records to partitions, usually keyed so that records with the same key land in the same partition and therefore keep their order. Consumers belong to consumer groups, and each partition is assigned to exactly one consumer in a group at a time, which gives both parallelism and ordering within a partition. Records are not deleted when consumed. They remain in the log until retention expires, which may be hours, days, or effectively forever with compacted topics. That retention is the feature that changes everything, because it turns the broker from a delivery mechanism into a durable record of events.

The consequences of that difference show up quickly in real systems. In RabbitMQ, if a new consumer comes online, it sees only new messages. Historical messages are gone once acknowledged. In Kafka, a new consumer can start from the beginning of the topic and replay the entire history, which makes it possible to rebuild a service's state, backfill a new feature, or reprocess events after fixing a bug. That replay ability is why Kafka is the default choice for event sourcing, audit trails, change data capture, and stream processing pipelines. It is also why Kafka is often described as a platform for event streaming rather than just a queue. The broker is not only moving messages, it is storing the history that downstream systems derive their state from.

Throughput and latency pull in opposite directions between the two systems, and the difference is architectural rather than incidental. Kafka is designed for high throughput: sequential disk writes, batched sends, zero copy reads, and partition parallelism let a modest cluster move millions of messages per second. Latency is low, but the design optimizes for sustained volume. RabbitMQ is designed for flexible routing and low latency per message, and it handles tens of thousands of messages per second comfortably, but it does not scale to Kafka's throughput on the same hardware, especially when messages are persistent and queues grow long. Long queues in RabbitMQ can also degrade performance, because the broker must manage unacknowledged messages and memory pressure. Kafka, by contrast, is largely indifferent to how far behind a consumer is, because reading old data from a log is a normal operation, not a pathological one.

Ordering guarantees illustrate the contrast clearly. In RabbitMQ, ordering is best effort within a single queue under normal conditions, but it is not a hard guarantee when consumers run in parallel or messages are requeued. In Kafka, ordering is guaranteed within a single partition, which is a strong and useful promise, but it requires discipline: the producer must choose keys so that related events share a partition, and the team must accept that a single partition cannot be processed in parallel by multiple consumers in the same group. That tradeoff, parallelism versus ordering, is fundamental. Systems that need strict ordering for a particular entity, such as all events for one user or one order, map naturally onto Kafka partitions. Systems that need any available worker to take the next job, with no ordering requirement, map naturally onto RabbitMQ queues.

Failure handling also differs. RabbitMQ's acknowledgement and dead letter model makes per message failure handling straightforward. A poison message that crashes every consumer can be isolated, inspected, and retried or discarded deliberately. Kafka's model handles failure at the consumer level: if a consumer fails, its partitions are reassigned to another member of the group, and processing resumes from the last committed offset. That is robust for stream processing, but a poison record can block a partition if the consumer keeps failing on it and never commits past it. The standard answer is a dead letter topic pattern implemented by the consumer, which publishes failed records to a separate topic for later inspection. Both systems can handle poison messages, but RabbitMQ provides more of the machinery out of the box, while Kafka expects the application to build it.

Operational complexity is the other side of the comparison, and it often decides the choice in practice. RabbitMQ is comparatively simple to run: a single node or a small cluster, straightforward clustering and mirrored queues, and a management interface that makes queues, exchanges, and connections visible. Kafka is heavier: ZooKeeper historically, KRaft mode in newer versions, brokers, controllers, partition replication, and careful tuning of retention, replication factor, and consumer group rebalancing. Managed services reduce that burden for both, but the underlying complexity remains, and a team that only needs a job queue should think carefully before taking on a Kafka cluster. Conversely, a team that needs event replay and stream processing will find RabbitMQ's model limiting no matter how well it is operated.

Here is a small illustration of the two producer styles. The RabbitMQ example publishes a task to a queue and moves on. The Kafka example appends an event to a keyed topic, where the key preserves ordering for that entity.

# RabbitMQ style: publish a task, one worker will consume it.
import pika

connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()
channel.queue_declare(queue='image_resize', durable=True)
channel.basic_publish(
    exchange='',
    routing_key='image_resize',
    body='{"image_id": 42, "size": "thumbnail"}',
    properties=pika.BasicProperties(delivery_mode=2),
)
connection.close()

# Kafka style: append an event to a keyed topic.
# All events for user 42 land in the same partition, in order.
from kafka import KafkaProducer
import json

producer = KafkaProducer(
    bootstrap_servers='localhost:9092',
    value_serializer=lambda v: json.dumps(v).encode('utf-8'),
)
producer.send('user_activity', key=b'user-42', value={'event': 'login', 'user_id': 42})
producer.flush()

When you are choosing, ask four questions in order. First, do consumers need to replay history, or only receive new messages? If replay matters, Kafka has a structural advantage. Second, does routing complexity matter more than throughput? Complex routing favors RabbitMQ. Third, what ordering do you actually need, and at what granularity? Per entity ordering favors Kafka partitions, while no ordering favors RabbitMQ work queues. Fourth, what can your team operate reliably? A broker that your team cannot run well is worse than a simpler broker that fits the workload. Answering those four questions honestly will usually point to one system clearly, and when it does not, the honest answer may be that you need both, each for the part of the system it fits.

NestJS Request Lifecycle: Guards, Interceptors, and Pipes in Order

Mubashir Taj — Mon, 02 Nov 2026 13:13:40 +0000

NestJS is often described as an opinionated framework, but its real value is that the opinion is explicit. A request does not wander through a pile of middleware and hope for the best. It passes through a defined sequence of stages, each with a clear job, and each stage can stop the request, transform it, or pass it onward. Developers who learn that sequence stop guessing about where a piece of logic belongs. They also stop writing the same check in three different places, because the lifecycle tells them exactly one place where it fits.

The full order, for a request that reaches a controller, is as follows. First, middleware runs. Second, guards run. Third, interceptors run before the handler. Fourth, pipes run on the handler arguments. Fifth, the handler itself runs. Sixth, interceptors run after the handler, wrapping the result. Seventh, exception filters run if anything threw. That sentence is worth memorizing, because almost every NestJS debugging session comes down to asking which stage you are in and what that stage is allowed to do.

Middleware is the outermost layer, and it is the closest thing NestJS has to plain Express middleware. It runs before routing is fully resolved in some cases, and it is best suited to concerns that apply broadly and do not need NestJS features such as dependency injection into the request context, or at least do not need them in a sophisticated way. Logging, request identifiers, CORS handling, and raw body parsing are natural middleware jobs. Middleware can also short circuit a request by sending a response directly, but it cannot easily use guards, and it does not know which controller method will handle the request. That limitation is the main reason to move logic inward when it needs routing context.

Guards come next, and they answer a single question: is this request allowed to proceed? A guard returns true, false, or a promise of either, and it can also throw an exception. Authentication and authorization are the canonical guard jobs. A guard that checks a JWT, loads the user, and attaches that user to the request object is doing exactly what guards are for. So is a role guard that checks whether the authenticated user has permission for the route. The important property of guards is that they run before interceptors and before pipes. That means an unauthenticated request never reaches your validation logic, never triggers an interceptor that might log sensitive data, and never wastes work parsing a body that will be rejected anyway. Putting authorization in a guard is therefore not just a style choice. It is a security ordering choice.

A common mistake is to put authorization logic inside the handler itself, after the service has already loaded data. That pattern leaks in two ways. It wastes a database query on a request that should have been rejected, and it invites inconsistency, because every handler must remember to perform the check. A guard centralizes the check and makes it visible at the route definition, where a reviewer can see it without reading the handler body. Consider the following guard, which is intentionally small. It does one job, and it does it before anything else in the handler chain can run.

import { Injectable, CanActivate, ExecutionContext, UnauthorizedException } from '@nestjs/common';

@Injectable()
export class JwtAuthGuard implements CanActivate {
  canActivate(context: ExecutionContext): boolean {
    const request = context.switchToHttp().getRequest();
    const user = request.user;
    if (!user) {
      throw new UnauthorizedException('Authentication required');
    }
    return true;
  }
}

Interceptors are the most misunderstood stage, because they wrap the handler on both sides. An interceptor that runs before the handler can log the start of a request, measure timing, or modify the incoming context. The same interceptor, after the handler returns, receives the result as an observable stream and can transform it, cache it, or measure the total duration. That two sided nature makes interceptors the right home for cross cutting behavior that needs to see both the request and the response: response shaping, serialization of outgoing data, timeout handling, and performance logging. Interceptors are not the place for authorization, because they run after guards, and they are not the place for input validation, because they run before pipes but operate at a coarser level than individual arguments.

The distinction between interceptors and middleware deserves attention, because they look similar from a distance. Middleware is framework agnostic and request focused. Interceptors are NestJS native, they have access to the execution context, they can use dependency injection fully, and they can transform the response stream with RxJS operators. If your logic needs to know which handler is running, or needs to transform the handler's return value, it belongs in an interceptor. If it needs to run for every request including requests that never reach a controller, middleware is the better fit. Choosing between them by asking what context the logic needs will get the answer right most of the time.

Pipes run after interceptors and before the handler, and they operate on individual arguments. A pipe receives a value, validates it or transforms it, and returns the result. If validation fails, the pipe throws, and the handler never runs. This is where NestJS validation belongs. The built in validation pipe, combined with class validator decorators on a data transfer object, turns a messy body parsing problem into a declarative one. You describe the shape you expect, and NestJS rejects anything else with a structured error before your business logic sees it. Pipes are also used for transformation, such as parsing a route parameter from a string into an integer, or parsing a query string into a typed object. The key property is granularity: pipes see one argument at a time, which makes them precise but unsuitable for decisions that depend on the whole request.

Here is a typical data transfer object and controller usage. Notice how little code the controller itself contains. The pipe has already guaranteed that the input matches the declared shape, so the handler can focus on behavior rather than defensive checks.

import { IsString, IsInt, Min, MaxLength } from 'class-validator';

export class CreatePostDto {
  @IsString()
  @MaxLength(200)
  title: string;

  @IsString()
  body: string;

  @IsInt()
  @Min(1)
  authorId: number;
}

import { Body, Controller, Post, UseGuards, UsePipes, ValidationPipe } from '@nestjs/common';

@Controller('posts')
export class PostsController {
  @Post()
  @UseGuards(JwtAuthGuard)
  @UsePipes(new ValidationPipe({ whitelist: true, forbidNonWhitelisted: true }))
  create(@Body() dto: CreatePostDto) {
    // At this point dto is validated and stripped of unknown fields.
    return { title: dto.title };
  }
}

The handler itself is the center of the lifecycle, and by the time it runs, the framework has already done a great deal of work. Authentication has passed, authorization has passed, the input has been validated and transformed, and any before logic in interceptors has run. The handler should therefore be thin. It calls a service, returns a result, and lets the framework handle the rest. Handlers that revalidate input, recheck permissions, or manually serialize output are usually compensating for a stage that was skipped earlier. Moving that logic back to its proper stage simplifies the handler and makes the system easier to test, because each stage can be tested in isolation.

After the handler returns, interceptors run again in reverse order, transforming the result. Then the response is sent. If anything in the chain threw, exception filters run. Exception filters are the last stage, and they convert thrown errors into HTTP responses. NestJS ships with a default filter that handles HTTP exceptions sensibly, but custom filters are valuable for two reasons. First, they let you normalize error shapes across the whole application, so clients receive a consistent structure regardless of which layer failed. Second, they let you map domain errors, such as a record not found in the service layer, into the correct HTTP status without leaking internal details. A filter should log the error with enough context to debug it, then return a safe response to the client. It should never return stack traces or database error messages to callers.

Execution order also applies to scope. Guards, interceptors, pipes, and filters can be bound globally, at the controller level, or at the individual route level. More specific bindings run after more general ones in some stages and before them in others, and the documentation's ordering rules repay careful reading. The practical rule is to bind broadly only when the behavior truly applies everywhere. A global validation pipe is usually correct. A global guard that requires authentication is correct only if every route, including health checks and public endpoints, should require it. Otherwise, prefer controller or route level bindings, and exempt public routes explicitly rather than weakening a global guard with special cases.

Docker Multi-Stage Builds: Small Images Without the Pain

Mubashir Taj — Sat, 31 Oct 2026 13:13:44 +0000

Every team has shipped a Docker image that was three times bigger than it needed to be, usually because the build tools traveled with the application. Node modules, compilers, test frameworks, and cached layers all ride along into production, slowing deploys, widening the attack surface, and confusing anyone who inspects the image. Multi-stage builds fix this with one idea: build in one stage, copy only the result into the next, and ship only the final stage.

The problem in one Dockerfile

A typical single-stage Node image looks like this:

FROM node:20
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build
CMD ["node", "dist/index.js"]

The final image contains the source, the dev dependencies, the build cache, and npm itself. None of that serves traffic. It only makes the image bigger and the deploy slower. A Go equivalent is worse: the compiler toolchain is hundreds of megabytes, and it has no business running in production.

The multi-stage fix

Split the Dockerfile into stages. The first stage builds. The second stage runs, and it copies only the build output:

# Stage 1: build
FROM node:20 AS build
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build
RUN npm prune --omit=dev

# Stage 2: run
FROM node:20-slim
WORKDIR /app
COPY --from=build /app/dist ./dist
COPY --from=build /app/node_modules ./node_modules
COPY package.json ./
USER node
CMD ["node", "dist/index.js"]

The final image contains the runtime, the built output, and production dependencies. Nothing else. The build stage is discarded after the copy, so its layers never ship. A Go version is even cleaner: build a static binary in a golang image, copy the single binary into a scratch or distroless image, and ship an artifact measured in megabytes.

# Stage 1: build a static binary
FROM golang:1.23 AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -o /app/server ./cmd/server

# Stage 2: ship only the binary
FROM scratch
COPY --from=build /app/server /server
ENTRYPOINT ["/server"]

A scratch image has no shell, no package manager, and nothing to exploit. That is a security property as much as a size property: an attacker who compromises the process finds no tools inside the container.

Three habits that make it work

First, order your layers for cache. Copy dependency manifests before source code, so dependency installation is cached across source changes. A Dockerfile that copies everything before installing dependencies rebuilds the world on every commit, and multi-stage or not, your CI time will show it.

Second, name your stages and copy explicitly. COPY --from=build with a named stage is readable and refactorable. Copying from an unnamed stage index is a bug waiting for someone to insert a stage above it.

Third, run as a non-root user in the final stage whenever the base image allows it. Multi-stage builds shrink the image; a non-root user shrinks the blast radius. Both belong in the same Dockerfile review checklist.

What to watch out for

Multi-stage builds have two common traps. The first is debugging: a production image with no shell cannot be exec'd into for inspection. Keep a debug variant of the image, or ship debug tooling in a sidecar, rather than bloating the production image for the rare debugging session.

The second is secret leakage across stages. A build argument or secret used in the build stage can persist in that stage's layers, and while those layers do not ship in the final image, they do exist in the build cache and in the image history if you are careless. Use BuildKit secret mounts for credentials instead of build arguments, and never copy credential files into any stage.

The payoff is concrete: images that are a fraction of their former size, deploys that finish in seconds instead of minutes, and production containers that contain only what the application needs to run. Multi-stage builds are not an optimization for later. They are the default shape of a serious Dockerfile, and retrofitting them onto an existing service takes an afternoon. That afternoon pays for itself on the first deploy.

Distroless, slim, and scratch: how small should you go

Once the build stage is separated, the remaining choice is the final base image, and the spectrum runs from full distributions down to nothing. A slim image keeps a package manager and a shell, which makes debugging easy and the image moderately sized. A distroless image keeps the language runtime and its certificates but drops the shell and package manager, which removes most of the tools an attacker would reach for. A scratch image keeps literally nothing but your binary, which is the smallest and most secure option and the hardest to debug.

The right choice depends on the runtime. Static Go binaries are natural scratch citizens: one file, no dependencies, no shell needed. Dynamic runtimes like Node or Python need their interpreter and system libraries, so slim or distroless fits better. The rule is to go as small as the runtime honestly allows, and to stop shrinking at the point where debugging becomes impossible rather than merely inconvenient. A team that cannot inspect a failing container will eventually add the tooling back under incident pressure, and an emergency image change is worse than a slightly larger image chosen deliberately.

BuildKit features worth turning on

Two BuildKit features multiply the value of multi-stage builds. The first is layer caching across CI runs: cache the dependency-install layer in a registry cache so that a code-only change rebuilds in seconds instead of minutes. The second is the secret mount, which lets the build use credentials without baking them into any layer. Both are one-line changes in most CI configurations, and both address the two complaints teams raise about Docker builds: that they are slow, and that they leak secrets. A multi-stage Dockerfile without BuildKit caching is still correct, but it leaves the most visible win on the table.

Measure the result after you convert. Record the old image size, the new image size, the old build time, and the new build time, and put the numbers in the pull request. "Smaller and faster" is a claim; "from 1.2 gigabytes to 180 megabytes, build time unchanged" is evidence, and evidence is what gets the next service converted too. One converted service is an experiment. A second one is a pattern. A tenth one is a standard nobody has to argue for again.

The checklist, in one place

Converting an existing Dockerfile takes an afternoon, and the checklist fits on one page. Split build and run into separate stages. Copy dependency manifests first for cache. Copy only build output into the final stage. Choose the smallest base image the runtime honestly allows. Run as a non-root user. Mount secrets with BuildKit instead of passing them as arguments. Keep a debug variant for inspection. Measure image size and build time before and after, and record both numbers. Then convert the next service, because the second conversion takes half the time and the tenth takes minutes. The habit compounds faster than the images shrink.

RAG Pipelines: Why Your Chatbot Hallucinates and How to Fix It

Mubashir Taj — Fri, 30 Oct 2026 13:14:56 +0000

Ask a chatbot a question about your own documentation and it answers confidently, fluently, and sometimes completely wrong. The model is not broken. It is doing exactly what it was trained to do: produce plausible text. Retrieval-augmented generation, or RAG, exists to change the job from "produce plausible text" to "answer from these documents." When RAG works, hallucinations drop sharply. When it fails, it usually fails in one of a handful of predictable places, and each one is fixable.

What RAG actually is

A RAG pipeline has four stages, and the model is only the last one. First, your documents are split into chunks. Second, each chunk is converted into a vector embedding, a list of numbers that captures its meaning. Third, at query time, the user's question is embedded the same way and the nearest chunks are retrieved. Fourth, those chunks are pasted into the model's prompt and the model answers from them.

Documents -> chunks -> embeddings -> vector store
Question -> embedding -> top-k chunks -> prompt + model -> answer

Most teams obsess over the model and neglect the first three stages. In practice, retrieval quality decides answer quality. A model given the right three paragraphs answers correctly. A model given the wrong three paragraphs answers confidently and wrong, because nothing in its prompt tells it otherwise.

Failure 1: chunks that destroy meaning

The most common RAG bug is chunking by fixed size with no respect for structure. A 500-token window that cuts a table in half, splits a function from its explanation, or separates a warning from the procedure it warns about produces chunks that are individually meaningless. Retrieval then returns fragments, and the model stitches fragments into fiction.

Fix it by chunking on structure: split by headings, by function, by paragraph, and keep small overlaps between adjacent chunks so context survives the boundary. Overlap of 10 to 20 percent is a reasonable starting point. Then read retrieved chunks for real queries and judge them by eye: would a human given only these chunks answer correctly? If not, the chunking is wrong, not the model.

Failure 2: retrieval that returns the wrong neighbors

Vector similarity is not relevance. A question about "refunds for annual plans" can retrieve a chunk about "refunds for monthly plans" because the words are close, while the chunk about annual plans sits just outside the top-k cutoff. Pure vector search also misses exact terms: a product code, an error string, a person's name.

The standard fix is hybrid retrieval: combine vector search with keyword search, typically BM25, and merge the results. The vector search catches paraphrases; the keyword search catches exact terms. Reranking the merged set with a small cross-encoder model before handing it to the generator is the second upgrade, and it is usually the highest-return change in the whole pipeline.

# Hybrid retrieval in sketch form: keyword hits plus vector hits, reranked.
keyword_hits = bm25_search(query, top_n=20)
vector_hits = vector_search(embed(query), top_n=20)
candidates = dedupe(keyword_hits + vector_hits)
ranked = cross_encoder_rerank(query, candidates)[:5]
answer = llm.generate(prompt_with(ranked))

Failure 3: prompts that invite invention

Even with perfect retrieval, a prompt that says "answer the question" invites the model to fill gaps from its training data. The fix is boring and effective: instruct the model to answer only from the provided context, and to say it does not know when the context does not contain the answer. "I don't know" is a feature, not a failure, and teams that treat it as one will keep shipping hallucinations.

Add citations to the output: require the model to quote or reference the chunk each claim comes from. Citations do two jobs at once. They let users verify answers, and they make hallucination visible during evaluation, because an uncited claim is a claim the pipeline cannot defend.

Failure 4: never measuring anything

The deepest RAG failure is building the pipeline and never evaluating it. Without a test set of real questions with known-good answers, every change is a guess. Build a set of 50 to 100 questions your users actually ask, with the correct answer and the source document noted. Then measure: how often does retrieval return the right chunk, and how often does the final answer match? Most teams discover that retrieval, not generation, is the bottleneck, and that a week of chunking and reranking work beats a month of model tuning.

Evaluation also catches data drift. Documents change, products rename, policies update. A chunk store built from last quarter's documentation answers last quarter's questions. Rebuild embeddings when source documents change, version the index, and re-run the test set after every rebuild. A RAG system without a refresh process is a snapshot pretending to be a service.

The honest summary

Hallucination in a RAG system is usually a retrieval failure wearing a model's clothes. Chunk on structure, retrieve with hybrid search, rerank before generating, prompt for grounded answers with citations, and measure against real questions. None of that is glamorous, and all of it is the difference between a chatbot that helps and one that confidently misleads. The model is the last 10 percent of the system. The pipeline is the other 90, and it is where the work belongs.

When RAG is the wrong tool

Not every question-answering problem is a RAG problem, and forcing it causes a different class of failure. If the answer requires computation, aggregation, or live data, retrieval cannot help. "How many open tickets do we have?" is a database query, not a document lookup, and a RAG pipeline that retrieves a stale wiki page about ticket counts will answer confidently and wrong. Route those questions to tools: let the model call a function that queries the live system, and reserve RAG for knowledge questions with stable, document-backed answers.

The same logic applies to questions with a single authoritative answer that changes often. Prices, quotas, and policy thresholds belong in a structured store queried at answer time, not in embeddings rebuilt nightly. A useful rule: if the answer would change between index rebuilds, do not put it in the index. Put a pointer in the index and fetch the value live.

Finally, respect the case where the honest answer is that no document covers the question. A RAG system that always answers, even when retrieval returns nothing relevant, is a hallucination machine with good branding. Build the abstention path deliberately: when the best retrieved chunk scores below a threshold, say so, and offer the closest related topic or a human contact. Users forgive "I don't know" far faster than they forgive a wrong answer delivered with confidence.

Start with one pipeline, not ten

If your team is new to RAG, resist the urge to build one pipeline per department. Build one pipeline for one well-understood document set, with a test set of real questions and a named owner for the index. Run it for a month, fix the chunking problems you find, add hybrid retrieval and reranking, and only then expand. A single pipeline that answers correctly teaches the organization what good looks like. Ten pipelines built at once teach it that RAG is unreliable, and that reputation is hard to reverse. Quality first, coverage second, always.