DEV Community: mihir mohapatra

Building a Battery Management System (BMS) Core in Rust

mihir mohapatra — Tue, 27 Oct 2026 05:44:04 +0000

Every lithium-ion battery you use, from a phone to an EV, has a small piece of software and hardware babysitting it. That is the Battery Management System (BMS). Without one, a pack can overcharge, overheat, or drift out of balance, and the failure modes range from a dead pack to a fire.

In this post I'll cover what a BMS does and then build a minimal BMS core in Rust.

What does a BMS actually do?

A BMS has four main jobs:

Monitoring: it measures the voltage of each cell, the pack current, and temperatures.
State estimation: it estimates State of Charge (SoC, "how full is it?") and State of Health (SoH, "how worn out is it?").
Protection: it opens the contactor or MOSFET when something goes out of range (over-voltage, under-voltage, over-temperature, over-current).
Balancing: cells in series drift apart over time, so it bleeds off the stronger cells to keep the pack even.

The flow in the diagram above is the whole loop: sensors feed the controller, the controller decides, and it acts on the pack through the contactor and balancing circuits.

Why Rust for a BMS?

A BMS is safety-critical and often runs on constrained hardware. Rust fits well because:

Memory safety without a GC: no use-after-free or data races in the protection logic.
Enums and pattern matching: faults and states become types the compiler forces you to handle.
no_std support: the same logic can run on a microcontroller (STM32, nRF, ESP32).
Zero-cost abstractions: predictable timing for control loops.

The Rust implementation

The example below is deliberately simple and uses only the standard library, so you can run it with cargo run. It covers:

Per-cell readings
SoC tracking with coulomb counting
Fault detection with a latched fault state
A cell balancing check

use std::time::Duration;

#[derive(Debug, Clone, Copy)]
struct CellReading {
    voltage_v: f32,
    temp_c: f32,
}

#[derive(Debug, Clone, Copy, PartialEq)]
enum Fault {
    OverVoltage(usize),
    UnderVoltage(usize),
    OverTemp(usize),
    OverCurrent(f32),
}

#[derive(Debug, PartialEq)]
enum State {
    Ok,
    Faulted(Fault),
}

struct Limits {
    v_max: f32,
    v_min: f32,
    temp_max_c: f32,
    current_max_a: f32,
}

struct Bms {
    limits: Limits,
    capacity_ah: f32,
    soc: f32, // 0.0 ..= 1.0
    state: State,
}

impl Bms {
    fn new(limits: Limits, capacity_ah: f32, initial_soc: f32) -> Self {
        Self { limits, capacity_ah, soc: initial_soc, state: State::Ok }
    }

    /// Returns the first fault found, if any.
    fn check(&self, cells: &[CellReading], current_a: f32) -> Option<Fault> {
        if current_a.abs() > self.limits.current_max_a {
            return Some(Fault::OverCurrent(current_a));
        }
        for (i, c) in cells.iter().enumerate() {
            if c.voltage_v > self.limits.v_max {
                return Some(Fault::OverVoltage(i));
            }
            if c.voltage_v < self.limits.v_min {
                return Some(Fault::UnderVoltage(i));
            }
            if c.temp_c > self.limits.temp_max_c {
                return Some(Fault::OverTemp(i));
            }
        }
        None
    }

    /// One control-loop tick. Positive current = discharging.
    fn update(&mut self, cells: &[CellReading], current_a: f32, dt: Duration) {
        // A fault is latched: once tripped, stay tripped until reset.
        if self.state != State::Ok {
            return;
        }
        if let Some(fault) = self.check(cells, current_a) {
            self.state = State::Faulted(fault);
            // In real hardware: open the contactor here.
            return;
        }
        // Coulomb counting: SoC -= (I * dt) / capacity
        let dt_h = dt.as_secs_f32() / 3600.0;
        self.soc = (self.soc - current_a * dt_h / self.capacity_ah).clamp(0.0, 1.0);
    }
}

/// Cells noticeably above the weakest cell get bled down.
fn cells_to_balance(cells: &[CellReading], threshold_v: f32) -> Vec<usize> {
    let min = cells.iter().map(|c| c.voltage_v).fold(f32::MAX, f32::min);
    cells
        .iter()
        .enumerate()
        .filter(|(_, c)| c.voltage_v - min > threshold_v)
        .map(|(i, _)| i)
        .collect()
}

fn main() {
    let limits = Limits {
        v_max: 4.2,
        v_min: 3.0,
        temp_max_c: 60.0,
        current_max_a: 100.0,
    };
    // 50 Ah pack, starting at 90% charge
    let mut bms = Bms::new(limits, 50.0, 0.9);

    let mut cells = vec![
        CellReading { voltage_v: 4.10, temp_c: 30.0 },
        CellReading { voltage_v: 4.12, temp_c: 31.0 },
        CellReading { voltage_v: 4.09, temp_c: 30.5 },
        CellReading { voltage_v: 4.15, temp_c: 31.5 },
    ];

    println!("Cells to balance: {:?}", cells_to_balance(&cells, 0.02));

    // Discharge at 25 A for 30 minutes, in 60 s ticks
    for _ in 0..30 {
        bms.update(&cells, 25.0, Duration::from_secs(60));
    }
    println!("SoC after 30 min: {:.1}%", bms.soc * 100.0);

    // Simulate a thermal event on cell 2
    cells[2].temp_c = 65.0;
    bms.update(&cells, 25.0, Duration::from_secs(60));
    println!("State: {:?}", bms.state);
}

Expected output:

Cells to balance: [1, 3]
SoC after 30 min: 65.0%
State: Faulted(OverTemp(2))

Walking through the code

Faults are a type, not a string. Fault is an enum that carries the index of the offending cell. Anywhere you match on it, the compiler makes you handle every case, so adding a new fault type can't silently slip past your protection logic.

The fault is latched. Once state becomes Faulted, update returns early and stops accumulating SoC. Real BMS designs do this on purpose: a pack that tripped on over-temperature should not auto-resume the moment the reading dips back under the limit.

SoC uses coulomb counting. It integrates current over time: SoC -= I * dt / capacity. It's simple and cheap, but errors accumulate (sensor offset, temperature effects), so production systems correct it using open-circuit voltage lookups or an Extended Kalman Filter.

Balancing is a simple comparison. We find the lowest cell voltage and flag any cell more than 20 mV above it. Hardware then bleeds those cells through a resistor (passive balancing).

Where to take it next

This is a toy, but it's a solid skeleton. Ways to extend it:

Go no_std: replace Vec with heapless::Vec and run it on an STM32 or ESP32 with embedded-hal.
Add hysteresis and debounce: require a fault to persist for N ticks before tripping, to avoid false positives from noisy sensors.
Better SoC: add an EKF, or a voltage-vs-SoC lookup table to correct drift.
Add SoH tracking: estimate capacity fade and internal resistance growth over cycles.
Talk CAN bus: publish pack state to the rest of the vehicle with the socketcan crate.
Test it properly: Rust's built-in test framework and property-based testing (proptest) are great for hammering protection logic with random inputs.

Wrapping up

A BMS is a great example of where Rust's strengths line up with the problem: strict correctness, predictable performance, and a type system that encodes safety rules. If you're building anything with batteries, from e-bikes to grid storage, this is a fun area to dig into.

If you build on this, I'd love to hear about it in the comments.

Designing an AI Gateway from Scratch: Routing, Failover, and Cost Control

mihir mohapatra — Fri, 23 Oct 2026 16:08:11 +0000

Why You Need an AI Gateway

The first LLM feature ships fast: one API key, one provider, one SDK call. Six months later you have:

4 services calling 3 different providers
API keys scattered across repos
No idea which team is burning the budget
An outage at one provider that took down your product

An AI gateway is a single entry point between your applications and LLM providers. It's the same idea as an API gateway, but it handles problems specific to LLMs: token-based billing, slow streaming responses, flaky providers, and non-deterministic outputs.

In this post we'll design one from scratch.

Requirements

Functional

One unified API (OpenAI-compatible) for all providers
Route requests across providers and models
Automatic retries and failover
Per-team API keys, quotas, and rate limits
Response caching
Streaming (SSE) support

Non-functional

Added latency overhead under 10 ms (p99)
No single point of failure
Full visibility into cost, latency, and errors per request
Provider keys never exposed to clients

High-Level Architecture

            ┌──────────────┐
 Clients ──▶│  Load Balancer│
            └──────┬───────┘
                   ▼
        ┌─────────────────────┐
        │   Gateway (stateless)│
        │  ┌───────────────┐  │
        │  │ Auth + Quotas │  │
        │  ├───────────────┤  │
        │  │ Cache lookup  │  │
        │  ├───────────────┤  │
        │  │ Router        │  │
        │  ├───────────────┤  │
        │  │ Provider      │  │
        │  │ adapters      │  │
        │  └───────────────┘  │
        └───┬─────┬─────┬─────┘
            ▼     ▼     ▼
        OpenAI Anthropic  Self-hosted
                          (vLLM)

   Side channels: Redis (limits, cache) · Postgres (keys, config)
                  Kafka/queue (usage events) · Metrics + traces

The key decision: keep the gateway stateless. All state (rate-limit counters, cache, key metadata) lives in Redis or Postgres, so you can scale gateway nodes horizontally behind a load balancer.

Core Components

1. Unified API Layer

Pick one schema and translate everything to it. OpenAI's chat completions format is the de facto standard, so clients can use existing SDKs by changing only the base URL.

Each provider gets an adapter that implements the same trait:

#[async_trait]
pub trait Provider: Send + Sync {
    fn name(&self) -> &str;

    async fn chat(&self, req: ChatRequest) -> Result<ChatResponse, ProviderError>;

    async fn chat_stream(
        &self,
        req: ChatRequest,
    ) -> Result<BoxStream<'static, Result<ChatChunk, ProviderError>>, ProviderError>;
}

Adding a new provider means writing one adapter. Nothing else changes.

2. Authentication and Quotas

Clients get virtual keys issued by the gateway. Real provider keys stay server-side.

A virtual key maps to:

the owning team or project
allowed models
rate limits (requests/min and tokens/min)
a monthly budget

Store hashed keys in Postgres and cache hot keys in memory with a short TTL.

3. Rate Limiting

LLM traffic needs two limits, not one:

Limit	Why
Requests per minute	Protects against bursts and abuse
Tokens per minute	Matches how providers actually bill and throttle

A token bucket in Redis, updated atomically with a Lua script, works well. Since output tokens aren't known up front, estimate on the way in, then reconcile with actual usage when the response completes.

4. The Router

The router picks a provider and model for each request. Strategies, from simple to advanced:

Static mapping: gpt-4o goes to OpenAI, claude-sonnet goes to Anthropic.
Weighted: split traffic 80/20 across providers for the same model class.
Latency-based: prefer the provider with the lowest recent p50.
Cost-based: send low-complexity prompts to a cheaper model.
Fallback chains: try A, then B, then C.

A config-driven fallback chain is a good place to start:

routes:
  - name: default-chat
    strategy: fallback
    targets:
      - provider: openai
        model: gpt-4o
        timeout_ms: 20000
      - provider: anthropic
        model: claude-sonnet
        timeout_ms: 20000
      - provider: self-hosted
        model: llama-3-70b
        timeout_ms: 30000

5. Retries, Timeouts, and Circuit Breakers

Failure handling is where a gateway earns its keep.

Retry only what's safe: 429s, 5xx, and connection errors. Never retry 4xx validation errors.
Exponential backoff with jitter to avoid thundering herds.
Respect Retry-After headers from providers.
Circuit breaker per provider: after N failures in a window, stop sending traffic for a cool-down period, then probe with a few requests.
Streaming caveat: once you've sent the first token to the client, you can't transparently fail over. Fail over only before the first byte.

closed ──(failures > threshold)──▶ open
  ▲                                  │
  │                          (cool-down elapsed)
  └──(probe succeeds)── half-open ◀──┘

6. Caching

Two layers:

Exact-match cache: hash of (model, messages, params). Cheap, safe, and effective for repeated prompts. Only cache when temperature is 0 or the caller opts in.
Semantic cache: embed the prompt and look up near neighbors in a vector store. Higher hit rate, but risk of returning a subtly wrong answer. Use a strict similarity threshold and make it opt-in per route.

7. Observability

Every request should emit one structured event:

{
  "request_id": "req_8f2c",
  "team": "search",
  "provider": "anthropic",
  "model": "claude-sonnet",
  "latency_ms": 1840,
  "ttft_ms": 420,
  "input_tokens": 912,
  "output_tokens": 304,
  "cost_usd": 0.0071,
  "cache": "miss",
  "retries": 1,
  "status": 200
}

Push these to a queue and process them asynchronously. Never block the response path on logging or billing writes. Track:

TTFT (time to first token), which is what users actually feel
Cost per team, model, and route
Error rate and fallback rate per provider

Handling Streaming

Streaming adds real complexity:

Proxy SSE chunks as they arrive. Don't buffer the full response.
Normalize each provider's chunk format into your unified schema.
Count tokens as they stream so quotas stay accurate.
Handle client disconnects by cancelling the upstream request. Otherwise you pay for tokens nobody reads.

Scaling and Reliability

Stateless nodes behind a load balancer; scale on CPU and open connections.
Redis for counters and cache. Run it replicated, and fail open on limits if Redis is down (a deliberate trade-off: availability over strict enforcement).
Multi-region: deploy gateways close to your apps. Keep config in Postgres with read replicas.
Config hot-reload: change routes without redeploying.
Backpressure: cap concurrent upstream requests per provider and shed load gracefully with a clear 429.

Trade-offs to Call Out

Decision	Upside	Downside
Single OpenAI-compatible schema	Easy adoption	Provider-specific features get lost or need passthrough
Gateway in the hot path	Central control	Extra hop and a critical dependency
Semantic caching	Big cost savings	Risk of wrong or stale answers
Fail-open rate limits	High availability	Brief over-consumption possible

Build vs. Buy

Existing options (LiteLLM, Portkey, Cloudflare AI Gateway, and others) cover a lot. Build your own when you need tight control over routing logic, strict data-residency rules, or very low overhead. Otherwise, start with an existing tool and replace pieces as you hit limits.

Wrapping Up

An AI gateway comes down to a few ideas:

A unified API with provider adapters
Virtual keys, quotas, and token-aware rate limits
Routing with fallbacks and circuit breakers
Caching where it's safe
Per-request observability for cost and latency

Start with a static router and exact-match cache, then add sophistication as real traffic shows you where the pain is.

Next in the series: RAG Pipelines in Production. Follow along so you don't miss it.

What does your gateway setup look like? Are you routing by cost, latency, or something else? Let me know in the comments.

AI & LLM Terminology Glossary: From Tokens to Orchestration

mihir mohapatra — Mon, 20 Jul 2026 08:33:49 +0000

Every field accumulates jargon, and AI/LLM engineering has accumulated it faster than most — partly because the field is young, and partly because terms get reused across research papers, vendor marketing, and production engineering with slightly different meanings each time.

This is the reference I wish existed when I started building AI infrastructure: one place with the terms that actually come up in day-to-day work, grouped by where you'll encounter them, explained at the level a backend engineer needs — not a research-paper level, and not a marketing-page level.

Core model concepts

Token — The basic unit a language model reads and generates. Not quite a word, not quite a character — usually a sub-word chunk (e.g. "tokenization" might split into "token" + "ization"). Every cost, context limit, and latency figure you'll deal with is measured in tokens, not words or characters.

Context window — The maximum number of tokens (input + output combined) a model can attend to in a single request. Everything the model "knows" about the current conversation has to fit inside this window — anything older gets truncated or has to be summarized.

Inference — The act of running a trained model to produce an output, as opposed to training it. When people talk about "inference cost" or "inference latency," they mean the cost/speed of using an already-trained model, not building one.

Parameters (weights) — The learned numerical values inside a model that encode what it "knows." Model size is usually described by parameter count (e.g. "a 7B model" = 7 billion parameters). More parameters generally means more capability, but also more memory and compute to run.

Fine-tuning — Continuing to train an already-trained (pretrained) model on a narrower, task-specific dataset, so it specializes without starting from scratch.

LoRA (Low-Rank Adaptation) — A fine-tuning technique that freezes the original model weights and trains a small set of additional low-rank matrices instead. It gets most of the benefit of full fine-tuning at a fraction of the compute and memory cost, and it's why you'll see so many "fine-tuned" open-weight variants floating around — full fine-tuning is expensive, LoRA isn't.

Quantization — Reducing the numerical precision used to store model weights (or the KV cache — see below) — for example fp16 down to int8 or int4. This shrinks memory footprint and often improves throughput, at some cost to output quality depending on how aggressive the quantization is.

Attention mechanism terms

Attention — The mechanism that lets a model weigh how relevant each previous token is when producing the next one. It's the core operation that makes transformers work, and it's also the most expensive part computationally.

KV cache (Key/Value cache) — Storage of the key and value tensors computed for previous tokens, so the model doesn't have to recompute them for every new token generated. It's usually the actual memory bottleneck in production serving — not the model weights — because it grows with both sequence length and number of concurrent requests.

MHA (Multi-Head Attention) — The original attention design, where every query head has its own dedicated set of key/value heads. Best quality, but the most expensive KV cache.

GQA (Grouped-Query Attention) — Multiple query heads share a smaller set of key/value heads, shrinking the KV cache significantly (often 4–8x) with minimal quality loss. Most modern open-weight models use this by default.

MQA (Multi-Query Attention) — The extreme version of GQA: all query heads share a single key/value head. Smallest possible cache, but a larger quality tradeoff than GQA.

Prefix caching — Reusing the KV cache for a shared prompt prefix (like a system prompt or few-shot examples) across multiple requests, instead of recomputing and re-storing it every time. Valuable for agent and chat workloads where the same instructions repeat across many calls.

PagedAttention — The memory management technique (popularized by vLLM) that treats the KV cache like OS-style virtual memory pages instead of requiring one contiguous block per request. This is what makes prefix caching and high-concurrency serving practical without the cache fragmenting into unusable gaps.

Retrieval & knowledge terms

Embedding — A numerical vector representation of text (or images, audio, etc.) that captures semantic meaning, such that similar content ends up with similar vectors. This is the foundation of search and retrieval in AI systems.

Vector database — A database optimized for storing embeddings and doing fast similarity search over them (e.g. "find the 5 most similar chunks to this query"). Examples include Pinecone, Weaviate, Qdrant, and pgvector as a Postgres extension.

RAG (Retrieval-Augmented Generation) — An architecture where the model's response is grounded by first retrieving relevant documents/chunks (usually via a vector database) and injecting them into the prompt, rather than relying purely on what the model memorized during training. This is how you get a model to answer questions about your own private documents.

Chunking — Splitting source documents into smaller pieces before embedding them, since embedding an entire document as one vector loses too much granularity for precise retrieval. Chunk size and overlap strategy directly affect RAG retrieval quality.

Reranking — A second-pass step in RAG pipelines where an initial broad set of retrieved candidates gets re-scored by a more precise (often more expensive) model before the final few are used, improving precision over similarity search alone.

Agentic system terms

Agent — A system where a model doesn't just respond once, but runs in a loop: reasoning, calling tools, observing results, and deciding what to do next until the task is complete.

Tool use / function calling — The mechanism by which a model can invoke external functions (search the web, query a database, run code) as part of its reasoning process, rather than being limited to generating text.

MCP (Model Context Protocol) — A standardized protocol for connecting models to external tools and data sources, so tool integrations aren't reinvented per-framework. Increasingly the common interface across agent frameworks and IDEs.

Orchestration — Coordinating multiple agents working together on a task — a central coordinator decomposing work and dispatching it to specialized worker agents, as opposed to one agent trying to do everything.

Handoff — A pattern where one agent, mid-task, transfers control (and relevant context) to a different, better-suited agent — common in support/triage-style systems.

Production & serving terms

Latency — Time taken to get a response. Often split into time to first token (TTFT) — how long before generation starts — and inter-token latency — how long between subsequent tokens once generation is underway. These respond to different optimizations.

Throughput — How many requests (or tokens) a system can process per unit time, as opposed to how fast any single request completes. Batching improves throughput, sometimes at the cost of individual request latency.

Continuous batching — A serving technique where new requests can join an in-progress batch as soon as a slot frees up, rather than waiting for the whole batch to finish. This is a major reason serving frameworks like vLLM outperform naive request-at-a-time loops.

Tracing — Recording the full path of a request through a system — including retries, tool calls, and fallbacks — as a structured, queryable record, rather than relying on scattered logs. Essential once an LLM app has more than one moving part.

Eval / eval loop — A systematic, often continuous process for scoring model output quality — structural checks (did the JSON parse), golden-set regression tests, or sampled judge-model scoring — used to catch quality regressions the way tests catch code regressions.

Cost & efficiency terms

Prompt / completion tokens — Most API pricing is split into input (prompt) tokens and output (completion) tokens, usually priced differently, with output tokens typically costing more per token.

Distillation — Training a smaller "student" model to mimic the behavior of a larger "teacher" model, producing something cheaper to run that retains much of the larger model's capability on the tasks it was distilled for.

Open-weight vs. closed model — Open-weight models have downloadable weights you can self-host and run under your own infrastructure (e.g. Llama, Mistral, Qwen); closed models are only accessible through a provider's hosted API (e.g. Claude, GPT). This distinction drives a real cost/control tradeoff, not just a licensing preference.

Why the vocabulary matters

None of these terms are difficult individually — the friction is that they get used loosely across blog posts, vendor docs, and papers, often without definition, on the assumption you already know them. Having a shared, precise vocabulary is what makes the difference between skimming an infra discussion and actually being able to make the tradeoff decisions it's describing — which quantization level, which attention variant, whether a task actually needs orchestration or just a well-scoped single agent.

This glossary is part of a series on AI infrastructure patterns. If a term you've run into isn't here, drop it in the comments and I'll add it.

Agentic Systems: From Single Agent to Orchestration

mihir mohapatra — Thu, 16 Jul 2026 09:11:30 +0000

A single agent — one model, a system prompt, a handful of tools, and a loop — gets you surprisingly far. It's also where most people stop, because it works for demos and falls over the moment a task needs more than one kind of reasoning happening at once.

Multi-agent orchestration isn't a hype upgrade over a single agent — it's what you reach for when a task naturally decomposes into sub-tasks that benefit from different context, different tools, or genuinely different roles. This post covers when that shift is warranted, the orchestration patterns actually used in production, and a small working example of a coordinator delegating to worker agents.

Why a single agent stops being enough

A single agent's context window and tool list grow with every capability you bolt on. Three failure modes show up as that happens:

Context pollution — a research task's intermediate scratch work bleeds into the reasoning for an unrelated coding sub-task, degrading both
Tool overload — past a certain number of tools in one system prompt, selection accuracy drops; the model spends more of its reasoning deciding which tool than actually using it well
No separation of concerns — a single agent can't easily hold "be a careful planner" and "be a fast, narrow executor" as two different behavioral modes at once

Orchestration solves this by splitting one large, blurry job into a coordinator plus specialized workers, each with a narrow role, its own context, and only the tools it actually needs.

The patterns that show up in practice

There isn't one "multi-agent architecture" — a few concrete patterns cover most real systems:

Sequential (pipeline) — agent A's output feeds agent B, feeds agent C. Good for linear workflows like research → draft → review.
Concurrent (fan-out/fan-in) — the coordinator dispatches the same or related sub-tasks to multiple agents in parallel, then merges results. Good for independent sub-tasks — e.g. summarizing five documents at once.
Handoff — one agent decides mid-task that another agent is better suited and transfers control (and context) to it. Good for triage-style systems, like a support bot handing off to a specialist agent.
Hierarchical (coordinator/worker) — a central planner decomposes the task and delegates to specialized workers, then assembles their outputs into a final answer. This is the most common production pattern and the one the example below uses.

Frameworks like LangGraph model this as an explicit state graph, CrewAI models it as role-based "crews," and minimal SDKs like the OpenAI Agents SDK treat sub-agents as callable tools from a parent agent. The underlying pattern is the same regardless of framework — what differs is how much structure and observability you get by default.

A working example: coordinator + worker agents in Rust

Here's a minimal hierarchical setup: a coordinator agent receives a research request, decomposes it into sub-tasks, dispatches them to worker agents concurrently, and merges the results.

use futures::future::join_all;
use serde::{Deserialize, Serialize};

#[derive(Debug, Clone, Serialize, Deserialize)]
struct SubTask {
    id: String,
    description: String,
    assigned_role: AgentRole,
}

#[derive(Debug, Clone, Serialize, Deserialize)]
enum AgentRole {
    Researcher,
    CodeAnalyst,
    Summarizer,
}

#[derive(Debug, Serialize)]
struct WorkerResult {
    task_id: String,
    output: String,
    tokens_used: u32,
}

struct Coordinator {
    llm_client: LlmClient,
}

impl Coordinator {
    /// Decompose a high-level request into sub-tasks using the coordinator's own LLM call.
    async fn plan(&self, request: &str) -> Vec<SubTask> {
        let planning_prompt = format!(
            "Break this request into 2-4 independent sub-tasks, each assigned \
             to Researcher, CodeAnalyst, or Summarizer. Request: {request}"
        );
        let response = self.llm_client.complete(&planning_prompt).await;
        parse_subtasks(&response) // structured output parsing, omitted for brevity
    }

    /// Dispatch sub-tasks to worker agents concurrently, then merge.
    async fn execute(&self, request: &str) -> String {
        let subtasks = self.plan(request).await;

        let worker_futures = subtasks.into_iter().map(|task| {
            let client = self.llm_client.clone();
            async move { run_worker(client, task).await }
        });

        let results: Vec<WorkerResult> = join_all(worker_futures).await;

        self.merge(results).await
    }

    /// A final LLM call that synthesizes worker outputs into one coherent answer.
    async fn merge(&self, results: Vec<WorkerResult>) -> String {
        let combined = results.iter()
            .map(|r| format!("[{}]: {}", r.task_id, r.output))
            .collect::<Vec<_>>()
            .join("\n\n");

        let merge_prompt = format!(
            "Synthesize these sub-task results into one coherent response:\n\n{combined}"
        );
        self.llm_client.complete(&merge_prompt).await
    }
}

async fn run_worker(client: LlmClient, task: SubTask) -> WorkerResult {
    let role_prompt = match task.assigned_role {
        AgentRole::Researcher => "You are a focused researcher. Be concise and cite sources.",
        AgentRole::CodeAnalyst => "You are a code analyst. Focus only on technical accuracy.",
        AgentRole::Summarizer => "You are a summarizer. Compress without losing key facts.",
    };

    let prompt = format!("{role_prompt}\n\nTask: {}", task.description);
    let output = client.complete(&prompt).await;

    WorkerResult {
        task_id: task.id,
        output,
        tokens_used: estimate_tokens(&output),
    }
}

A few things worth calling out in this shape:

Each worker gets its own narrow system prompt (role_prompt) instead of sharing the coordinator's full context — this is the actual mechanism that fixes context pollution.
join_all runs workers concurrently, not sequentially — for independent sub-tasks, this is a straightforward latency win that a single-agent loop can't get you.
The merge step is itself an LLM call, not string concatenation — worker outputs are often redundant or need reconciling, and skipping this step is a common source of disjointed final answers.

Where this pattern breaks down

Multi-agent orchestration isn't free — it's worth naming the costs explicitly, since a lot of write-ups skip this part:

More tokens spent overall — planning, dispatch, and merge steps are all additional LLM calls layered on top of the actual work
Harder to debug — a wrong final answer could originate in planning, in any worker, or in the merge step; this is exactly why the tracing patterns from the observability post in this series matter more, not less, once you add orchestration
Coordination overhead can exceed the benefit for genuinely simple, linear tasks — a single well-scoped agent with good tools is still the right default until you hit one of the failure modes above

The Gartner-cited pattern worth taking seriously: a meaningful share of agentic projects get cancelled not because agents don't work, but because teams reached for orchestration before the task actually needed it, and the coordination overhead outweighed the gain.

A reasonable decision rule

Reach for a single agent by default. Move to orchestration when at least one of these is true:

The task has genuinely independent sub-parts that would otherwise share (and pollute) one context window
Different sub-tasks need meaningfully different tool sets or behavioral instructions
You need parallelism for latency reasons, not just organizational tidiness

If none of those apply, a single well-designed agent with good tool selection will usually outperform a multi-agent system on both cost and reliability.

What's next

The next post in this series steps back to a cost/control framework: when running open-weight models yourself makes sense versus calling a closed API, and how to reason about that tradeoff instead of defaulting to whichever one is trending.

This is part of a series on AI infrastructure patterns, drawing on production work building multi-provider AI gateways in Rust. Follow along for the rest of the series.

Inference Optimization for the Rest of Us — KV Cache, Quantization, and Latency Tradeoffs

mihir mohapatra — Thu, 09 Jul 2026 16:58:26 +0000

Most inference optimization content is written for people running fleets of H100s. If you're serving models out of a single GPU box, a modest EC2/ECS setup, or even CPU inference for smaller models, a lot of that advice doesn't transfer — the constraints are different, and the "just add more GPUs" answer isn't available to you.

This post covers the three levers that actually move the needle at small-to-mid scale: KV cache management, quantization, and the latency tradeoffs that come with both. No cluster-scale assumptions.

Why this matters before it's a crisis

Inference cost and latency don't show up as a line item until they do — usually right around the point where a side project or an MVP gets real traffic. At that point you have three options: spend more on hardware, spend more on hosted API calls, or actually understand what's eating your latency and memory budget. The third option is the only one that scales your understanding along with your infrastructure.

Lever 1: KV cache — what it is and why it dominates memory

Every token a transformer generates attends back to all previous tokens. Naively, that means recomputing attention over the whole sequence at every step — wildly wasteful. The KV cache stores the key and value tensors from previous tokens so each new token only needs one forward pass, not a full replay.

The tradeoff: that cache grows linearly with sequence length, and its memory footprint is often the actual bottleneck — not the model weights themselves — once you're serving multiple concurrent requests with long contexts.

A rough sizing formula, per request:

kv_cache_bytes = 2 * num_layers * num_heads * head_dim * seq_len * bytes_per_element

The 2 is for both K and V. For a 7B-class model with 32 layers, 32 heads, head_dim 128, at fp16 (2 bytes) and a 4096-token context, that's already several hundred MB — per concurrent request. Multiply by your target concurrency and you'll see why KV cache, not weights, is usually what determines how many simultaneous users a single GPU can serve.

Practical levers on KV cache size

Multi-query / grouped-query attention (GQA) — if you're choosing a model, prefer architectures using GQA (most modern open-weight models do). It shares K/V heads across multiple query heads, cutting cache size significantly with minimal quality loss.
Sliding window attention — cap how far back the cache needs to reach for local-context tasks. Not appropriate for tasks needing full long-range recall, but a real win for chat-style workloads with short working context.
Cache eviction policies — for multi-turn chat apps, evict or summarize older turns instead of keeping the full cache alive indefinitely. This is an application-level decision, not just an inference-engine setting.
PagedAttention-style memory management (used in vLLM and similar serving frameworks) — treats the KV cache like virtual memory pages instead of requiring contiguous allocation, which dramatically reduces fragmentation waste when you have many concurrent requests with different sequence lengths.

If you're rolling your own serving layer rather than using vLLM/TGI, this is the single highest-leverage thing to borrow from those projects even if you don't adopt them wholesale.

Lever 2: Quantization — trading precision for throughput

Quantization reduces the numeric precision of model weights (and sometimes activations), shrinking both memory footprint and, often, latency.

Precision	Relative size	Typical use case
fp16/bf16	1x (baseline)	Default for most hosted APIs
int8	~0.5x	Good quality retention, solid speedup
int4 (GPTQ/AWQ)	~0.25x	Runs 7B-class models on consumer GPUs
GGUF quantized (Q4_K_M etc.)	~0.25–0.35x	CPU/edge inference via llama.cpp

The practical decision isn't "should I quantize" — for anything outside a hyperscaler budget, you almost certainly should. It's which quantization scheme, and that depends on your bottleneck:

Memory-bound (model barely fits, or you want more concurrent requests) → int4 or GGUF gets you the most headroom
Latency-bound on CPU → GGUF via llama.cpp with an appropriate quant level (Q4_K_M is a reasonable default starting point)
Quality-sensitive (structured output, code generation, anything requiring precise instruction-following) → stick to int8 or test carefully before dropping to int4; quality degradation isn't linear and some tasks are far more sensitive than others

A pattern worth adopting: don't guess — benchmark your actual task against 2–3 quantization levels before committing. A model that quantizes cleanly on general chat can degrade noticeably on structured extraction or tool-calling tasks.

// Rough shape of a quant-level config in a gateway/router setup
#[derive(Debug, Clone)]
struct ModelVariant {
    name: String,
    quant: QuantLevel,      // Fp16, Int8, Int4Awq, GgufQ4KM
    vram_gb: f32,
    avg_tokens_per_sec: f32,
}

fn select_variant(task: TaskProfile, variants: &[ModelVariant]) -> &ModelVariant {
    // route structured/tool-calling tasks to higher precision,
    // route casual chat/summarization to the cheapest variant that fits
    variants.iter()
        .filter(|v| task.quality_floor <= v.quant.quality_score())
        .min_by(|a, b| a.vram_gb.partial_cmp(&b.vram_gb).unwrap())
        .unwrap_or(&variants[0])
}

This is the same instinct as multi-provider routing in a gateway — route by task requirements, not a single fixed model for everything.

Lever 3: Latency tradeoffs — where the milliseconds actually go

Once cache and quantization are handled, the remaining latency budget splits into pieces that respond to different fixes:

Time to first token (TTFT) — dominated by prompt processing (prefill). Long system prompts or heavy RAG context directly inflate this. If TTFT matters more than total generation time for your use case (e.g. interactive chat), trimming prompt bloat is often a bigger win than any inference engine tuning.
Inter-token latency — dominated by memory bandwidth during decode, which is exactly where quantization and KV cache size have direct leverage.
Batching — continuous batching (as opposed to static batching) lets new requests join a batch mid-flight instead of waiting for the whole batch to finish, which matters a lot under real, uneven traffic patterns. This is one of the core reasons frameworks like vLLM outperform naive serving loops even before touching quantization.

A concrete tradeoff worth naming explicitly: batching improves throughput but can hurt per-request latency under load, since a request may wait briefly for a batch slot. For a single interactive assistant, that's rarely visible. For a shared production endpoint under real concurrency, it's the difference between p50 and p95 latency looking very different — and it's worth deciding up front which one your use case actually cares about.

Putting it together: a practical decision order

For a small-to-mid scale deployment, the order that gets you the most benefit per hour invested:

Pick a GQA-architecture model if you have a choice — this is free and structural
Quantize and benchmark against your actual task, not a generic leaderboard — int8 as a safe default, int4/GGUF if memory-bound
Adopt continuous batching via an existing serving framework (vLLM, TGI) rather than writing your own naive batching loop
Trim prompt bloat if TTFT matters for your UX — this is often the cheapest win and the most overlooked
Only then consider hardware changes — by this point you'll actually know whether you're memory-bound or compute-bound, which determines what upgrade (if any) is worth the cost

None of this requires hyperscaler infrastructure. It requires knowing which of these five levers you're actually constrained by before spending money on the wrong fix.

What's next

The next post in this series moves back up the stack — open-weight versus closed API models, and a practical cost/control framework for deciding which to run yourself versus call through a provider.

This is part of a series on AI infrastructure patterns, drawing on production work building multi-provider AI gateways in Rust. Follow along for the rest of the series.

Observability for LLM Apps: Tracing, Cost Tracking, and Eval Loops

mihir mohapatra — Mon, 06 Jul 2026 08:43:57 +0000

If you've shipped a traditional backend service, you already know the observability checklist: logs, metrics, traces, alerts. LLM-powered apps need all of that — plus a few things that don't exist in a normal request/response world: token spend, prompt/response pairs, and quality drift that no HTTP status code will ever tell you about.

This post covers the three pillars that actually matter once an LLM app leaves the demo stage:

Tracing — following a request across retries, tool calls, and multi-provider fallbacks
Cost tracking — turning token counts into real-time dollar visibility per route, user, or feature
Eval loops — catching quality regressions before your users do

I'll use patterns from a multi-provider AI gateway (Rust, Axum, Prometheus) as the running example, but the ideas apply regardless of stack.

Why normal APM isn't enough

A standard APM trace tells you latency and status code. For an LLM call, that's barely half the picture. The same 200 OK can represent:

A cheap 400-token completion from a fast model
A $2 completion from a frontier model after two retries
A technically successful but hallucinated response

None of that shows up in a duration histogram. You need traces that carry LLM-specific attributes, not just HTTP ones.

Pillar 1: Tracing

The unit of work in an LLM app usually isn't a single call — it's a chain: prompt construction → provider call → possible retry/fallback → tool invocation → final response. Each of those needs to be a span, not just the outer request.

A minimal span shape for an LLM call looks like this:

#[derive(Debug, Serialize)]
struct LlmSpan {
    trace_id: String,
    span_id: String,
    parent_span_id: Option<String>,
    provider: String,        // "anthropic", "openai", "ollama"
    model: String,
    prompt_tokens: u32,
    completion_tokens: u32,
    latency_ms: u64,
    status: SpanStatus,      // Success, Retried, Failed, RateLimited
    retry_count: u8,
    fallback_from: Option<String>, // which provider it fell back from
}

The two fields that matter most for debugging in production are retry_count and fallback_from. Without them, a "successful" trace hides the fact that your primary provider is degrading and your fallback logic is quietly absorbing the pain — until it can't.

Wire this into OpenTelemetry so it plays nicely with whatever trace backend you already run:

use opentelemetry::trace::{Tracer, Span};

let tracer = global::tracer("llm-gateway");
let mut span = tracer.start("llm.completion");
span.set_attribute(KeyValue::new("llm.provider", provider.to_string()));
span.set_attribute(KeyValue::new("llm.model", model.to_string()));
span.set_attribute(KeyValue::new("llm.prompt_tokens", prompt_tokens as i64));
span.set_attribute(KeyValue::new("llm.completion_tokens", completion_tokens as i64));
span.set_attribute(KeyValue::new("llm.retry_count", retry_count as i64));

This gives you a waterfall view in Jaeger/Tempo where a fallback chain is visually obvious instead of buried in logs.

Pillar 2: Cost tracking

Token counts alone don't tell a story anyone outside engineering cares about. The move is to convert tokens to cost at write-time and expose it as a first-class Prometheus metric, not something you reconstruct later from a token count times a price you looked up once and forgot to update.

use prometheus::{register_counter_vec, CounterVec};

static LLM_COST_USD: Lazy<CounterVec> = Lazy::new(|| {
    register_counter_vec!(
        "llm_cost_usd_total",
        "Cumulative LLM spend in USD",
        &["provider", "model", "route"]
    ).unwrap()
});

fn record_cost(provider: &str, model: &str, route: &str, prompt_tokens: u32, completion_tokens: u32) {
    let (input_rate, output_rate) = pricing_for(provider, model);
    let cost = (prompt_tokens as f64 * input_rate) + (completion_tokens as f64 * output_rate);
    LLM_COST_USD
        .with_label_values(&[provider, model, route])
        .inc_by(cost);
}

Keep pricing_for in a config file, not hardcoded — provider pricing changes more often than you'd like, and a stale rate table quietly lies to your finance dashboard.

With provider, model, and route as labels, one PromQL query answers the question every team eventually asks:

sum by (route) (
  rate(llm_cost_usd_total[1h])
)

That's "which feature is burning the budget," per hour, without a spreadsheet. Pair it with a budget-based alert:

- alert: LLMCostSpike
  expr: sum(rate(llm_cost_usd_total[15m])) * 3600 > 5
  for: 10m
  annotations:
    summary: "LLM spend exceeding $5/hr sustained rate"

This is the single highest-leverage dashboard panel you can add — it's usually the first thing a manager asks for once real usage starts.

Pillar 3: Eval loops

This is the part with no equivalent in traditional backend observability. A response can be fast, cheap, and a 200 — and still be wrong. Nothing in your metrics stack catches that unless you build for it.

The practical version of an eval loop, at gateway scale, isn't a big offline benchmark suite — it's a lightweight, continuous check on production traffic:

Golden set regression — a small, versioned set of prompt/expected-output pairs run against every model/provider change before it ships
Sampled production scoring — periodically re-score a random 1–2% of real completions using a cheaper judge model, and track the score as a metric over time
Structural checks — cheap, deterministic checks that don't need a judge model at all: did the JSON parse, is the response non-empty, is it within expected length bounds

struct EvalResult {
    trace_id: String,
    check_name: String,       // "json_valid", "length_bounds", "judge_score"
    passed: bool,
    score: Option<f32>,
}

async fn run_structural_checks(response: &str) -> Vec<EvalResult> {
    vec![
        EvalResult {
            trace_id: current_trace_id(),
            check_name: "json_valid".into(),
            passed: serde_json::from_str::<Value>(response).is_ok(),
            score: None,
        },
        EvalResult {
            trace_id: current_trace_id(),
            check_name: "length_bounds".into(),
            passed: (20..4000).contains(&response.len()),
            score: None,
        },
    ]
}

Feed the pass rate into the same Prometheus instance as everything else:

static EVAL_PASS_RATE: Lazy<GaugeVec> = Lazy::new(|| {
    register_gauge_vec!(
        "llm_eval_pass_rate",
        "Rolling eval pass rate by check",
        &["check_name"]
    ).unwrap()
});

Now a model swap, a prompt template change, or a silent provider-side model update all show up as a dip on the same dashboard where you already watch latency and cost — instead of as a support ticket three days later.

Putting it together

The dashboard that actually gets used day-to-day has three rows, matching the three pillars:

Tracing row: p50/p95 latency by provider, retry rate, fallback rate
Cost row: spend by route/model, hourly burn rate, budget alert status
Quality row: structural check pass rate, sampled judge score trend

None of these pillars is optional once an LLM feature has real users. Tracing tells you what happened, cost tracking tells you what it's costing you, and eval loops tell you whether it's still working — and that third one is the piece most teams skip until it's already a production incident.

What's next

The next post in this series covers inference optimization — KV cache strategies and quantization tradeoffs for teams that aren't running hyperscaler-scale infrastructure but still need to control latency and cost at the model-serving layer.

This is part of a series on AI infrastructure patterns, drawing on production work building multi-provider AI gateways in Rust. Follow along for the rest of the series.

FastAPI Crash Course: Build a CRUD REST API in ~40 Lines of Python

mihir mohapatra — Sun, 28 Jun 2026 09:38:51 +0000

I spend most of my days writing Java and Kotlin for FinTech systems at scale. But every now and then, Python pulls me back in — and FastAPI is exactly the reason why. This is a hands-on crash course based on a project I published on GitHub.

👉 View the full source on GitHub

Why FastAPI?

Coming from a Spring Boot background, my first instinct was scepticism. Another Python framework? But FastAPI is genuinely different:

Automatic interactive docs via Swagger UI — no extra config needed
Request validation powered by Pydantic — think of it as @Valid + @RequestBody baked in
Type hints as the source of truth — Python's type system does the heavy lifting
Async-ready from the ground up — ASGI all the way

The best part? You can build a fully working REST API in under 50 lines.

What We're Building

A Tea CRUD API — simple, but covers every REST pattern:

Method	Endpoint	What It Does
GET	`/`	Health / welcome message
GET	`/teas`	List all teas
POST	`/teas`	Add a new tea
PUT	`/teas/{tea_id}`	Update an existing tea
DELETE	`/teas/{tea_id}`	Remove a tea

No database — we're using an in-memory list to keep the focus on FastAPI itself.

Prerequisites

Python 3.10+
pip

Verify:

python --version
pip --version

Setup

Create and activate a virtual environment:

# Windows (PowerShell)
python -m venv venv
.\venv\Scripts\Activate

# macOS / Linux
python3 -m venv venv
source venv/bin/activate

Install dependencies:

pip install "fastapi[standard]"

This single command installs FastAPI, Uvicorn (the ASGI server), and all optional extras — all you need to get running.

The Full Code

Create a file called main.py and drop this in:

from fastapi import FastAPI
from pydantic import BaseModel
from typing import List

app = FastAPI()

class Tea(BaseModel):
    id: int
    name: str
    origin: str

teas: List[Tea] = []

@app.get("/")
def read_root():
    return {"message": "Welcome to chai code"}

@app.get("/teas")
def get_teas():
    return teas

@app.post("/teas")
def add_teas(tea: Tea):
    teas.append(tea)
    return tea

@app.put("/teas/{tea_id}")
def update_tea(tea_id: int, update_tea: Tea):
    for index, tea in enumerate(teas):
        if tea.id == tea_id:
            teas[index] = update_tea
            return update_tea
    return {"error": "Tea not found"}

@app.delete("/teas/{tea_id}")
def delete_tea(tea_id: int):
    for index, tea in enumerate(teas):
        if tea.id == tea_id:
            deleted = teas.pop(index)
            return deleted
    return {"error": "Tea not found"}

That's it. 43 lines. Let's break it down.

Code Walkthrough

1. The App Instance

app = FastAPI()

This is your application object — equivalent to new SpringApplication() in Spring Boot, but in one line with no annotations, XML, or config files.

2. The Pydantic Model

class Tea(BaseModel):
    id: int
    name: str
    origin: str

BaseModel from Pydantic is the magic here. Any class that extends it automatically gets:

Request body parsing — FastAPI reads the incoming JSON and maps it to this class
Type validation — if id isn't an integer, FastAPI returns a 422 Unprocessable Entity with a clear error message
Serialization — returning a Tea instance automatically serializes it to JSON

For Java developers: think @Data + @Valid + Jackson — except it's one import and zero annotations on your fields.

3. The In-Memory Store

teas: List[Tea] = []

Just a Python list typed with List[Tea]. It acts as our database for this example. Doesn't survive restarts — but that's intentional for a learning project. You'd swap this with SQLAlchemy + PostgreSQL in a production setup.

4. GET — List All Teas

@app.get("/teas")
def get_teas():
    return teas

Return the list directly. FastAPI serializes List[Tea] to a JSON array automatically. No ResponseEntity, no @ResponseBody, no boilerplate.

5. POST — Add a Tea

@app.post("/teas")
def add_teas(tea: Tea):
    teas.append(tea)
    return tea

The function parameter tea: Tea tells FastAPI: "expect a JSON body matching the Tea schema." Pydantic validates it. If validation fails, FastAPI returns a detailed 422 error before your function is even called.

6. PUT — Update by ID

@app.put("/teas/{tea_id}")
def update_tea(tea_id: int, update_tea: Tea):
    for index, tea in enumerate(teas):
        if tea.id == tea_id:
            teas[index] = update_tea
            return update_tea
    return {"error": "Tea not found"}

{tea_id} in the path is a path parameter — FastAPI extracts it and coerces it to int automatically. You get both a path param (tea_id) and a request body (update_tea) from a single function signature. Clean.

7. DELETE — Remove by ID

@app.delete("/teas/{tea_id}")
def delete_tea(tea_id: int):
    for index, tea in enumerate(teas):
        if tea.id == tea_id:
            deleted = teas.pop(index)
            return deleted
    return {"error": "Tea not found"}

Linear scan through our in-memory list — fine for learning, but you'd index by ID with a dict or use a database in production.

Run It

python -m uvicorn main:app --reload

Expected output:

INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO:     Started reloader process

The --reload flag means the server restarts automatically on every file save — great for development.

Interactive API Docs (Free!)

This is FastAPI's killer feature. Open your browser:

URL	What You Get
`http://127.0.0.1:8000/docs`	Swagger UI — test every endpoint interactively
`http://127.0.0.1:8000/redoc`	ReDoc — clean, readable API reference

No extra setup. No Postman required for a quick test. The docs are generated directly from your type hints and Pydantic models.

Quick Test via Swagger UI

Go to http://127.0.0.1:8000/docs
Click POST /teas → Try it out
Paste this body:

{
  "id": 1,
  "name": "Assam Tea",
  "origin": "India"
}

Click Execute
Now hit GET /teas — you'll see your tea in the list

What's Next (Production Roadmap)

This in-memory app is intentionally minimal. Here's how you'd evolve it toward production:

Concern	Tool
Persistent storage	SQLAlchemy + PostgreSQL or SQLite
Migrations	Alembic
Auth	JWT via `python-jose` + `passlib`
Dependency Injection	FastAPI's built-in `Depends()` system
Environment config	`pydantic-settings`
Containerization	Docker + `uvicorn` in production mode
Testing	`pytest` + `httpx` (async test client)
Async DB	`asyncpg` + SQLAlchemy async sessions

Java/Kotlin Developer Take

Having spent years with Spring Boot, here's my honest comparison:

FastAPI wins on:

Speed of prototyping — no boilerplate ceremony
Auto-generated docs that are actually useful
Cleaner validation without annotation overload

Spring Boot still wins on:

Enterprise ecosystem maturity
Structured DI with complex bean graphs
Mature monitoring (Actuator, Micrometer)

FastAPI fills a real gap: it's the right tool when you want Python's data ecosystem (ML, data pipelines) behind a production-quality HTTP API. The two aren't in competition — they're tools for different contexts.

Source Code

Everything in this post is available here:

github.com/MihirMohapatra/fast-api-example

Drop a ⭐ if it helped you get started.

Building toward senior remote roles at US/EU companies. Follow along as I explore FastAPI, Rust, Go, and AI infrastructure — writing from the perspective of a practising backend engineer, not a tutorial blog.

From Learning to Shipping — Docker, Graceful Shutdown & ECS Fargate

mihir mohapatra — Thu, 25 Jun 2026 05:13:15 +0000

From Learning to Shipping — Docker, Graceful Shutdown & ECS Fargate

This is the final post in the series. In part 6 I covered testing — table-driven tests, httptest, benchmarks. Now I'm taking everything built across the last six posts and shipping it: a multi-stage Docker build, graceful shutdown wired into the Gin server, and deployment to AWS ECS Fargate using Terraform — the same stack I used for rust-ai-gateway.

If you've followed along, this is where the learning project becomes something you can actually point a recruiter at.

Multi-Stage Docker Build

The biggest Go deployment win over the JVM is binary size. A Go service compiles to a single self-contained binary with no runtime dependency — no JVM to install, no classpath to assemble. That means the production image can be built in two stages: a full Go toolchain image to compile, then a minimal scratch or distroless image to run.

# ---- build stage ----
FROM golang:1.22-alpine AS builder

WORKDIR /app

COPY go.mod go.sum ./
RUN go mod download

COPY . .

RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 \
    go build -ldflags="-s -w" -o orders-api ./cmd/main.go

# ---- run stage ----
FROM gcr.io/distroless/static-debian12

WORKDIR /app
COPY --from=builder /app/orders-api .

EXPOSE 8080
USER nonroot:nonroot

ENTRYPOINT ["/app/orders-api"]

Three things worth unpacking here. CGO_ENABLED=0 disables C bindings, which is required for a fully static binary that works in distroless. The -ldflags="-s -w" strip the debug symbol table and DWARF info, shaving 20–30% off the binary size for free. And distroless/static is the production choice over scratch — it has no shell (good for security) but includes timezone data and SSL certificates that your service will almost certainly need.

The resulting image typically comes in under 20MB. The equivalent Spring Boot fat jar with a JRE base image usually lands somewhere north of 300MB.

Graceful Shutdown

Out of the box, r.Run(":8080") in Gin blocks forever and stops immediately on SIGTERM — mid-request, no cleanup, no connection draining. In a containerised environment where ECS sends SIGTERM before terminating a task, that's dropped requests on every deployment. The fix is to run the server in a goroutine and listen for the OS signal yourself:

// cmd/main.go
package main

import (
    "context"
    "log"
    "net/http"
    "os"
    "os/signal"
    "syscall"
    "time"

    "orders-api/handler"
    "orders-api/middleware"
    "orders-api/store"

    "github.com/gin-gonic/gin"
)

func main() {
    s := store.NewInMemoryStore()
    h := handler.NewOrderHandler(s)

    r := gin.New()
    r.Use(middleware.Logger())

    api := r.Group("/api/v1")
    {
        orders := api.Group("/orders")
        orders.POST("", h.Create)
        orders.GET("", h.List)
        orders.GET("/:id", h.GetByID)
    }

    srv := &http.Server{
        Addr:    ":8080",
        Handler: r,
    }

    // start server in background goroutine
    go func() {
        log.Println("server listening on :8080")
        if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
            log.Fatalf("listen error: %v", err)
        }
    }()

    // block until SIGINT or SIGTERM
    quit := make(chan os.Signal, 1)
    signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM)
    <-quit

    log.Println("shutdown signal received, draining connections...")

    ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer cancel()

    if err := srv.Shutdown(ctx); err != nil {
        log.Fatalf("forced shutdown: %v", err)
    }

    log.Println("server exited cleanly")
}

srv.Shutdown(ctx) tells the HTTP server to stop accepting new connections and wait for in-flight requests to finish, up to the 30-second deadline. This is the goroutine and channel pattern from part 3 applied directly to real production code — quit is a channel, <-quit blocks until a signal arrives, and then the shutdown sequence runs. No framework magic, just the primitives we've been using all along.

Health Check Endpoint

ECS needs a health check to know when a task is ready for traffic and when it's gone unhealthy. Two lines in the router:

r.GET("/health", func(c *gin.Context) {
    c.JSON(http.StatusOK, gin.H{"status": "ok"})
})

r.GET("/ready", func(c *gin.Context) {
    // in a real service: check DB connectivity here
    c.JSON(http.StatusOK, gin.H{"status": "ready"})
})

/health is the liveness probe — is the process running. /ready is the readiness probe — is the service ready to serve traffic (database connected, caches warm). Keeping them separate lets ECS restart an unhealthy container without pulling it from the load balancer unnecessarily during a slow startup.

Terraform: ECS Fargate

The infrastructure follows the same pattern as rust-ai-gateway — ECR for the image, ECS Fargate for the task, an ALB in front. The key pieces:

# ecr.tf
resource "aws_ecr_repository" "orders_api" {
  name                 = "orders-api"
  image_tag_mutability = "MUTABLE"

  image_scanning_configuration {
    scan_on_push = true
  }
}

# ecs.tf
resource "aws_ecs_task_definition" "orders_api" {
  family                   = "orders-api"
  requires_compatibilities = ["FARGATE"]
  network_mode             = "awsvpc"
  cpu                      = "256"
  memory                   = "512"
  execution_role_arn       = aws_iam_role.ecs_task_execution.arn

  container_definitions = jsonencode([{
    name      = "orders-api"
    image     = "${aws_ecr_repository.orders_api.repository_url}:latest"
    essential = true

    portMappings = [{
      containerPort = 8080
      protocol      = "tcp"
    }]

    healthCheck = {
      command     = ["CMD-SHELL", "wget -qO- http://localhost:8080/health || exit 1"]
      interval    = 30
      timeout     = 5
      retries     = 3
      startPeriod = 10
    }

    logConfiguration = {
      logDriver = "awslogs"
      options = {
        "awslogs-group"         = "/ecs/orders-api"
        "awslogs-region"        = var.aws_region
        "awslogs-stream-prefix" = "ecs"
      }
    }
  }])
}

resource "aws_ecs_service" "orders_api" {
  name            = "orders-api"
  cluster         = aws_ecs_cluster.main.id
  task_definition = aws_ecs_task_definition.orders_api.arn
  desired_count   = 2
  launch_type     = "FARGATE"

  network_configuration {
    subnets          = var.private_subnet_ids
    security_groups  = [aws_security_group.ecs_tasks.id]
    assign_public_ip = false
  }

  load_balancer {
    target_group_arn = aws_lb_target_group.orders_api.arn
    container_name   = "orders-api"
    container_port   = 8080
  }

  deployment_minimum_healthy_percent = 100
  deployment_maximum_percent         = 200
}

desired_count = 2 with deployment_minimum_healthy_percent = 100 means ECS will never take both tasks down simultaneously during a deployment — new tasks must become healthy before old ones are stopped, which combined with the graceful shutdown above gives zero-downtime deploys.

CI/CD: GitHub Actions

The deploy pipeline that ties it together — build, push to ECR, force a new ECS deployment:

# .github/workflows/deploy.yml
name: Deploy

on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: ${{ secrets.AWS_REGION }}

      - name: Login to ECR
        id: login-ecr
        uses: aws-actions/amazon-ecr-login@v2

      - name: Build, tag and push image
        env:
          ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
          IMAGE_TAG: ${{ github.sha }}
        run: |
          docker build -t $ECR_REGISTRY/orders-api:$IMAGE_TAG .
          docker push $ECR_REGISTRY/orders-api:$IMAGE_TAG
          docker tag $ECR_REGISTRY/orders-api:$IMAGE_TAG $ECR_REGISTRY/orders-api:latest
          docker push $ECR_REGISTRY/orders-api:latest

      - name: Force ECS deployment
        run: |
          aws ecs update-service \
            --cluster main \
            --service orders-api \
            --force-new-deployment

Using github.sha as the image tag means every deploy is traceable — you can look at a running ECS task, find the image tag, and map it back to an exact commit. Tagging latest separately lets Terraform and the task definition reference a stable name, while the SHA tag is there for auditability.

What This Series Built

Seven posts ago I was explaining why I was picking up Go. Here's what actually got built:

A type system that composes instead of inheriting, with interfaces that match on shape rather than declaration
A concurrency model that puts goroutines and channels in reach by default
Error handling that makes failure visible at every call site
A Gin REST API with middleware, request validation, and clean handler separation
A full test suite: table-driven unit tests, httptest handler tests, and benchmarks
A production deployment: multi-stage Docker under 20MB, graceful shutdown, ECS Fargate behind an ALB, GitHub Actions CI/CD

The thing I didn't expect: Go's simplicity compounds. Every new piece — a new handler, a new test, a new middleware — follows the same small set of patterns. There's less to remember, and the code that comes out of it is easier to read six months later than Java or Kotlin at comparable complexity. I get why teams that care about maintainability and onboarding speed keep reaching for it.

Next on my list for this codebase: swapping the in-memory store for pgx and PostgreSQL, adding OpenTelemetry tracing, and wiring in a real secrets manager. That might become another series, or it might just end up in the GitHub repo.

If you've followed along from part 1 — thanks for reading. And if you're just landing here and want the whole project, the full source is at github.com/MihirMohapatra/go-orders-api once I push it.

What should the next series be? Another Go deep-dive, or something at the intersection of Go and AI infrastructure?

Testing in Go — Table-Driven Tests, Benchmarks, and go test Habits

mihir mohapatra — Tue, 23 Jun 2026 11:27:55 +0000

Testing in Go — Table-Driven Tests, Benchmarks, and go test Habits

In part 5 I built a working Orders API with Gin — routes, middleware, an in-memory store, and proper error handling. This time I'm writing tests for it, and the story here is one of the things I've genuinely liked about Go from the start: testing is a first-class citizen with zero framework ceremony. No JUnit, no Mockito, no Spring Boot test context to spin up. Just go test and the standard library.

The go test Mental Model

Every _test.go file in a Go package is compiled and run by go test. Test functions follow one convention — they take a single *testing.T argument and start with Test:

func TestSomething(t *testing.T) {
    // ...
}

That's the entire contract. No annotations, no base classes, no test runner configuration file. Running go test ./... from the project root picks up every test across every package recursively. Running with -v shows each test name and its pass/fail. Running with -race enables the data race detector we talked about in part 3. These three invocations cover 90% of what I reach for day to day:

go test ./...                  # run everything
go test -v ./handler/...       # verbose, one package
go test -race ./...            # race detector on
go test -run TestCreateOrder . # run one test by name

Table-Driven Tests: The Go Idiom

The single most important testing pattern in Go is the table-driven test. Instead of writing one test function per scenario, you define a slice of cases as a struct, loop over them, and let the test framework tell you exactly which case failed. Coming from JUnit's @ParameterizedTest, it's the same idea — except it's just a for loop, no framework required:

// store/order_test.go
package store_test

import (
    "testing"

    "orders-api/model"
    "orders-api/store"
)

func TestCreateOrder(t *testing.T) {
    cases := []struct {
        name      string
        input     model.CreateOrderRequest
        wantErr   bool
    }{
        {
            name:    "valid order",
            input:   model.CreateOrderRequest{Customer: "Alice", Amount: 99.99},
            wantErr: false,
        },
        {
            name:    "zero amount",
            input:   model.CreateOrderRequest{Customer: "Bob", Amount: 0},
            wantErr: false, // store doesn't validate, handler does
        },
        {
            name:    "empty customer",
            input:   model.CreateOrderRequest{Customer: "", Amount: 50.0},
            wantErr: false,
        },
    }

    for _, tc := range cases {
        t.Run(tc.name, func(t *testing.T) {
            s := store.NewInMemoryStore()
            order, err := s.Create(tc.input)

            if tc.wantErr && err == nil {
                t.Errorf("expected error, got nil")
            }
            if !tc.wantErr && err != nil {
                t.Errorf("unexpected error: %v", err)
            }
            if !tc.wantErr && order.ID == "" {
                t.Errorf("expected non-empty order ID, got empty string")
            }
        })
    }
}

t.Run creates a named sub-test for each case. When a sub-test fails, Go prints the full path — TestCreateOrder/zero_amount — so you know instantly which case broke without digging through output. This is the pattern the entire Go standard library uses internally, and it's the first habit worth locking in.

Testing the Store: ErrNotFound

The GetByID path needs its own table — specifically the not-found branch we wired into the handler in part 5:

func TestGetByID(t *testing.T) {
    s := store.NewInMemoryStore()

    // seed one order
    created, _ := s.Create(model.CreateOrderRequest{
        Customer: "Alice",
        Amount:   49.99,
    })

    cases := []struct {
        name    string
        id      string
        wantErr error
    }{
        {"existing order", created.ID, nil},
        {"missing order", "does-not-exist", store.ErrNotFound},
        {"empty id", "", store.ErrNotFound},
    }

    for _, tc := range cases {
        t.Run(tc.name, func(t *testing.T) {
            _, err := s.GetByID(tc.id)
            if !errors.Is(err, tc.wantErr) {
                t.Errorf("got err %v, want %v", err, tc.wantErr)
            }
        })
    }
}

errors.Is in the test assertion mirrors exactly how the handler checks errors — the sentinel defined in the store package is the contract, and we're verifying both sides of it honour it.

Handler Tests with httptest

Testing the handler layer without spinning up a real server is where Go's net/http/httptest package earns its place. It gives you a fake ResponseRecorder that captures everything the handler writes — status code, headers, body — without any network involvement:

// handler/order_test.go
package handler_test

import (
    "bytes"
    "encoding/json"
    "net/http"
    "net/http/httptest"
    "testing"

    "orders-api/handler"
    "orders-api/model"
    "orders-api/store"

    "github.com/gin-gonic/gin"
)

func setupRouter(h *handler.OrderHandler) *gin.Engine {
    gin.SetMode(gin.TestMode)
    r := gin.New()
    r.POST("/orders", h.Create)
    r.GET("/orders/:id", h.GetByID)
    r.GET("/orders", h.List)
    return r
}

func TestCreateOrderHandler(t *testing.T) {
    cases := []struct {
        name       string
        body       any
        wantStatus int
    }{
        {
            name:       "valid request",
            body:       model.CreateOrderRequest{Customer: "Alice", Amount: 99.99},
            wantStatus: http.StatusCreated,
        },
        {
            name:       "missing customer",
            body:       map[string]any{"amount": 49.99},
            wantStatus: http.StatusBadRequest,
        },
        {
            name:       "negative amount",
            body:       map[string]any{"customer": "Bob", "amount": -10},
            wantStatus: http.StatusBadRequest,
        },
    }

    for _, tc := range cases {
        t.Run(tc.name, func(t *testing.T) {
            s := store.NewInMemoryStore()
            h := handler.NewOrderHandler(s)
            r := setupRouter(h)

            bodyBytes, _ := json.Marshal(tc.body)
            req := httptest.NewRequest(http.MethodPost, "/orders", bytes.NewReader(bodyBytes))
            req.Header.Set("Content-Type", "application/json")
            w := httptest.NewRecorder()

            r.ServeHTTP(w, req)

            if w.Code != tc.wantStatus {
                t.Errorf("status: got %d, want %d — body: %s",
                    w.Code, tc.wantStatus, w.Body.String())
            }
        })
    }
}

gin.SetMode(gin.TestMode) suppresses the debug output Gin prints during tests. Each case builds a fresh store and handler so tests are fully isolated — no shared state leaking between cases. The w.Body.String() in the error message means a failing test prints the actual response body, which cuts debug time significantly.

Benchmarks: Built Right In

Benchmarks follow the same file and naming conventions as tests, but use *testing.B and prefix with Benchmark. The runner calls your function repeatedly, adjusting iteration count until the timing is stable:

func BenchmarkStoreList(b *testing.B) {
    s := store.NewInMemoryStore()

    // seed 1000 orders
    for i := 0; i < 1000; i++ {
        s.Create(model.CreateOrderRequest{
            Customer: fmt.Sprintf("customer-%d", i),
            Amount:   float64(i) * 1.5,
        })
    }

    b.ResetTimer() // don't count seeding time

    for i := 0; i < b.N; i++ {
        _, err := s.List()
        if err != nil {
            b.Fatal(err)
        }
    }
}

Running go test -bench=. -benchmem ./store/... gives you nanoseconds per operation and allocations per operation:

BenchmarkStoreList-8    42361    28204 ns/op    24576 B/op    3 allocs/op

b.ResetTimer() after the seeding step is the habit that matters most here — you want to measure List, not Create × 1000. The -benchmem flag surfacing allocations per operation is the other one: in a hot path, allocation count matters as much as raw speed.

The go test Habits Worth Locking In

After a few weeks of writing Go tests, these are the ones I've made automatic:

Always use t.Run for multi-case scenarios. Even two cases. The named sub-test output pays for the extra line immediately when something breaks.

t.Parallel() for independent tests. Adding t.Parallel() at the top of a test (or sub-test) tells the runner it can execute concurrently with other parallel tests, cutting total test time on multi-core machines with no other changes.

-race in CI, always. Data races in Go are silent in normal test runs and catastrophic in production. The race detector adds overhead but catches things that would otherwise survive code review and only surface under load.

testify/assert if your team agrees, testing if they don't. The standard testing package requires writing your own error messages for every assertion. testify/assert gives you assert.Equal, assert.NoError, and friends with good default messages. Both are valid choices — the important thing is consistency within a codebase.

Up Next

Part 7 is the finish line: multi-stage Docker build, graceful shutdown, and getting this service onto ECS Fargate — the same deployment target I used for rust-ai-gateway. The testing patterns from this post will feed directly into the CI pipeline we wire up there.

Do you have a preferred Go testing library — pure testing, testify, or something else? Curious whether the standard library alone is enough once the projects get bigger.

Building My First Real API in Go — with Gin

mihir mohapatra — Mon, 22 Jun 2026 07:40:08 +0000

Building My First Real API in Go — with Gin

In part 4 I worked through Go's error handling model and came around on if err != nil. This is the post where everything from the last four parts — structs, interfaces, goroutines, error wrapping — stops being theoretical and actually gets wired into something that runs, accepts requests, and talks to a database.

I'm building a small Orders API: create an order, fetch one by ID, list all of them. Simple enough that the routing and middleware patterns are clear, complex enough that the error handling and concurrency patterns from earlier posts actually matter.

Why Gin

The Go standard library's net/http package is genuinely capable — you can write a production API with nothing else. But Gin adds three things that make it worth reaching for immediately: a clean router with path parameters and route groups, a middleware chain that composes nicely, and response helpers (c.JSON, c.AbortWithStatusJSON) that remove a lot of boilerplate. It's the closest thing Go has to a lightweight Express or Axum — fast, minimal, and doesn't try to own your entire architecture.

go mod init orders-api
go get github.com/gin-gonic/gin

Project Structure

Before a single line of code, here's the layout I settled on:

orders-api/
├── main.go
├── handler/
│   └── order.go
├── model/
│   └── order.go
├── store/
│   └── order.go
└── middleware/
    └── logger.go

This is the standard Go flat-package layout — no src/, no deep nesting, no framework-imposed folders. handler owns HTTP concerns, model owns data shapes, store owns persistence logic. The separation means each layer depends on the one below it, and none of them import each other in circles.

The Model

// model/order.go
package model

import "time"

type Order struct {
    ID        string    `json:"id"`
    Customer  string    `json:"customer"`
    Amount    float64   `json:"amount"`
    Status    string    `json:"status"`
    CreatedAt time.Time `json:"created_at"`
}

type CreateOrderRequest struct {
    Customer string  `json:"customer" binding:"required"`
    Amount   float64 `json:"amount"   binding:"required,gt=0"`
}

Two things worth noting here. The struct tags (json:"id") control serialisation — without them Gin would use the field name as-is, so CreatedAt would become CreatedAt in JSON rather than created_at. The binding tags on CreateOrderRequest are Gin's built-in validation: binding:"required" rejects blank fields, gt=0 rejects zero or negative amounts. This is the closest Go gets to Java Bean Validation annotations, except it's enforced at the handler layer rather than the model layer — I'll show how in a moment.

The Store

For now the store is in-memory — a map behind a sync.RWMutex. Part 7 will swap this for pgx. The interface is the important part:

// store/order.go
package store

import (
    "errors"
    "sync"
    "time"

    "orders-api/model"

    "github.com/google/uuid"
)

var ErrNotFound = errors.New("order not found")

type OrderStore interface {
    Create(req model.CreateOrderRequest) (model.Order, error)
    GetByID(id string) (model.Order, error)
    List() ([]model.Order, error)
}

type inMemoryStore struct {
    mu     sync.RWMutex
    orders map[string]model.Order
}

func NewInMemoryStore() OrderStore {
    return &inMemoryStore{orders: make(map[string]model.Order)}
}

func (s *inMemoryStore) Create(req model.CreateOrderRequest) (model.Order, error) {
    order := model.Order{
        ID:        uuid.New().String(),
        Customer:  req.Customer,
        Amount:    req.Amount,
        Status:    "pending",
        CreatedAt: time.Now(),
    }
    s.mu.Lock()
    s.orders[order.ID] = order
    s.mu.Unlock()
    return order, nil
}

func (s *inMemoryStore) GetByID(id string) (model.Order, error) {
    s.mu.RLock()
    defer s.mu.RUnlock()
    order, ok := s.orders[id]
    if !ok {
        return model.Order{}, ErrNotFound
    }
    return order, nil
}

func (s *inMemoryStore) List() ([]model.Order, error) {
    s.mu.RLock()
    defer s.mu.RUnlock()
    orders := make([]model.Order, 0, len(s.orders))
    for _, o := range s.orders {
        orders = append(orders, o)
    }
    return orders, nil
}

sync.RWMutex lets multiple concurrent reads proceed at the same time but serialises writes — the right primitive for a read-heavy in-memory store. The OrderStore interface is defined here, in the store package, but that's just for organisation — the handler depends on the interface, not the concrete type, which means swapping in a real database later is a one-line change in main.go.

The Handler

// handler/order.go
package handler

import (
    "errors"
    "net/http"

    "orders-api/model"
    "orders-api/store"

    "github.com/gin-gonic/gin"
)

type OrderHandler struct {
    store store.OrderStore
}

func NewOrderHandler(s store.OrderStore) *OrderHandler {
    return &OrderHandler{store: s}
}

func (h *OrderHandler) Create(c *gin.Context) {
    var req model.CreateOrderRequest
    if err := c.ShouldBindJSON(&req); err != nil {
        c.AbortWithStatusJSON(http.StatusBadRequest, gin.H{"error": err.Error()})
        return
    }

    order, err := h.store.Create(req)
    if err != nil {
        c.AbortWithStatusJSON(http.StatusInternalServerError, gin.H{"error": "could not create order"})
        return
    }

    c.JSON(http.StatusCreated, order)
}

func (h *OrderHandler) GetByID(c *gin.Context) {
    id := c.Param("id")

    order, err := h.store.GetByID(id)
    if err != nil {
        if errors.Is(err, store.ErrNotFound) {
            c.AbortWithStatusJSON(http.StatusNotFound, gin.H{"error": "order not found"})
            return
        }
        c.AbortWithStatusJSON(http.StatusInternalServerError, gin.H{"error": "could not fetch order"})
        return
    }

    c.JSON(http.StatusOK, order)
}

func (h *OrderHandler) List(c *gin.Context) {
    orders, err := h.store.List()
    if err != nil {
        c.AbortWithStatusJSON(http.StatusInternalServerError, gin.H{"error": "could not list orders"})
        return
    }
    c.JSON(http.StatusOK, orders)
}

ShouldBindJSON is where the binding tags from the model earn their keep — it validates and decodes the request body in one call, returning a descriptive error if anything fails. errors.Is(err, store.ErrNotFound) is the same pattern from part 4: unwrap through layers and match the sentinel. The handler doesn't know or care whether the store is in-memory or PostgreSQL — it just speaks to the interface.

Middleware: Request Logging

// middleware/logger.go
package middleware

import (
    "fmt"
    "time"

    "github.com/gin-gonic/gin"
)

func Logger() gin.HandlerFunc {
    return func(c *gin.Context) {
        start := time.Now()
        c.Next()
        fmt.Printf("[%s] %s %s %d (%s)\n",
            start.Format("15:04:05"),
            c.Request.Method,
            c.Request.URL.Path,
            c.Writer.Status(),
            time.Since(start),
        )
    }
}

c.Next() hands control to the next handler in the chain, then execution returns here after the response is written — which is why time.Since(start) gives you the actual request duration. This is the middleware pattern in every Gin-based service: wrap, call Next(), do something with what happened after.

Wiring It Together

// main.go
package main

import (
    "orders-api/handler"
    "orders-api/middleware"
    "orders-api/store"

    "github.com/gin-gonic/gin"
)

func main() {
    s := store.NewInMemoryStore()
    h := handler.NewOrderHandler(s)

    r := gin.New()
    r.Use(middleware.Logger())

    api := r.Group("/api/v1")
    {
        orders := api.Group("/orders")
        orders.POST("", h.Create)
        orders.GET("", h.List)
        orders.GET("/:id", h.GetByID)
    }

    r.Run(":8080")
}

gin.New() instead of gin.Default() gives a blank engine — no Gin's built-in logger or recovery middleware added automatically, since we're providing our own. The r.Group nesting gives every orders route the /api/v1/orders prefix without repeating it on each registration. Running this and hitting POST /api/v1/orders with a JSON body gets you a real response, with validation, error handling, and request logging, in about 120 lines across all files.

What This Draws On From Earlier Posts

Part 2's interfaces: OrderStore is exactly the kind of single-responsibility interface Go favours — three methods, defined by the consumer (the handler), satisfied implicitly by the concrete store.

Part 3's concurrency: sync.RWMutex in the store is the in-memory safe concurrency answer. In part 7 when this becomes a real database, the connection pool handles this instead.

Part 4's error handling: errors.Is(err, store.ErrNotFound) translating a store-layer sentinel into an HTTP 404 is the wrapping pattern paying off at the HTTP boundary.

Up Next

Part 6 is testing — table-driven tests, the httptest package for handler tests without spinning up a real server, and the benchmarking habits that will matter once we push toward production in part 7.

What's your preferred approach to structuring Go API projects — flat packages like this, or a domain-driven layout with more nesting? Genuinely curious how people land on this one.

Error Handling — Learning to Love `if err != nil`

mihir mohapatra — Sun, 21 Jun 2026 09:48:23 +0000

Error Handling — Learning to Love `if err != nil`

In part 3 I covered goroutines and channels, and how Go's concurrency model sidesteps a lot of the ceremony I was used to from the JVM. This time I'm tackling the thing I complained about in part 1 of this series before I'd even really tried it: error handling. I called if err != nil repetitive back then. A few weeks and a lot of real code later, I owe Go a partial apology.

No Exceptions, On Purpose

Coming from Java, the absence of try/catch is the first thing that feels like a missing feature. It isn't — it's a deliberate design choice. In Go, errors are just values. A function that can fail returns an error as its last return value, and the caller decides what to do with it, right there, inline:

func divide(a, b float64) (float64, error) {
    if b == 0 {
        return 0, errors.New("division by zero")
    }
    return a / b, nil
}

func main() {
    result, err := divide(10, 0)
    if err != nil {
        fmt.Println("error:", err)
        return
    }
    fmt.Println("result:", result)
}

That's the pattern you'll write hundreds of times in Go: call a function, check err, handle it or bail out, move on. No hidden control flow jumping up the call stack to whichever catch block happens to match. No checked-exception signatures cluttering method declarations. No RuntimeException quietly skipping past five layers of code that had no idea it could happen. Whatever can fail is sitting right there in the function signature, and you're forced to look at it.

Why the Repetition Is the Point

My part 1 complaint was that if err != nil everywhere feels manual. It is manual — and that's exactly the trade Go is making. In Java, an exception thrown deep in a call stack can silently propagate through layers of code that never declared they might fail, and you only find out where things actually break by reading a stack trace after the fact. In Go, every single point where something can go wrong is visible in the source, in order, as you read top to bottom:

func processOrder(id string) error {
    order, err := fetchOrder(id)
    if err != nil {
        return fmt.Errorf("fetching order: %w", err)
    }

    if err := validateOrder(order); err != nil {
        return fmt.Errorf("validating order: %w", err)
    }

    if err := chargePayment(order); err != nil {
        return fmt.Errorf("charging payment: %w", err)
    }

    return nil
}

Three things can fail here, and you can see all three without jumping anywhere. That %w verb in fmt.Errorf is doing something specific — it's wrapping the original error rather than discarding it, which means the caller can still inspect what actually went wrong underneath your added context. It's the closest thing Go has to a stack trace, except you control exactly how much context gets added at each layer instead of getting a wall of frames you have to parse yourself.

errors.Is and errors.As: Inspecting Wrapped Errors

Once you're wrapping errors as they bubble up, you need a way to check what's actually inside the wrapping. That's what errors.Is and errors.As are for:

var ErrNotFound = errors.New("order not found")

func fetchOrder(id string) (Order, error) {
    if id == "" {
        return Order{}, ErrNotFound
    }
    return Order{ID: id}, nil
}

func main() {
    err := processOrder("")
    if errors.Is(err, ErrNotFound) {
        fmt.Println("order doesn't exist, returning 404")
        return
    }
    fmt.Println("unexpected error:", err)
}

Even though processOrder wrapped the original error with fmt.Errorf("fetching order: %w", err), errors.Is can still see through the wrapping and match it against the sentinel ErrNotFound. This is the Go equivalent of catch (OrderNotFoundException e) — except instead of a type hierarchy of custom exception classes, you define a handful of sentinel error values (or custom error types, when you need to carry extra data) and compare against them explicitly wherever the distinction matters.

errors.As does the same thing but for pulling out a specific error type rather than matching a specific value — useful when you've defined a custom struct that implements the error interface and carries extra fields:

type ValidationError struct {
    Field string
    Msg   string
}

func (e *ValidationError) Error() string {
    return fmt.Sprintf("%s: %s", e.Field, e.Msg)
}

func main() {
    err := processOrder("123")
    var valErr *ValidationError
    if errors.As(err, &valErr) {
        fmt.Println("invalid field:", valErr.Field)
    }
}

Any type with an Error() string method satisfies the error interface — same implicit-satisfaction rule from part 2, just applied to errors specifically.

panic and recover: Not Your Exception Mechanism

Go does have panic and recover, and the instinct coming from Java is to treat them like throw and catch. That's a mistake that took me a minute to unlearn. panic is reserved for genuinely unrecoverable situations — a programming bug, a nil pointer you should never have dereferenced, an invariant that's been violated so badly that continuing execution would be worse than crashing:

func mustParseConfig(raw string) Config {
    cfg, err := parseConfig(raw)
    if err != nil {
        panic(fmt.Sprintf("invalid config, cannot start: %v", err))
    }
    return cfg
}

Notice the must prefix — that's a real Go convention signaling "this panics instead of returning an error, use it only when failure should be impossible or fatal." You'll see it on things like regexp.MustCompile, used for regex patterns that are hardcoded at compile time and should never actually fail to parse. If a function can fail in a way the caller might reasonably want to handle, it returns an error. panic is for "this program cannot continue," not "this operation didn't work."

Where I've Landed

A few weeks in, the repetition stopped bothering me, for a specific reason: I always know exactly where my code can fail, just by reading it. No exception is escaping from three function calls deep and surfacing somewhere I didn't expect. The trade-off is real — Go code has more visible lines dedicated to error checking than the equivalent Java would — but every one of those lines is telling me something true about the function I'm calling, instead of trusting that a try/catch block somewhere upstream will catch what I forgot to handle.

What still isn't fully natural yet: remembering to wrap errors with enough context every single time (it's easy to return a bare err and lose the "where did this happen" information), and resisting the urge to define a sprawling hierarchy of custom error types out of old OOP habits when a couple of sentinel errors would do.

Up Next

With types, concurrency, and error handling covered, part 5 is where this series gets concrete: building a real REST API with Gin, including the error-handling and concurrency patterns from the last two posts actually wired into request handlers.

If you've made the jump from exceptions to Go's explicit error returns, did the repetition bother you at first the way it bothered me — and did it fade the same way?

Goroutines & Channels — Concurrency Without the JVM's Baggage

mihir mohapatra — Sat, 20 Jun 2026 08:44:14 +0000

Goroutines & Channels — Concurrency Without the JVM's Baggage

In part 2 I went through Go's type system — structs, implicit interfaces, and composition replacing inheritance entirely. This time I'm tackling the thing that actually pulled me toward Go in the first place: concurrency.

If you've spent years on the JVM like I have, "concurrency" usually means a mental checklist — ExecutorService, thread pool sizing, Future<T>, maybe CompletableFuture chains, and more recently virtual threads if you're on a modern JDK. It's powerful, but it's also a lot of ceremony for what's often a simple idea: "do these things at the same time, then collect the results." Go strips almost all of that ceremony away, and that's been the most genuinely refreshing part of this whole learning process.

Goroutines: Threads, But Cheap

A goroutine is Go's unit of concurrent execution. The syntax is almost suspiciously simple — you put go in front of a function call, and it runs concurrently:

func main() {
    go sayHello("world")
    time.Sleep(50 * time.Millisecond) // give it time to run
}

func sayHello(name string) {
    fmt.Println("Hello,", name)
}

That time.Sleep hack is the first lesson everyone learns the hard way: main doesn't wait for goroutines to finish. If main exits, every goroutine it spawned just gets cut off mid-flight. We'll fix that properly in a moment with sync.WaitGroup, but the sleep hack is worth seeing once because it makes the "goroutines don't block their caller" behavior concrete.

What makes goroutines genuinely different from JVM threads isn't the syntax — it's the cost. A JVM thread typically reserves around a megabyte of stack space and is managed by the OS scheduler. A goroutine starts with a stack of about 2KB that grows as needed, and Go's own runtime scheduler multiplexes thousands of them onto a small number of OS threads. Spinning up 100,000 goroutines is a normal thing to do in Go. Spinning up 100,000 JVM platform threads will bring most systems to their knees — which is exactly the problem virtual threads were introduced to solve in recent JDKs. Go just had this answer from day one.

WaitGroups: The Right Way to Wait

Instead of sleeping and hoping, sync.WaitGroup lets you wait for a known number of goroutines to actually finish:

func main() {
    var wg sync.WaitGroup

    services := []string{"orders", "payments", "inventory"}

    for _, name := range services {
        wg.Add(1)
        go func(s string) {
            defer wg.Done()
            fetchFromService(s)
        }(name)
    }

    wg.Wait()
    fmt.Println("all services responded")
}

func fetchFromService(name string) {
    time.Sleep(100 * time.Millisecond)
    fmt.Println(name, "done")
}

wg.Add(1) registers one goroutine to wait for, defer wg.Done() marks it complete when the function returns, and wg.Wait() blocks until the count hits zero. Notice the name is passed as a parameter into the closure rather than captured directly — that's intentional. Capturing the loop variable directly is a classic Go footgun (pre-1.22, every closure would share the same variable and you'd get the last value for everything). Passing it as an argument sidesteps that entirely, and is still good practice to know even on newer Go versions where the loop variable semantics changed.

Channels: Goroutines Talking to Each Other

WaitGroups tell you when things are done. Channels let goroutines actually communicate — passing data between them safely, without manual locks:

func main() {
    results := make(chan string, 3)

    go fetchAndSend("orders", results)
    go fetchAndSend("payments", results)
    go fetchAndSend("inventory", results)

    for i := 0; i < 3; i++ {
        fmt.Println(<-results)
    }
    close(results)
}

func fetchAndSend(name string, results chan<- string) {
    time.Sleep(100 * time.Millisecond)
    results <- fmt.Sprintf("%s: 200 OK", name)
}

A channel is a typed pipe. results <- value sends, <-results receives, and both block until someone's on the other end (unless the channel is buffered, like the make(chan string, 3) above, which allows up to 3 sends before blocking). The chan<- string parameter type is a send-only channel — a nice bit of compiler-enforced documentation saying "this function only writes here, it never reads."

This is the part that reframes concurrency entirely coming from Java. The Go proverb is "don't communicate by sharing memory, share memory by communicating" — instead of multiple threads fighting over a synchronized block or a ConcurrentHashMap, you pass ownership of data through a channel, one direction at a time. There's no lock to forget, because there's nothing shared to protect.

select: Handling Multiple Channels at Once

Once you have more than one channel, select lets you react to whichever one is ready first — it's the concurrency equivalent of a switch statement:

func main() {
    payments := make(chan string)
    timeout := time.After(2 * time.Second)

    go processPayment(payments)

    select {
    case result := <-payments:
        fmt.Println("payment result:", result)
    case <-timeout:
        fmt.Println("payment timed out")
    }
}

func processPayment(payments chan<- string) {
    time.Sleep(3 * time.Second) // simulate a slow downstream call
    payments <- "success"
}

This pattern — racing a real channel against time.After — is how Go handles timeouts idiomatically. No Future.get(timeout, unit) with a checked TimeoutException to catch. Just two channels and a select.

Where This Actually Bites You

I don't want to oversell this as friction-free, because it isn't. A few things I've already tripped over:

Goroutine leaks are real and easy to cause. If a goroutine is blocked sending on a channel that nobody's ever going to read from again, it just sits there forever, leaking memory. Unlike a thread pool with bounded size, there's no natural backpressure unless you design it in yourself — usually with buffered channels, context.Context cancellation, or both.

Deadlocks look different than JVM deadlocks. Instead of two threads each holding a lock the other wants, a Go deadlock is more often "every goroutine is blocked waiting on a channel, and nothing's left to unblock them." The runtime is actually good at detecting this for the simplest cases (fatal error: all goroutines are asleep - deadlock!), which is more help than the JVM ever gave me.

Unbuffered channels make ordering subtle. An unbuffered channel send blocks until a receiver is ready, which means the timing of goroutines relative to each other isn't always obvious from reading the code top to bottom. It took a few go run experiments with print statements before the mental model clicked.

go vet and the race detector are not optional. Running go run -race during development catches data races that would otherwise be silent and only show up under load in production. I now run it by default the way I'd run a linter.

Why This Matters for the Series

This concurrency model is going to show up constantly once I build a real API in part 5 — handling concurrent requests, fanning out to multiple downstream services, and managing timeouts cleanly are exactly what channels and select are built for. It's also a big part of why Go shows up so often in infra and backend tooling: the concurrency primitives are cheap enough that you reach for them by default, not just when you've hit a performance wall.

Next up in part 4: error handling. I promised in part 1 that if err != nil would get its own post, and after spending real time with goroutines that return errors over channels, I have a lot more sympathy for Go's approach than I expected to.

If you've worked with both JVM threads and goroutines, I'd like to know — did the "share memory by communicating" mental shift come naturally, or did you find yourself reaching for locks out of habit at first?