DEV Community: mihir mohapatra

AI & LLM Terminology Glossary: From Tokens to Orchestration

mihir mohapatra — Mon, 20 Jul 2026 08:33:49 +0000

Every field accumulates jargon, and AI/LLM engineering has accumulated it faster than most — partly because the field is young, and partly because terms get reused across research papers, vendor marketing, and production engineering with slightly different meanings each time.

This is the reference I wish existed when I started building AI infrastructure: one place with the terms that actually come up in day-to-day work, grouped by where you'll encounter them, explained at the level a backend engineer needs — not a research-paper level, and not a marketing-page level.

Core model concepts

Token — The basic unit a language model reads and generates. Not quite a word, not quite a character — usually a sub-word chunk (e.g. "tokenization" might split into "token" + "ization"). Every cost, context limit, and latency figure you'll deal with is measured in tokens, not words or characters.

Context window — The maximum number of tokens (input + output combined) a model can attend to in a single request. Everything the model "knows" about the current conversation has to fit inside this window — anything older gets truncated or has to be summarized.

Inference — The act of running a trained model to produce an output, as opposed to training it. When people talk about "inference cost" or "inference latency," they mean the cost/speed of using an already-trained model, not building one.

Parameters (weights) — The learned numerical values inside a model that encode what it "knows." Model size is usually described by parameter count (e.g. "a 7B model" = 7 billion parameters). More parameters generally means more capability, but also more memory and compute to run.

Fine-tuning — Continuing to train an already-trained (pretrained) model on a narrower, task-specific dataset, so it specializes without starting from scratch.

LoRA (Low-Rank Adaptation) — A fine-tuning technique that freezes the original model weights and trains a small set of additional low-rank matrices instead. It gets most of the benefit of full fine-tuning at a fraction of the compute and memory cost, and it's why you'll see so many "fine-tuned" open-weight variants floating around — full fine-tuning is expensive, LoRA isn't.

Quantization — Reducing the numerical precision used to store model weights (or the KV cache — see below) — for example fp16 down to int8 or int4. This shrinks memory footprint and often improves throughput, at some cost to output quality depending on how aggressive the quantization is.

Attention mechanism terms

Attention — The mechanism that lets a model weigh how relevant each previous token is when producing the next one. It's the core operation that makes transformers work, and it's also the most expensive part computationally.

KV cache (Key/Value cache) — Storage of the key and value tensors computed for previous tokens, so the model doesn't have to recompute them for every new token generated. It's usually the actual memory bottleneck in production serving — not the model weights — because it grows with both sequence length and number of concurrent requests.

MHA (Multi-Head Attention) — The original attention design, where every query head has its own dedicated set of key/value heads. Best quality, but the most expensive KV cache.

GQA (Grouped-Query Attention) — Multiple query heads share a smaller set of key/value heads, shrinking the KV cache significantly (often 4–8x) with minimal quality loss. Most modern open-weight models use this by default.

MQA (Multi-Query Attention) — The extreme version of GQA: all query heads share a single key/value head. Smallest possible cache, but a larger quality tradeoff than GQA.

Prefix caching — Reusing the KV cache for a shared prompt prefix (like a system prompt or few-shot examples) across multiple requests, instead of recomputing and re-storing it every time. Valuable for agent and chat workloads where the same instructions repeat across many calls.

PagedAttention — The memory management technique (popularized by vLLM) that treats the KV cache like OS-style virtual memory pages instead of requiring one contiguous block per request. This is what makes prefix caching and high-concurrency serving practical without the cache fragmenting into unusable gaps.

Retrieval & knowledge terms

Embedding — A numerical vector representation of text (or images, audio, etc.) that captures semantic meaning, such that similar content ends up with similar vectors. This is the foundation of search and retrieval in AI systems.

Vector database — A database optimized for storing embeddings and doing fast similarity search over them (e.g. "find the 5 most similar chunks to this query"). Examples include Pinecone, Weaviate, Qdrant, and pgvector as a Postgres extension.

RAG (Retrieval-Augmented Generation) — An architecture where the model's response is grounded by first retrieving relevant documents/chunks (usually via a vector database) and injecting them into the prompt, rather than relying purely on what the model memorized during training. This is how you get a model to answer questions about your own private documents.

Chunking — Splitting source documents into smaller pieces before embedding them, since embedding an entire document as one vector loses too much granularity for precise retrieval. Chunk size and overlap strategy directly affect RAG retrieval quality.

Reranking — A second-pass step in RAG pipelines where an initial broad set of retrieved candidates gets re-scored by a more precise (often more expensive) model before the final few are used, improving precision over similarity search alone.

Agentic system terms

Agent — A system where a model doesn't just respond once, but runs in a loop: reasoning, calling tools, observing results, and deciding what to do next until the task is complete.

Tool use / function calling — The mechanism by which a model can invoke external functions (search the web, query a database, run code) as part of its reasoning process, rather than being limited to generating text.

MCP (Model Context Protocol) — A standardized protocol for connecting models to external tools and data sources, so tool integrations aren't reinvented per-framework. Increasingly the common interface across agent frameworks and IDEs.

Orchestration — Coordinating multiple agents working together on a task — a central coordinator decomposing work and dispatching it to specialized worker agents, as opposed to one agent trying to do everything.

Handoff — A pattern where one agent, mid-task, transfers control (and relevant context) to a different, better-suited agent — common in support/triage-style systems.

Production & serving terms

Latency — Time taken to get a response. Often split into time to first token (TTFT) — how long before generation starts — and inter-token latency — how long between subsequent tokens once generation is underway. These respond to different optimizations.

Throughput — How many requests (or tokens) a system can process per unit time, as opposed to how fast any single request completes. Batching improves throughput, sometimes at the cost of individual request latency.

Continuous batching — A serving technique where new requests can join an in-progress batch as soon as a slot frees up, rather than waiting for the whole batch to finish. This is a major reason serving frameworks like vLLM outperform naive request-at-a-time loops.

Tracing — Recording the full path of a request through a system — including retries, tool calls, and fallbacks — as a structured, queryable record, rather than relying on scattered logs. Essential once an LLM app has more than one moving part.

Eval / eval loop — A systematic, often continuous process for scoring model output quality — structural checks (did the JSON parse), golden-set regression tests, or sampled judge-model scoring — used to catch quality regressions the way tests catch code regressions.

Cost & efficiency terms

Prompt / completion tokens — Most API pricing is split into input (prompt) tokens and output (completion) tokens, usually priced differently, with output tokens typically costing more per token.

Distillation — Training a smaller "student" model to mimic the behavior of a larger "teacher" model, producing something cheaper to run that retains much of the larger model's capability on the tasks it was distilled for.

Open-weight vs. closed model — Open-weight models have downloadable weights you can self-host and run under your own infrastructure (e.g. Llama, Mistral, Qwen); closed models are only accessible through a provider's hosted API (e.g. Claude, GPT). This distinction drives a real cost/control tradeoff, not just a licensing preference.

Why the vocabulary matters

None of these terms are difficult individually — the friction is that they get used loosely across blog posts, vendor docs, and papers, often without definition, on the assumption you already know them. Having a shared, precise vocabulary is what makes the difference between skimming an infra discussion and actually being able to make the tradeoff decisions it's describing — which quantization level, which attention variant, whether a task actually needs orchestration or just a well-scoped single agent.

This glossary is part of a series on AI infrastructure patterns. If a term you've run into isn't here, drop it in the comments and I'll add it.

Agentic Systems: From Single Agent to Orchestration

mihir mohapatra — Thu, 16 Jul 2026 09:11:30 +0000

A single agent — one model, a system prompt, a handful of tools, and a loop — gets you surprisingly far. It's also where most people stop, because it works for demos and falls over the moment a task needs more than one kind of reasoning happening at once.

Multi-agent orchestration isn't a hype upgrade over a single agent — it's what you reach for when a task naturally decomposes into sub-tasks that benefit from different context, different tools, or genuinely different roles. This post covers when that shift is warranted, the orchestration patterns actually used in production, and a small working example of a coordinator delegating to worker agents.

Why a single agent stops being enough

A single agent's context window and tool list grow with every capability you bolt on. Three failure modes show up as that happens:

Context pollution — a research task's intermediate scratch work bleeds into the reasoning for an unrelated coding sub-task, degrading both
Tool overload — past a certain number of tools in one system prompt, selection accuracy drops; the model spends more of its reasoning deciding which tool than actually using it well
No separation of concerns — a single agent can't easily hold "be a careful planner" and "be a fast, narrow executor" as two different behavioral modes at once

Orchestration solves this by splitting one large, blurry job into a coordinator plus specialized workers, each with a narrow role, its own context, and only the tools it actually needs.

The patterns that show up in practice

There isn't one "multi-agent architecture" — a few concrete patterns cover most real systems:

Sequential (pipeline) — agent A's output feeds agent B, feeds agent C. Good for linear workflows like research → draft → review.
Concurrent (fan-out/fan-in) — the coordinator dispatches the same or related sub-tasks to multiple agents in parallel, then merges results. Good for independent sub-tasks — e.g. summarizing five documents at once.
Handoff — one agent decides mid-task that another agent is better suited and transfers control (and context) to it. Good for triage-style systems, like a support bot handing off to a specialist agent.
Hierarchical (coordinator/worker) — a central planner decomposes the task and delegates to specialized workers, then assembles their outputs into a final answer. This is the most common production pattern and the one the example below uses.

Frameworks like LangGraph model this as an explicit state graph, CrewAI models it as role-based "crews," and minimal SDKs like the OpenAI Agents SDK treat sub-agents as callable tools from a parent agent. The underlying pattern is the same regardless of framework — what differs is how much structure and observability you get by default.

A working example: coordinator + worker agents in Rust

Here's a minimal hierarchical setup: a coordinator agent receives a research request, decomposes it into sub-tasks, dispatches them to worker agents concurrently, and merges the results.

use futures::future::join_all;
use serde::{Deserialize, Serialize};

#[derive(Debug, Clone, Serialize, Deserialize)]
struct SubTask {
    id: String,
    description: String,
    assigned_role: AgentRole,
}

#[derive(Debug, Clone, Serialize, Deserialize)]
enum AgentRole {
    Researcher,
    CodeAnalyst,
    Summarizer,
}

#[derive(Debug, Serialize)]
struct WorkerResult {
    task_id: String,
    output: String,
    tokens_used: u32,
}

struct Coordinator {
    llm_client: LlmClient,
}

impl Coordinator {
    /// Decompose a high-level request into sub-tasks using the coordinator's own LLM call.
    async fn plan(&self, request: &str) -> Vec<SubTask> {
        let planning_prompt = format!(
            "Break this request into 2-4 independent sub-tasks, each assigned \
             to Researcher, CodeAnalyst, or Summarizer. Request: {request}"
        );
        let response = self.llm_client.complete(&planning_prompt).await;
        parse_subtasks(&response) // structured output parsing, omitted for brevity
    }

    /// Dispatch sub-tasks to worker agents concurrently, then merge.
    async fn execute(&self, request: &str) -> String {
        let subtasks = self.plan(request).await;

        let worker_futures = subtasks.into_iter().map(|task| {
            let client = self.llm_client.clone();
            async move { run_worker(client, task).await }
        });

        let results: Vec<WorkerResult> = join_all(worker_futures).await;

        self.merge(results).await
    }

    /// A final LLM call that synthesizes worker outputs into one coherent answer.
    async fn merge(&self, results: Vec<WorkerResult>) -> String {
        let combined = results.iter()
            .map(|r| format!("[{}]: {}", r.task_id, r.output))
            .collect::<Vec<_>>()
            .join("\n\n");

        let merge_prompt = format!(
            "Synthesize these sub-task results into one coherent response:\n\n{combined}"
        );
        self.llm_client.complete(&merge_prompt).await
    }
}

async fn run_worker(client: LlmClient, task: SubTask) -> WorkerResult {
    let role_prompt = match task.assigned_role {
        AgentRole::Researcher => "You are a focused researcher. Be concise and cite sources.",
        AgentRole::CodeAnalyst => "You are a code analyst. Focus only on technical accuracy.",
        AgentRole::Summarizer => "You are a summarizer. Compress without losing key facts.",
    };

    let prompt = format!("{role_prompt}\n\nTask: {}", task.description);
    let output = client.complete(&prompt).await;

    WorkerResult {
        task_id: task.id,
        output,
        tokens_used: estimate_tokens(&output),
    }
}

A few things worth calling out in this shape:

Each worker gets its own narrow system prompt (role_prompt) instead of sharing the coordinator's full context — this is the actual mechanism that fixes context pollution.
join_all runs workers concurrently, not sequentially — for independent sub-tasks, this is a straightforward latency win that a single-agent loop can't get you.
The merge step is itself an LLM call, not string concatenation — worker outputs are often redundant or need reconciling, and skipping this step is a common source of disjointed final answers.

Where this pattern breaks down

Multi-agent orchestration isn't free — it's worth naming the costs explicitly, since a lot of write-ups skip this part:

More tokens spent overall — planning, dispatch, and merge steps are all additional LLM calls layered on top of the actual work
Harder to debug — a wrong final answer could originate in planning, in any worker, or in the merge step; this is exactly why the tracing patterns from the observability post in this series matter more, not less, once you add orchestration
Coordination overhead can exceed the benefit for genuinely simple, linear tasks — a single well-scoped agent with good tools is still the right default until you hit one of the failure modes above

The Gartner-cited pattern worth taking seriously: a meaningful share of agentic projects get cancelled not because agents don't work, but because teams reached for orchestration before the task actually needed it, and the coordination overhead outweighed the gain.

A reasonable decision rule

Reach for a single agent by default. Move to orchestration when at least one of these is true:

The task has genuinely independent sub-parts that would otherwise share (and pollute) one context window
Different sub-tasks need meaningfully different tool sets or behavioral instructions
You need parallelism for latency reasons, not just organizational tidiness

If none of those apply, a single well-designed agent with good tool selection will usually outperform a multi-agent system on both cost and reliability.

What's next

The next post in this series steps back to a cost/control framework: when running open-weight models yourself makes sense versus calling a closed API, and how to reason about that tradeoff instead of defaulting to whichever one is trending.

This is part of a series on AI infrastructure patterns, drawing on production work building multi-provider AI gateways in Rust. Follow along for the rest of the series.

Inference Optimization for the Rest of Us — KV Cache, Quantization, and Latency Tradeoffs

mihir mohapatra — Thu, 09 Jul 2026 16:58:26 +0000

Most inference optimization content is written for people running fleets of H100s. If you're serving models out of a single GPU box, a modest EC2/ECS setup, or even CPU inference for smaller models, a lot of that advice doesn't transfer — the constraints are different, and the "just add more GPUs" answer isn't available to you.

This post covers the three levers that actually move the needle at small-to-mid scale: KV cache management, quantization, and the latency tradeoffs that come with both. No cluster-scale assumptions.

Why this matters before it's a crisis

Inference cost and latency don't show up as a line item until they do — usually right around the point where a side project or an MVP gets real traffic. At that point you have three options: spend more on hardware, spend more on hosted API calls, or actually understand what's eating your latency and memory budget. The third option is the only one that scales your understanding along with your infrastructure.

Lever 1: KV cache — what it is and why it dominates memory

Every token a transformer generates attends back to all previous tokens. Naively, that means recomputing attention over the whole sequence at every step — wildly wasteful. The KV cache stores the key and value tensors from previous tokens so each new token only needs one forward pass, not a full replay.

The tradeoff: that cache grows linearly with sequence length, and its memory footprint is often the actual bottleneck — not the model weights themselves — once you're serving multiple concurrent requests with long contexts.

A rough sizing formula, per request:

kv_cache_bytes = 2 * num_layers * num_heads * head_dim * seq_len * bytes_per_element

The 2 is for both K and V. For a 7B-class model with 32 layers, 32 heads, head_dim 128, at fp16 (2 bytes) and a 4096-token context, that's already several hundred MB — per concurrent request. Multiply by your target concurrency and you'll see why KV cache, not weights, is usually what determines how many simultaneous users a single GPU can serve.

Practical levers on KV cache size

Multi-query / grouped-query attention (GQA) — if you're choosing a model, prefer architectures using GQA (most modern open-weight models do). It shares K/V heads across multiple query heads, cutting cache size significantly with minimal quality loss.
Sliding window attention — cap how far back the cache needs to reach for local-context tasks. Not appropriate for tasks needing full long-range recall, but a real win for chat-style workloads with short working context.
Cache eviction policies — for multi-turn chat apps, evict or summarize older turns instead of keeping the full cache alive indefinitely. This is an application-level decision, not just an inference-engine setting.
PagedAttention-style memory management (used in vLLM and similar serving frameworks) — treats the KV cache like virtual memory pages instead of requiring contiguous allocation, which dramatically reduces fragmentation waste when you have many concurrent requests with different sequence lengths.

If you're rolling your own serving layer rather than using vLLM/TGI, this is the single highest-leverage thing to borrow from those projects even if you don't adopt them wholesale.

Lever 2: Quantization — trading precision for throughput

Quantization reduces the numeric precision of model weights (and sometimes activations), shrinking both memory footprint and, often, latency.

Precision	Relative size	Typical use case
fp16/bf16	1x (baseline)	Default for most hosted APIs
int8	~0.5x	Good quality retention, solid speedup
int4 (GPTQ/AWQ)	~0.25x	Runs 7B-class models on consumer GPUs
GGUF quantized (Q4_K_M etc.)	~0.25–0.35x	CPU/edge inference via llama.cpp

The practical decision isn't "should I quantize" — for anything outside a hyperscaler budget, you almost certainly should. It's which quantization scheme, and that depends on your bottleneck:

Memory-bound (model barely fits, or you want more concurrent requests) → int4 or GGUF gets you the most headroom
Latency-bound on CPU → GGUF via llama.cpp with an appropriate quant level (Q4_K_M is a reasonable default starting point)
Quality-sensitive (structured output, code generation, anything requiring precise instruction-following) → stick to int8 or test carefully before dropping to int4; quality degradation isn't linear and some tasks are far more sensitive than others

A pattern worth adopting: don't guess — benchmark your actual task against 2–3 quantization levels before committing. A model that quantizes cleanly on general chat can degrade noticeably on structured extraction or tool-calling tasks.

// Rough shape of a quant-level config in a gateway/router setup
#[derive(Debug, Clone)]
struct ModelVariant {
    name: String,
    quant: QuantLevel,      // Fp16, Int8, Int4Awq, GgufQ4KM
    vram_gb: f32,
    avg_tokens_per_sec: f32,
}

fn select_variant(task: TaskProfile, variants: &[ModelVariant]) -> &ModelVariant {
    // route structured/tool-calling tasks to higher precision,
    // route casual chat/summarization to the cheapest variant that fits
    variants.iter()
        .filter(|v| task.quality_floor <= v.quant.quality_score())
        .min_by(|a, b| a.vram_gb.partial_cmp(&b.vram_gb).unwrap())
        .unwrap_or(&variants[0])
}

This is the same instinct as multi-provider routing in a gateway — route by task requirements, not a single fixed model for everything.

Lever 3: Latency tradeoffs — where the milliseconds actually go

Once cache and quantization are handled, the remaining latency budget splits into pieces that respond to different fixes:

Time to first token (TTFT) — dominated by prompt processing (prefill). Long system prompts or heavy RAG context directly inflate this. If TTFT matters more than total generation time for your use case (e.g. interactive chat), trimming prompt bloat is often a bigger win than any inference engine tuning.
Inter-token latency — dominated by memory bandwidth during decode, which is exactly where quantization and KV cache size have direct leverage.
Batching — continuous batching (as opposed to static batching) lets new requests join a batch mid-flight instead of waiting for the whole batch to finish, which matters a lot under real, uneven traffic patterns. This is one of the core reasons frameworks like vLLM outperform naive serving loops even before touching quantization.

A concrete tradeoff worth naming explicitly: batching improves throughput but can hurt per-request latency under load, since a request may wait briefly for a batch slot. For a single interactive assistant, that's rarely visible. For a shared production endpoint under real concurrency, it's the difference between p50 and p95 latency looking very different — and it's worth deciding up front which one your use case actually cares about.

Putting it together: a practical decision order

For a small-to-mid scale deployment, the order that gets you the most benefit per hour invested:

Pick a GQA-architecture model if you have a choice — this is free and structural
Quantize and benchmark against your actual task, not a generic leaderboard — int8 as a safe default, int4/GGUF if memory-bound
Adopt continuous batching via an existing serving framework (vLLM, TGI) rather than writing your own naive batching loop
Trim prompt bloat if TTFT matters for your UX — this is often the cheapest win and the most overlooked
Only then consider hardware changes — by this point you'll actually know whether you're memory-bound or compute-bound, which determines what upgrade (if any) is worth the cost

None of this requires hyperscaler infrastructure. It requires knowing which of these five levers you're actually constrained by before spending money on the wrong fix.

What's next

The next post in this series moves back up the stack — open-weight versus closed API models, and a practical cost/control framework for deciding which to run yourself versus call through a provider.

This is part of a series on AI infrastructure patterns, drawing on production work building multi-provider AI gateways in Rust. Follow along for the rest of the series.

Observability for LLM Apps: Tracing, Cost Tracking, and Eval Loops

mihir mohapatra — Mon, 06 Jul 2026 08:43:57 +0000

If you've shipped a traditional backend service, you already know the observability checklist: logs, metrics, traces, alerts. LLM-powered apps need all of that — plus a few things that don't exist in a normal request/response world: token spend, prompt/response pairs, and quality drift that no HTTP status code will ever tell you about.

This post covers the three pillars that actually matter once an LLM app leaves the demo stage:

Tracing — following a request across retries, tool calls, and multi-provider fallbacks
Cost tracking — turning token counts into real-time dollar visibility per route, user, or feature
Eval loops — catching quality regressions before your users do

I'll use patterns from a multi-provider AI gateway (Rust, Axum, Prometheus) as the running example, but the ideas apply regardless of stack.

Why normal APM isn't enough

A standard APM trace tells you latency and status code. For an LLM call, that's barely half the picture. The same 200 OK can represent:

A cheap 400-token completion from a fast model
A $2 completion from a frontier model after two retries
A technically successful but hallucinated response

None of that shows up in a duration histogram. You need traces that carry LLM-specific attributes, not just HTTP ones.

Pillar 1: Tracing

The unit of work in an LLM app usually isn't a single call — it's a chain: prompt construction → provider call → possible retry/fallback → tool invocation → final response. Each of those needs to be a span, not just the outer request.

A minimal span shape for an LLM call looks like this:

#[derive(Debug, Serialize)]
struct LlmSpan {
    trace_id: String,
    span_id: String,
    parent_span_id: Option<String>,
    provider: String,        // "anthropic", "openai", "ollama"
    model: String,
    prompt_tokens: u32,
    completion_tokens: u32,
    latency_ms: u64,
    status: SpanStatus,      // Success, Retried, Failed, RateLimited
    retry_count: u8,
    fallback_from: Option<String>, // which provider it fell back from
}

The two fields that matter most for debugging in production are retry_count and fallback_from. Without them, a "successful" trace hides the fact that your primary provider is degrading and your fallback logic is quietly absorbing the pain — until it can't.

Wire this into OpenTelemetry so it plays nicely with whatever trace backend you already run:

use opentelemetry::trace::{Tracer, Span};

let tracer = global::tracer("llm-gateway");
let mut span = tracer.start("llm.completion");
span.set_attribute(KeyValue::new("llm.provider", provider.to_string()));
span.set_attribute(KeyValue::new("llm.model", model.to_string()));
span.set_attribute(KeyValue::new("llm.prompt_tokens", prompt_tokens as i64));
span.set_attribute(KeyValue::new("llm.completion_tokens", completion_tokens as i64));
span.set_attribute(KeyValue::new("llm.retry_count", retry_count as i64));

This gives you a waterfall view in Jaeger/Tempo where a fallback chain is visually obvious instead of buried in logs.

Pillar 2: Cost tracking

Token counts alone don't tell a story anyone outside engineering cares about. The move is to convert tokens to cost at write-time and expose it as a first-class Prometheus metric, not something you reconstruct later from a token count times a price you looked up once and forgot to update.

use prometheus::{register_counter_vec, CounterVec};

static LLM_COST_USD: Lazy<CounterVec> = Lazy::new(|| {
    register_counter_vec!(
        "llm_cost_usd_total",
        "Cumulative LLM spend in USD",
        &["provider", "model", "route"]
    ).unwrap()
});

fn record_cost(provider: &str, model: &str, route: &str, prompt_tokens: u32, completion_tokens: u32) {
    let (input_rate, output_rate) = pricing_for(provider, model);
    let cost = (prompt_tokens as f64 * input_rate) + (completion_tokens as f64 * output_rate);
    LLM_COST_USD
        .with_label_values(&[provider, model, route])
        .inc_by(cost);
}

Keep pricing_for in a config file, not hardcoded — provider pricing changes more often than you'd like, and a stale rate table quietly lies to your finance dashboard.

With provider, model, and route as labels, one PromQL query answers the question every team eventually asks:

sum by (route) (
  rate(llm_cost_usd_total[1h])
)

That's "which feature is burning the budget," per hour, without a spreadsheet. Pair it with a budget-based alert:

- alert: LLMCostSpike
  expr: sum(rate(llm_cost_usd_total[15m])) * 3600 > 5
  for: 10m
  annotations:
    summary: "LLM spend exceeding $5/hr sustained rate"

This is the single highest-leverage dashboard panel you can add — it's usually the first thing a manager asks for once real usage starts.

Pillar 3: Eval loops

This is the part with no equivalent in traditional backend observability. A response can be fast, cheap, and a 200 — and still be wrong. Nothing in your metrics stack catches that unless you build for it.

The practical version of an eval loop, at gateway scale, isn't a big offline benchmark suite — it's a lightweight, continuous check on production traffic:

Golden set regression — a small, versioned set of prompt/expected-output pairs run against every model/provider change before it ships
Sampled production scoring — periodically re-score a random 1–2% of real completions using a cheaper judge model, and track the score as a metric over time
Structural checks — cheap, deterministic checks that don't need a judge model at all: did the JSON parse, is the response non-empty, is it within expected length bounds

struct EvalResult {
    trace_id: String,
    check_name: String,       // "json_valid", "length_bounds", "judge_score"
    passed: bool,
    score: Option<f32>,
}

async fn run_structural_checks(response: &str) -> Vec<EvalResult> {
    vec![
        EvalResult {
            trace_id: current_trace_id(),
            check_name: "json_valid".into(),
            passed: serde_json::from_str::<Value>(response).is_ok(),
            score: None,
        },
        EvalResult {
            trace_id: current_trace_id(),
            check_name: "length_bounds".into(),
            passed: (20..4000).contains(&response.len()),
            score: None,
        },
    ]
}

Feed the pass rate into the same Prometheus instance as everything else:

static EVAL_PASS_RATE: Lazy<GaugeVec> = Lazy::new(|| {
    register_gauge_vec!(
        "llm_eval_pass_rate",
        "Rolling eval pass rate by check",
        &["check_name"]
    ).unwrap()
});

Now a model swap, a prompt template change, or a silent provider-side model update all show up as a dip on the same dashboard where you already watch latency and cost — instead of as a support ticket three days later.

Putting it together

The dashboard that actually gets used day-to-day has three rows, matching the three pillars:

Tracing row: p50/p95 latency by provider, retry rate, fallback rate
Cost row: spend by route/model, hourly burn rate, budget alert status
Quality row: structural check pass rate, sampled judge score trend

None of these pillars is optional once an LLM feature has real users. Tracing tells you what happened, cost tracking tells you what it's costing you, and eval loops tell you whether it's still working — and that third one is the piece most teams skip until it's already a production incident.

What's next

The next post in this series covers inference optimization — KV cache strategies and quantization tradeoffs for teams that aren't running hyperscaler-scale infrastructure but still need to control latency and cost at the model-serving layer.

This is part of a series on AI infrastructure patterns, drawing on production work building multi-provider AI gateways in Rust. Follow along for the rest of the series.

FastAPI Crash Course: Build a CRUD REST API in ~40 Lines of Python

mihir mohapatra — Sun, 28 Jun 2026 09:38:51 +0000

I spend most of my days writing Java and Kotlin for FinTech systems at scale. But every now and then, Python pulls me back in — and FastAPI is exactly the reason why. This is a hands-on crash course based on a project I published on GitHub.

👉 View the full source on GitHub

Why FastAPI?

Coming from a Spring Boot background, my first instinct was scepticism. Another Python framework? But FastAPI is genuinely different:

Automatic interactive docs via Swagger UI — no extra config needed
Request validation powered by Pydantic — think of it as @Valid + @RequestBody baked in
Type hints as the source of truth — Python's type system does the heavy lifting
Async-ready from the ground up — ASGI all the way

The best part? You can build a fully working REST API in under 50 lines.

What We're Building

A Tea CRUD API — simple, but covers every REST pattern:

Method	Endpoint	What It Does
GET	`/`	Health / welcome message
GET	`/teas`	List all teas
POST	`/teas`	Add a new tea
PUT	`/teas/{tea_id}`	Update an existing tea
DELETE	`/teas/{tea_id}`	Remove a tea

No database — we're using an in-memory list to keep the focus on FastAPI itself.

Prerequisites

Python 3.10+
pip

Verify:

python --version
pip --version

Setup

Create and activate a virtual environment:

# Windows (PowerShell)
python -m venv venv
.\venv\Scripts\Activate

# macOS / Linux
python3 -m venv venv
source venv/bin/activate

Install dependencies:

pip install "fastapi[standard]"

This single command installs FastAPI, Uvicorn (the ASGI server), and all optional extras — all you need to get running.

The Full Code

Create a file called main.py and drop this in:

from fastapi import FastAPI
from pydantic import BaseModel
from typing import List

app = FastAPI()

class Tea(BaseModel):
    id: int
    name: str
    origin: str

teas: List[Tea] = []

@app.get("/")
def read_root():
    return {"message": "Welcome to chai code"}

@app.get("/teas")
def get_teas():
    return teas

@app.post("/teas")
def add_teas(tea: Tea):
    teas.append(tea)
    return tea

@app.put("/teas/{tea_id}")
def update_tea(tea_id: int, update_tea: Tea):
    for index, tea in enumerate(teas):
        if tea.id == tea_id:
            teas[index] = update_tea
            return update_tea
    return {"error": "Tea not found"}

@app.delete("/teas/{tea_id}")
def delete_tea(tea_id: int):
    for index, tea in enumerate(teas):
        if tea.id == tea_id:
            deleted = teas.pop(index)
            return deleted
    return {"error": "Tea not found"}

That's it. 43 lines. Let's break it down.

Code Walkthrough

1. The App Instance

app = FastAPI()

This is your application object — equivalent to new SpringApplication() in Spring Boot, but in one line with no annotations, XML, or config files.

2. The Pydantic Model

class Tea(BaseModel):
    id: int
    name: str
    origin: str

BaseModel from Pydantic is the magic here. Any class that extends it automatically gets:

Request body parsing — FastAPI reads the incoming JSON and maps it to this class
Type validation — if id isn't an integer, FastAPI returns a 422 Unprocessable Entity with a clear error message
Serialization — returning a Tea instance automatically serializes it to JSON

For Java developers: think @Data + @Valid + Jackson — except it's one import and zero annotations on your fields.

3. The In-Memory Store

teas: List[Tea] = []

Just a Python list typed with List[Tea]. It acts as our database for this example. Doesn't survive restarts — but that's intentional for a learning project. You'd swap this with SQLAlchemy + PostgreSQL in a production setup.

4. GET — List All Teas

@app.get("/teas")
def get_teas():
    return teas

Return the list directly. FastAPI serializes List[Tea] to a JSON array automatically. No ResponseEntity, no @ResponseBody, no boilerplate.

5. POST — Add a Tea

@app.post("/teas")
def add_teas(tea: Tea):
    teas.append(tea)
    return tea

The function parameter tea: Tea tells FastAPI: "expect a JSON body matching the Tea schema." Pydantic validates it. If validation fails, FastAPI returns a detailed 422 error before your function is even called.

6. PUT — Update by ID

@app.put("/teas/{tea_id}")
def update_tea(tea_id: int, update_tea: Tea):
    for index, tea in enumerate(teas):
        if tea.id == tea_id:
            teas[index] = update_tea
            return update_tea
    return {"error": "Tea not found"}

{tea_id} in the path is a path parameter — FastAPI extracts it and coerces it to int automatically. You get both a path param (tea_id) and a request body (update_tea) from a single function signature. Clean.

7. DELETE — Remove by ID

@app.delete("/teas/{tea_id}")
def delete_tea(tea_id: int):
    for index, tea in enumerate(teas):
        if tea.id == tea_id:
            deleted = teas.pop(index)
            return deleted
    return {"error": "Tea not found"}

Linear scan through our in-memory list — fine for learning, but you'd index by ID with a dict or use a database in production.

Run It

python -m uvicorn main:app --reload

Expected output:

INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO:     Started reloader process

The --reload flag means the server restarts automatically on every file save — great for development.

Interactive API Docs (Free!)

This is FastAPI's killer feature. Open your browser:

URL	What You Get
`http://127.0.0.1:8000/docs`	Swagger UI — test every endpoint interactively
`http://127.0.0.1:8000/redoc`	ReDoc — clean, readable API reference

No extra setup. No Postman required for a quick test. The docs are generated directly from your type hints and Pydantic models.

Quick Test via Swagger UI

Go to http://127.0.0.1:8000/docs
Click POST /teas → Try it out
Paste this body:

{
  "id": 1,
  "name": "Assam Tea",
  "origin": "India"
}

Click Execute
Now hit GET /teas — you'll see your tea in the list

What's Next (Production Roadmap)

This in-memory app is intentionally minimal. Here's how you'd evolve it toward production:

Concern	Tool
Persistent storage	SQLAlchemy + PostgreSQL or SQLite
Migrations	Alembic
Auth	JWT via `python-jose` + `passlib`
Dependency Injection	FastAPI's built-in `Depends()` system
Environment config	`pydantic-settings`
Containerization	Docker + `uvicorn` in production mode
Testing	`pytest` + `httpx` (async test client)
Async DB	`asyncpg` + SQLAlchemy async sessions

Java/Kotlin Developer Take

Having spent years with Spring Boot, here's my honest comparison:

FastAPI wins on:

Speed of prototyping — no boilerplate ceremony
Auto-generated docs that are actually useful
Cleaner validation without annotation overload

Spring Boot still wins on:

Enterprise ecosystem maturity
Structured DI with complex bean graphs
Mature monitoring (Actuator, Micrometer)

FastAPI fills a real gap: it's the right tool when you want Python's data ecosystem (ML, data pipelines) behind a production-quality HTTP API. The two aren't in competition — they're tools for different contexts.

Source Code

Everything in this post is available here:

github.com/MihirMohapatra/fast-api-example

Drop a ⭐ if it helped you get started.

Building toward senior remote roles at US/EU companies. Follow along as I explore FastAPI, Rust, Go, and AI infrastructure — writing from the perspective of a practising backend engineer, not a tutorial blog.

From Learning to Shipping — Docker, Graceful Shutdown & ECS Fargate

mihir mohapatra — Thu, 25 Jun 2026 05:13:15 +0000

From Learning to Shipping — Docker, Graceful Shutdown & ECS Fargate

This is the final post in the series. In part 6 I covered testing — table-driven tests, httptest, benchmarks. Now I'm taking everything built across the last six posts and shipping it: a multi-stage Docker build, graceful shutdown wired into the Gin server, and deployment to AWS ECS Fargate using Terraform — the same stack I used for rust-ai-gateway.

If you've followed along, this is where the learning project becomes something you can actually point a recruiter at.

Multi-Stage Docker Build

The biggest Go deployment win over the JVM is binary size. A Go service compiles to a single self-contained binary with no runtime dependency — no JVM to install, no classpath to assemble. That means the production image can be built in two stages: a full Go toolchain image to compile, then a minimal scratch or distroless image to run.

# ---- build stage ----
FROM golang:1.22-alpine AS builder

WORKDIR /app

COPY go.mod go.sum ./
RUN go mod download

COPY . .

RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 \
    go build -ldflags="-s -w" -o orders-api ./cmd/main.go

# ---- run stage ----
FROM gcr.io/distroless/static-debian12

WORKDIR /app
COPY --from=builder /app/orders-api .

EXPOSE 8080
USER nonroot:nonroot

ENTRYPOINT ["/app/orders-api"]

Three things worth unpacking here. CGO_ENABLED=0 disables C bindings, which is required for a fully static binary that works in distroless. The -ldflags="-s -w" strip the debug symbol table and DWARF info, shaving 20–30% off the binary size for free. And distroless/static is the production choice over scratch — it has no shell (good for security) but includes timezone data and SSL certificates that your service will almost certainly need.

The resulting image typically comes in under 20MB. The equivalent Spring Boot fat jar with a JRE base image usually lands somewhere north of 300MB.

Graceful Shutdown

Out of the box, r.Run(":8080") in Gin blocks forever and stops immediately on SIGTERM — mid-request, no cleanup, no connection draining. In a containerised environment where ECS sends SIGTERM before terminating a task, that's dropped requests on every deployment. The fix is to run the server in a goroutine and listen for the OS signal yourself:

// cmd/main.go
package main

import (
    "context"
    "log"
    "net/http"
    "os"
    "os/signal"
    "syscall"
    "time"

    "orders-api/handler"
    "orders-api/middleware"
    "orders-api/store"

    "github.com/gin-gonic/gin"
)

func main() {
    s := store.NewInMemoryStore()
    h := handler.NewOrderHandler(s)

    r := gin.New()
    r.Use(middleware.Logger())

    api := r.Group("/api/v1")
    {
        orders := api.Group("/orders")
        orders.POST("", h.Create)
        orders.GET("", h.List)
        orders.GET("/:id", h.GetByID)
    }

    srv := &http.Server{
        Addr:    ":8080",
        Handler: r,
    }

    // start server in background goroutine
    go func() {
        log.Println("server listening on :8080")
        if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
            log.Fatalf("listen error: %v", err)
        }
    }()

    // block until SIGINT or SIGTERM
    quit := make(chan os.Signal, 1)
    signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM)
    <-quit

    log.Println("shutdown signal received, draining connections...")

    ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer cancel()

    if err := srv.Shutdown(ctx); err != nil {
        log.Fatalf("forced shutdown: %v", err)
    }

    log.Println("server exited cleanly")
}

srv.Shutdown(ctx) tells the HTTP server to stop accepting new connections and wait for in-flight requests to finish, up to the 30-second deadline. This is the goroutine and channel pattern from part 3 applied directly to real production code — quit is a channel, <-quit blocks until a signal arrives, and then the shutdown sequence runs. No framework magic, just the primitives we've been using all along.

Health Check Endpoint

ECS needs a health check to know when a task is ready for traffic and when it's gone unhealthy. Two lines in the router:

r.GET("/health", func(c *gin.Context) {
    c.JSON(http.StatusOK, gin.H{"status": "ok"})
})

r.GET("/ready", func(c *gin.Context) {
    // in a real service: check DB connectivity here
    c.JSON(http.StatusOK, gin.H{"status": "ready"})
})

/health is the liveness probe — is the process running. /ready is the readiness probe — is the service ready to serve traffic (database connected, caches warm). Keeping them separate lets ECS restart an unhealthy container without pulling it from the load balancer unnecessarily during a slow startup.

Terraform: ECS Fargate

The infrastructure follows the same pattern as rust-ai-gateway — ECR for the image, ECS Fargate for the task, an ALB in front. The key pieces:

# ecr.tf
resource "aws_ecr_repository" "orders_api" {
  name                 = "orders-api"
  image_tag_mutability = "MUTABLE"

  image_scanning_configuration {
    scan_on_push = true
  }
}

# ecs.tf
resource "aws_ecs_task_definition" "orders_api" {
  family                   = "orders-api"
  requires_compatibilities = ["FARGATE"]
  network_mode             = "awsvpc"
  cpu                      = "256"
  memory                   = "512"
  execution_role_arn       = aws_iam_role.ecs_task_execution.arn

  container_definitions = jsonencode([{
    name      = "orders-api"
    image     = "${aws_ecr_repository.orders_api.repository_url}:latest"
    essential = true

    portMappings = [{
      containerPort = 8080
      protocol      = "tcp"
    }]

    healthCheck = {
      command     = ["CMD-SHELL", "wget -qO- http://localhost:8080/health || exit 1"]
      interval    = 30
      timeout     = 5
      retries     = 3
      startPeriod = 10
    }

    logConfiguration = {
      logDriver = "awslogs"
      options = {
        "awslogs-group"         = "/ecs/orders-api"
        "awslogs-region"        = var.aws_region
        "awslogs-stream-prefix" = "ecs"
      }
    }
  }])
}

resource "aws_ecs_service" "orders_api" {
  name            = "orders-api"
  cluster         = aws_ecs_cluster.main.id
  task_definition = aws_ecs_task_definition.orders_api.arn
  desired_count   = 2
  launch_type     = "FARGATE"

  network_configuration {
    subnets          = var.private_subnet_ids
    security_groups  = [aws_security_group.ecs_tasks.id]
    assign_public_ip = false
  }

  load_balancer {
    target_group_arn = aws_lb_target_group.orders_api.arn
    container_name   = "orders-api"
    container_port   = 8080
  }

  deployment_minimum_healthy_percent = 100
  deployment_maximum_percent         = 200
}

desired_count = 2 with deployment_minimum_healthy_percent = 100 means ECS will never take both tasks down simultaneously during a deployment — new tasks must become healthy before old ones are stopped, which combined with the graceful shutdown above gives zero-downtime deploys.

CI/CD: GitHub Actions

The deploy pipeline that ties it together — build, push to ECR, force a new ECS deployment:

# .github/workflows/deploy.yml
name: Deploy

on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: ${{ secrets.AWS_REGION }}

      - name: Login to ECR
        id: login-ecr
        uses: aws-actions/amazon-ecr-login@v2

      - name: Build, tag and push image
        env:
          ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
          IMAGE_TAG: ${{ github.sha }}
        run: |
          docker build -t $ECR_REGISTRY/orders-api:$IMAGE_TAG .
          docker push $ECR_REGISTRY/orders-api:$IMAGE_TAG
          docker tag $ECR_REGISTRY/orders-api:$IMAGE_TAG $ECR_REGISTRY/orders-api:latest
          docker push $ECR_REGISTRY/orders-api:latest

      - name: Force ECS deployment
        run: |
          aws ecs update-service \
            --cluster main \
            --service orders-api \
            --force-new-deployment

Using github.sha as the image tag means every deploy is traceable — you can look at a running ECS task, find the image tag, and map it back to an exact commit. Tagging latest separately lets Terraform and the task definition reference a stable name, while the SHA tag is there for auditability.

What This Series Built

Seven posts ago I was explaining why I was picking up Go. Here's what actually got built:

A type system that composes instead of inheriting, with interfaces that match on shape rather than declaration
A concurrency model that puts goroutines and channels in reach by default
Error handling that makes failure visible at every call site
A Gin REST API with middleware, request validation, and clean handler separation
A full test suite: table-driven unit tests, httptest handler tests, and benchmarks
A production deployment: multi-stage Docker under 20MB, graceful shutdown, ECS Fargate behind an ALB, GitHub Actions CI/CD

The thing I didn't expect: Go's simplicity compounds. Every new piece — a new handler, a new test, a new middleware — follows the same small set of patterns. There's less to remember, and the code that comes out of it is easier to read six months later than Java or Kotlin at comparable complexity. I get why teams that care about maintainability and onboarding speed keep reaching for it.

Next on my list for this codebase: swapping the in-memory store for pgx and PostgreSQL, adding OpenTelemetry tracing, and wiring in a real secrets manager. That might become another series, or it might just end up in the GitHub repo.

If you've followed along from part 1 — thanks for reading. And if you're just landing here and want the whole project, the full source is at github.com/MihirMohapatra/go-orders-api once I push it.

What should the next series be? Another Go deep-dive, or something at the intersection of Go and AI infrastructure?

Testing in Go — Table-Driven Tests, Benchmarks, and go test Habits

mihir mohapatra — Tue, 23 Jun 2026 11:27:55 +0000

Testing in Go — Table-Driven Tests, Benchmarks, and go test Habits

In part 5 I built a working Orders API with Gin — routes, middleware, an in-memory store, and proper error handling. This time I'm writing tests for it, and the story here is one of the things I've genuinely liked about Go from the start: testing is a first-class citizen with zero framework ceremony. No JUnit, no Mockito, no Spring Boot test context to spin up. Just go test and the standard library.

The go test Mental Model

Every _test.go file in a Go package is compiled and run by go test. Test functions follow one convention — they take a single *testing.T argument and start with Test:

func TestSomething(t *testing.T) {
    // ...
}

That's the entire contract. No annotations, no base classes, no test runner configuration file. Running go test ./... from the project root picks up every test across every package recursively. Running with -v shows each test name and its pass/fail. Running with -race enables the data race detector we talked about in part 3. These three invocations cover 90% of what I reach for day to day:

go test ./...                  # run everything
go test -v ./handler/...       # verbose, one package
go test -race ./...            # race detector on
go test -run TestCreateOrder . # run one test by name

Table-Driven Tests: The Go Idiom

The single most important testing pattern in Go is the table-driven test. Instead of writing one test function per scenario, you define a slice of cases as a struct, loop over them, and let the test framework tell you exactly which case failed. Coming from JUnit's @ParameterizedTest, it's the same idea — except it's just a for loop, no framework required:

// store/order_test.go
package store_test

import (
    "testing"

    "orders-api/model"
    "orders-api/store"
)

func TestCreateOrder(t *testing.T) {
    cases := []struct {
        name      string
        input     model.CreateOrderRequest
        wantErr   bool
    }{
        {
            name:    "valid order",
            input:   model.CreateOrderRequest{Customer: "Alice", Amount: 99.99},
            wantErr: false,
        },
        {
            name:    "zero amount",
            input:   model.CreateOrderRequest{Customer: "Bob", Amount: 0},
            wantErr: false, // store doesn't validate, handler does
        },
        {
            name:    "empty customer",
            input:   model.CreateOrderRequest{Customer: "", Amount: 50.0},
            wantErr: false,
        },
    }

    for _, tc := range cases {
        t.Run(tc.name, func(t *testing.T) {
            s := store.NewInMemoryStore()
            order, err := s.Create(tc.input)

            if tc.wantErr && err == nil {
                t.Errorf("expected error, got nil")
            }
            if !tc.wantErr && err != nil {
                t.Errorf("unexpected error: %v", err)
            }
            if !tc.wantErr && order.ID == "" {
                t.Errorf("expected non-empty order ID, got empty string")
            }
        })
    }
}

t.Run creates a named sub-test for each case. When a sub-test fails, Go prints the full path — TestCreateOrder/zero_amount — so you know instantly which case broke without digging through output. This is the pattern the entire Go standard library uses internally, and it's the first habit worth locking in.

Testing the Store: ErrNotFound

The GetByID path needs its own table — specifically the not-found branch we wired into the handler in part 5:

func TestGetByID(t *testing.T) {
    s := store.NewInMemoryStore()

    // seed one order
    created, _ := s.Create(model.CreateOrderRequest{
        Customer: "Alice",
        Amount:   49.99,
    })

    cases := []struct {
        name    string
        id      string
        wantErr error
    }{
        {"existing order", created.ID, nil},
        {"missing order", "does-not-exist", store.ErrNotFound},
        {"empty id", "", store.ErrNotFound},
    }

    for _, tc := range cases {
        t.Run(tc.name, func(t *testing.T) {
            _, err := s.GetByID(tc.id)
            if !errors.Is(err, tc.wantErr) {
                t.Errorf("got err %v, want %v", err, tc.wantErr)
            }
        })
    }
}

errors.Is in the test assertion mirrors exactly how the handler checks errors — the sentinel defined in the store package is the contract, and we're verifying both sides of it honour it.

Handler Tests with httptest

Testing the handler layer without spinning up a real server is where Go's net/http/httptest package earns its place. It gives you a fake ResponseRecorder that captures everything the handler writes — status code, headers, body — without any network involvement:

// handler/order_test.go
package handler_test

import (
    "bytes"
    "encoding/json"
    "net/http"
    "net/http/httptest"
    "testing"

    "orders-api/handler"
    "orders-api/model"
    "orders-api/store"

    "github.com/gin-gonic/gin"
)

func setupRouter(h *handler.OrderHandler) *gin.Engine {
    gin.SetMode(gin.TestMode)
    r := gin.New()
    r.POST("/orders", h.Create)
    r.GET("/orders/:id", h.GetByID)
    r.GET("/orders", h.List)
    return r
}

func TestCreateOrderHandler(t *testing.T) {
    cases := []struct {
        name       string
        body       any
        wantStatus int
    }{
        {
            name:       "valid request",
            body:       model.CreateOrderRequest{Customer: "Alice", Amount: 99.99},
            wantStatus: http.StatusCreated,
        },
        {
            name:       "missing customer",
            body:       map[string]any{"amount": 49.99},
            wantStatus: http.StatusBadRequest,
        },
        {
            name:       "negative amount",
            body:       map[string]any{"customer": "Bob", "amount": -10},
            wantStatus: http.StatusBadRequest,
        },
    }

    for _, tc := range cases {
        t.Run(tc.name, func(t *testing.T) {
            s := store.NewInMemoryStore()
            h := handler.NewOrderHandler(s)
            r := setupRouter(h)

            bodyBytes, _ := json.Marshal(tc.body)
            req := httptest.NewRequest(http.MethodPost, "/orders", bytes.NewReader(bodyBytes))
            req.Header.Set("Content-Type", "application/json")
            w := httptest.NewRecorder()

            r.ServeHTTP(w, req)

            if w.Code != tc.wantStatus {
                t.Errorf("status: got %d, want %d — body: %s",
                    w.Code, tc.wantStatus, w.Body.String())
            }
        })
    }
}

gin.SetMode(gin.TestMode) suppresses the debug output Gin prints during tests. Each case builds a fresh store and handler so tests are fully isolated — no shared state leaking between cases. The w.Body.String() in the error message means a failing test prints the actual response body, which cuts debug time significantly.

Benchmarks: Built Right In

Benchmarks follow the same file and naming conventions as tests, but use *testing.B and prefix with Benchmark. The runner calls your function repeatedly, adjusting iteration count until the timing is stable:

func BenchmarkStoreList(b *testing.B) {
    s := store.NewInMemoryStore()

    // seed 1000 orders
    for i := 0; i < 1000; i++ {
        s.Create(model.CreateOrderRequest{
            Customer: fmt.Sprintf("customer-%d", i),
            Amount:   float64(i) * 1.5,
        })
    }

    b.ResetTimer() // don't count seeding time

    for i := 0; i < b.N; i++ {
        _, err := s.List()
        if err != nil {
            b.Fatal(err)
        }
    }
}

Running go test -bench=. -benchmem ./store/... gives you nanoseconds per operation and allocations per operation:

BenchmarkStoreList-8    42361    28204 ns/op    24576 B/op    3 allocs/op

b.ResetTimer() after the seeding step is the habit that matters most here — you want to measure List, not Create × 1000. The -benchmem flag surfacing allocations per operation is the other one: in a hot path, allocation count matters as much as raw speed.

The go test Habits Worth Locking In

After a few weeks of writing Go tests, these are the ones I've made automatic:

Always use t.Run for multi-case scenarios. Even two cases. The named sub-test output pays for the extra line immediately when something breaks.

t.Parallel() for independent tests. Adding t.Parallel() at the top of a test (or sub-test) tells the runner it can execute concurrently with other parallel tests, cutting total test time on multi-core machines with no other changes.

-race in CI, always. Data races in Go are silent in normal test runs and catastrophic in production. The race detector adds overhead but catches things that would otherwise survive code review and only surface under load.

testify/assert if your team agrees, testing if they don't. The standard testing package requires writing your own error messages for every assertion. testify/assert gives you assert.Equal, assert.NoError, and friends with good default messages. Both are valid choices — the important thing is consistency within a codebase.

Up Next

Part 7 is the finish line: multi-stage Docker build, graceful shutdown, and getting this service onto ECS Fargate — the same deployment target I used for rust-ai-gateway. The testing patterns from this post will feed directly into the CI pipeline we wire up there.

Do you have a preferred Go testing library — pure testing, testify, or something else? Curious whether the standard library alone is enough once the projects get bigger.

Building My First Real API in Go — with Gin

mihir mohapatra — Mon, 22 Jun 2026 07:40:08 +0000

Building My First Real API in Go — with Gin

In part 4 I worked through Go's error handling model and came around on if err != nil. This is the post where everything from the last four parts — structs, interfaces, goroutines, error wrapping — stops being theoretical and actually gets wired into something that runs, accepts requests, and talks to a database.

I'm building a small Orders API: create an order, fetch one by ID, list all of them. Simple enough that the routing and middleware patterns are clear, complex enough that the error handling and concurrency patterns from earlier posts actually matter.

Why Gin

The Go standard library's net/http package is genuinely capable — you can write a production API with nothing else. But Gin adds three things that make it worth reaching for immediately: a clean router with path parameters and route groups, a middleware chain that composes nicely, and response helpers (c.JSON, c.AbortWithStatusJSON) that remove a lot of boilerplate. It's the closest thing Go has to a lightweight Express or Axum — fast, minimal, and doesn't try to own your entire architecture.

go mod init orders-api
go get github.com/gin-gonic/gin

Project Structure

Before a single line of code, here's the layout I settled on:

orders-api/
├── main.go
├── handler/
│   └── order.go
├── model/
│   └── order.go
├── store/
│   └── order.go
└── middleware/
    └── logger.go

This is the standard Go flat-package layout — no src/, no deep nesting, no framework-imposed folders. handler owns HTTP concerns, model owns data shapes, store owns persistence logic. The separation means each layer depends on the one below it, and none of them import each other in circles.

The Model

// model/order.go
package model

import "time"

type Order struct {
    ID        string    `json:"id"`
    Customer  string    `json:"customer"`
    Amount    float64   `json:"amount"`
    Status    string    `json:"status"`
    CreatedAt time.Time `json:"created_at"`
}

type CreateOrderRequest struct {
    Customer string  `json:"customer" binding:"required"`
    Amount   float64 `json:"amount"   binding:"required,gt=0"`
}

Two things worth noting here. The struct tags (json:"id") control serialisation — without them Gin would use the field name as-is, so CreatedAt would become CreatedAt in JSON rather than created_at. The binding tags on CreateOrderRequest are Gin's built-in validation: binding:"required" rejects blank fields, gt=0 rejects zero or negative amounts. This is the closest Go gets to Java Bean Validation annotations, except it's enforced at the handler layer rather than the model layer — I'll show how in a moment.

The Store

For now the store is in-memory — a map behind a sync.RWMutex. Part 7 will swap this for pgx. The interface is the important part:

// store/order.go
package store

import (
    "errors"
    "sync"
    "time"

    "orders-api/model"

    "github.com/google/uuid"
)

var ErrNotFound = errors.New("order not found")

type OrderStore interface {
    Create(req model.CreateOrderRequest) (model.Order, error)
    GetByID(id string) (model.Order, error)
    List() ([]model.Order, error)
}

type inMemoryStore struct {
    mu     sync.RWMutex
    orders map[string]model.Order
}

func NewInMemoryStore() OrderStore {
    return &inMemoryStore{orders: make(map[string]model.Order)}
}

func (s *inMemoryStore) Create(req model.CreateOrderRequest) (model.Order, error) {
    order := model.Order{
        ID:        uuid.New().String(),
        Customer:  req.Customer,
        Amount:    req.Amount,
        Status:    "pending",
        CreatedAt: time.Now(),
    }
    s.mu.Lock()
    s.orders[order.ID] = order
    s.mu.Unlock()
    return order, nil
}

func (s *inMemoryStore) GetByID(id string) (model.Order, error) {
    s.mu.RLock()
    defer s.mu.RUnlock()
    order, ok := s.orders[id]
    if !ok {
        return model.Order{}, ErrNotFound
    }
    return order, nil
}

func (s *inMemoryStore) List() ([]model.Order, error) {
    s.mu.RLock()
    defer s.mu.RUnlock()
    orders := make([]model.Order, 0, len(s.orders))
    for _, o := range s.orders {
        orders = append(orders, o)
    }
    return orders, nil
}

sync.RWMutex lets multiple concurrent reads proceed at the same time but serialises writes — the right primitive for a read-heavy in-memory store. The OrderStore interface is defined here, in the store package, but that's just for organisation — the handler depends on the interface, not the concrete type, which means swapping in a real database later is a one-line change in main.go.

The Handler

// handler/order.go
package handler

import (
    "errors"
    "net/http"

    "orders-api/model"
    "orders-api/store"

    "github.com/gin-gonic/gin"
)

type OrderHandler struct {
    store store.OrderStore
}

func NewOrderHandler(s store.OrderStore) *OrderHandler {
    return &OrderHandler{store: s}
}

func (h *OrderHandler) Create(c *gin.Context) {
    var req model.CreateOrderRequest
    if err := c.ShouldBindJSON(&req); err != nil {
        c.AbortWithStatusJSON(http.StatusBadRequest, gin.H{"error": err.Error()})
        return
    }

    order, err := h.store.Create(req)
    if err != nil {
        c.AbortWithStatusJSON(http.StatusInternalServerError, gin.H{"error": "could not create order"})
        return
    }

    c.JSON(http.StatusCreated, order)
}

func (h *OrderHandler) GetByID(c *gin.Context) {
    id := c.Param("id")

    order, err := h.store.GetByID(id)
    if err != nil {
        if errors.Is(err, store.ErrNotFound) {
            c.AbortWithStatusJSON(http.StatusNotFound, gin.H{"error": "order not found"})
            return
        }
        c.AbortWithStatusJSON(http.StatusInternalServerError, gin.H{"error": "could not fetch order"})
        return
    }

    c.JSON(http.StatusOK, order)
}

func (h *OrderHandler) List(c *gin.Context) {
    orders, err := h.store.List()
    if err != nil {
        c.AbortWithStatusJSON(http.StatusInternalServerError, gin.H{"error": "could not list orders"})
        return
    }
    c.JSON(http.StatusOK, orders)
}

ShouldBindJSON is where the binding tags from the model earn their keep — it validates and decodes the request body in one call, returning a descriptive error if anything fails. errors.Is(err, store.ErrNotFound) is the same pattern from part 4: unwrap through layers and match the sentinel. The handler doesn't know or care whether the store is in-memory or PostgreSQL — it just speaks to the interface.

Middleware: Request Logging

// middleware/logger.go
package middleware

import (
    "fmt"
    "time"

    "github.com/gin-gonic/gin"
)

func Logger() gin.HandlerFunc {
    return func(c *gin.Context) {
        start := time.Now()
        c.Next()
        fmt.Printf("[%s] %s %s %d (%s)\n",
            start.Format("15:04:05"),
            c.Request.Method,
            c.Request.URL.Path,
            c.Writer.Status(),
            time.Since(start),
        )
    }
}

c.Next() hands control to the next handler in the chain, then execution returns here after the response is written — which is why time.Since(start) gives you the actual request duration. This is the middleware pattern in every Gin-based service: wrap, call Next(), do something with what happened after.

Wiring It Together

// main.go
package main

import (
    "orders-api/handler"
    "orders-api/middleware"
    "orders-api/store"

    "github.com/gin-gonic/gin"
)

func main() {
    s := store.NewInMemoryStore()
    h := handler.NewOrderHandler(s)

    r := gin.New()
    r.Use(middleware.Logger())

    api := r.Group("/api/v1")
    {
        orders := api.Group("/orders")
        orders.POST("", h.Create)
        orders.GET("", h.List)
        orders.GET("/:id", h.GetByID)
    }

    r.Run(":8080")
}

gin.New() instead of gin.Default() gives a blank engine — no Gin's built-in logger or recovery middleware added automatically, since we're providing our own. The r.Group nesting gives every orders route the /api/v1/orders prefix without repeating it on each registration. Running this and hitting POST /api/v1/orders with a JSON body gets you a real response, with validation, error handling, and request logging, in about 120 lines across all files.

What This Draws On From Earlier Posts

Part 2's interfaces: OrderStore is exactly the kind of single-responsibility interface Go favours — three methods, defined by the consumer (the handler), satisfied implicitly by the concrete store.

Part 3's concurrency: sync.RWMutex in the store is the in-memory safe concurrency answer. In part 7 when this becomes a real database, the connection pool handles this instead.

Part 4's error handling: errors.Is(err, store.ErrNotFound) translating a store-layer sentinel into an HTTP 404 is the wrapping pattern paying off at the HTTP boundary.

Up Next

Part 6 is testing — table-driven tests, the httptest package for handler tests without spinning up a real server, and the benchmarking habits that will matter once we push toward production in part 7.

What's your preferred approach to structuring Go API projects — flat packages like this, or a domain-driven layout with more nesting? Genuinely curious how people land on this one.

Error Handling — Learning to Love `if err != nil`

mihir mohapatra — Sun, 21 Jun 2026 09:48:23 +0000

Error Handling — Learning to Love `if err != nil`

In part 3 I covered goroutines and channels, and how Go's concurrency model sidesteps a lot of the ceremony I was used to from the JVM. This time I'm tackling the thing I complained about in part 1 of this series before I'd even really tried it: error handling. I called if err != nil repetitive back then. A few weeks and a lot of real code later, I owe Go a partial apology.

No Exceptions, On Purpose

Coming from Java, the absence of try/catch is the first thing that feels like a missing feature. It isn't — it's a deliberate design choice. In Go, errors are just values. A function that can fail returns an error as its last return value, and the caller decides what to do with it, right there, inline:

func divide(a, b float64) (float64, error) {
    if b == 0 {
        return 0, errors.New("division by zero")
    }
    return a / b, nil
}

func main() {
    result, err := divide(10, 0)
    if err != nil {
        fmt.Println("error:", err)
        return
    }
    fmt.Println("result:", result)
}

That's the pattern you'll write hundreds of times in Go: call a function, check err, handle it or bail out, move on. No hidden control flow jumping up the call stack to whichever catch block happens to match. No checked-exception signatures cluttering method declarations. No RuntimeException quietly skipping past five layers of code that had no idea it could happen. Whatever can fail is sitting right there in the function signature, and you're forced to look at it.

Why the Repetition Is the Point

My part 1 complaint was that if err != nil everywhere feels manual. It is manual — and that's exactly the trade Go is making. In Java, an exception thrown deep in a call stack can silently propagate through layers of code that never declared they might fail, and you only find out where things actually break by reading a stack trace after the fact. In Go, every single point where something can go wrong is visible in the source, in order, as you read top to bottom:

func processOrder(id string) error {
    order, err := fetchOrder(id)
    if err != nil {
        return fmt.Errorf("fetching order: %w", err)
    }

    if err := validateOrder(order); err != nil {
        return fmt.Errorf("validating order: %w", err)
    }

    if err := chargePayment(order); err != nil {
        return fmt.Errorf("charging payment: %w", err)
    }

    return nil
}

Three things can fail here, and you can see all three without jumping anywhere. That %w verb in fmt.Errorf is doing something specific — it's wrapping the original error rather than discarding it, which means the caller can still inspect what actually went wrong underneath your added context. It's the closest thing Go has to a stack trace, except you control exactly how much context gets added at each layer instead of getting a wall of frames you have to parse yourself.

errors.Is and errors.As: Inspecting Wrapped Errors

Once you're wrapping errors as they bubble up, you need a way to check what's actually inside the wrapping. That's what errors.Is and errors.As are for:

var ErrNotFound = errors.New("order not found")

func fetchOrder(id string) (Order, error) {
    if id == "" {
        return Order{}, ErrNotFound
    }
    return Order{ID: id}, nil
}

func main() {
    err := processOrder("")
    if errors.Is(err, ErrNotFound) {
        fmt.Println("order doesn't exist, returning 404")
        return
    }
    fmt.Println("unexpected error:", err)
}

Even though processOrder wrapped the original error with fmt.Errorf("fetching order: %w", err), errors.Is can still see through the wrapping and match it against the sentinel ErrNotFound. This is the Go equivalent of catch (OrderNotFoundException e) — except instead of a type hierarchy of custom exception classes, you define a handful of sentinel error values (or custom error types, when you need to carry extra data) and compare against them explicitly wherever the distinction matters.

errors.As does the same thing but for pulling out a specific error type rather than matching a specific value — useful when you've defined a custom struct that implements the error interface and carries extra fields:

type ValidationError struct {
    Field string
    Msg   string
}

func (e *ValidationError) Error() string {
    return fmt.Sprintf("%s: %s", e.Field, e.Msg)
}

func main() {
    err := processOrder("123")
    var valErr *ValidationError
    if errors.As(err, &valErr) {
        fmt.Println("invalid field:", valErr.Field)
    }
}

Any type with an Error() string method satisfies the error interface — same implicit-satisfaction rule from part 2, just applied to errors specifically.

panic and recover: Not Your Exception Mechanism

Go does have panic and recover, and the instinct coming from Java is to treat them like throw and catch. That's a mistake that took me a minute to unlearn. panic is reserved for genuinely unrecoverable situations — a programming bug, a nil pointer you should never have dereferenced, an invariant that's been violated so badly that continuing execution would be worse than crashing:

func mustParseConfig(raw string) Config {
    cfg, err := parseConfig(raw)
    if err != nil {
        panic(fmt.Sprintf("invalid config, cannot start: %v", err))
    }
    return cfg
}

Notice the must prefix — that's a real Go convention signaling "this panics instead of returning an error, use it only when failure should be impossible or fatal." You'll see it on things like regexp.MustCompile, used for regex patterns that are hardcoded at compile time and should never actually fail to parse. If a function can fail in a way the caller might reasonably want to handle, it returns an error. panic is for "this program cannot continue," not "this operation didn't work."

Where I've Landed

A few weeks in, the repetition stopped bothering me, for a specific reason: I always know exactly where my code can fail, just by reading it. No exception is escaping from three function calls deep and surfacing somewhere I didn't expect. The trade-off is real — Go code has more visible lines dedicated to error checking than the equivalent Java would — but every one of those lines is telling me something true about the function I'm calling, instead of trusting that a try/catch block somewhere upstream will catch what I forgot to handle.

What still isn't fully natural yet: remembering to wrap errors with enough context every single time (it's easy to return a bare err and lose the "where did this happen" information), and resisting the urge to define a sprawling hierarchy of custom error types out of old OOP habits when a couple of sentinel errors would do.

Up Next

With types, concurrency, and error handling covered, part 5 is where this series gets concrete: building a real REST API with Gin, including the error-handling and concurrency patterns from the last two posts actually wired into request handlers.

If you've made the jump from exceptions to Go's explicit error returns, did the repetition bother you at first the way it bothered me — and did it fade the same way?

Goroutines & Channels — Concurrency Without the JVM's Baggage

mihir mohapatra — Sat, 20 Jun 2026 08:44:14 +0000

Goroutines & Channels — Concurrency Without the JVM's Baggage

In part 2 I went through Go's type system — structs, implicit interfaces, and composition replacing inheritance entirely. This time I'm tackling the thing that actually pulled me toward Go in the first place: concurrency.

If you've spent years on the JVM like I have, "concurrency" usually means a mental checklist — ExecutorService, thread pool sizing, Future<T>, maybe CompletableFuture chains, and more recently virtual threads if you're on a modern JDK. It's powerful, but it's also a lot of ceremony for what's often a simple idea: "do these things at the same time, then collect the results." Go strips almost all of that ceremony away, and that's been the most genuinely refreshing part of this whole learning process.

Goroutines: Threads, But Cheap

A goroutine is Go's unit of concurrent execution. The syntax is almost suspiciously simple — you put go in front of a function call, and it runs concurrently:

func main() {
    go sayHello("world")
    time.Sleep(50 * time.Millisecond) // give it time to run
}

func sayHello(name string) {
    fmt.Println("Hello,", name)
}

That time.Sleep hack is the first lesson everyone learns the hard way: main doesn't wait for goroutines to finish. If main exits, every goroutine it spawned just gets cut off mid-flight. We'll fix that properly in a moment with sync.WaitGroup, but the sleep hack is worth seeing once because it makes the "goroutines don't block their caller" behavior concrete.

What makes goroutines genuinely different from JVM threads isn't the syntax — it's the cost. A JVM thread typically reserves around a megabyte of stack space and is managed by the OS scheduler. A goroutine starts with a stack of about 2KB that grows as needed, and Go's own runtime scheduler multiplexes thousands of them onto a small number of OS threads. Spinning up 100,000 goroutines is a normal thing to do in Go. Spinning up 100,000 JVM platform threads will bring most systems to their knees — which is exactly the problem virtual threads were introduced to solve in recent JDKs. Go just had this answer from day one.

WaitGroups: The Right Way to Wait

Instead of sleeping and hoping, sync.WaitGroup lets you wait for a known number of goroutines to actually finish:

func main() {
    var wg sync.WaitGroup

    services := []string{"orders", "payments", "inventory"}

    for _, name := range services {
        wg.Add(1)
        go func(s string) {
            defer wg.Done()
            fetchFromService(s)
        }(name)
    }

    wg.Wait()
    fmt.Println("all services responded")
}

func fetchFromService(name string) {
    time.Sleep(100 * time.Millisecond)
    fmt.Println(name, "done")
}

wg.Add(1) registers one goroutine to wait for, defer wg.Done() marks it complete when the function returns, and wg.Wait() blocks until the count hits zero. Notice the name is passed as a parameter into the closure rather than captured directly — that's intentional. Capturing the loop variable directly is a classic Go footgun (pre-1.22, every closure would share the same variable and you'd get the last value for everything). Passing it as an argument sidesteps that entirely, and is still good practice to know even on newer Go versions where the loop variable semantics changed.

Channels: Goroutines Talking to Each Other

WaitGroups tell you when things are done. Channels let goroutines actually communicate — passing data between them safely, without manual locks:

func main() {
    results := make(chan string, 3)

    go fetchAndSend("orders", results)
    go fetchAndSend("payments", results)
    go fetchAndSend("inventory", results)

    for i := 0; i < 3; i++ {
        fmt.Println(<-results)
    }
    close(results)
}

func fetchAndSend(name string, results chan<- string) {
    time.Sleep(100 * time.Millisecond)
    results <- fmt.Sprintf("%s: 200 OK", name)
}

A channel is a typed pipe. results <- value sends, <-results receives, and both block until someone's on the other end (unless the channel is buffered, like the make(chan string, 3) above, which allows up to 3 sends before blocking). The chan<- string parameter type is a send-only channel — a nice bit of compiler-enforced documentation saying "this function only writes here, it never reads."

This is the part that reframes concurrency entirely coming from Java. The Go proverb is "don't communicate by sharing memory, share memory by communicating" — instead of multiple threads fighting over a synchronized block or a ConcurrentHashMap, you pass ownership of data through a channel, one direction at a time. There's no lock to forget, because there's nothing shared to protect.

select: Handling Multiple Channels at Once

Once you have more than one channel, select lets you react to whichever one is ready first — it's the concurrency equivalent of a switch statement:

func main() {
    payments := make(chan string)
    timeout := time.After(2 * time.Second)

    go processPayment(payments)

    select {
    case result := <-payments:
        fmt.Println("payment result:", result)
    case <-timeout:
        fmt.Println("payment timed out")
    }
}

func processPayment(payments chan<- string) {
    time.Sleep(3 * time.Second) // simulate a slow downstream call
    payments <- "success"
}

This pattern — racing a real channel against time.After — is how Go handles timeouts idiomatically. No Future.get(timeout, unit) with a checked TimeoutException to catch. Just two channels and a select.

Where This Actually Bites You

I don't want to oversell this as friction-free, because it isn't. A few things I've already tripped over:

Goroutine leaks are real and easy to cause. If a goroutine is blocked sending on a channel that nobody's ever going to read from again, it just sits there forever, leaking memory. Unlike a thread pool with bounded size, there's no natural backpressure unless you design it in yourself — usually with buffered channels, context.Context cancellation, or both.

Deadlocks look different than JVM deadlocks. Instead of two threads each holding a lock the other wants, a Go deadlock is more often "every goroutine is blocked waiting on a channel, and nothing's left to unblock them." The runtime is actually good at detecting this for the simplest cases (fatal error: all goroutines are asleep - deadlock!), which is more help than the JVM ever gave me.

Unbuffered channels make ordering subtle. An unbuffered channel send blocks until a receiver is ready, which means the timing of goroutines relative to each other isn't always obvious from reading the code top to bottom. It took a few go run experiments with print statements before the mental model clicked.

go vet and the race detector are not optional. Running go run -race during development catches data races that would otherwise be silent and only show up under load in production. I now run it by default the way I'd run a linter.

Why This Matters for the Series

This concurrency model is going to show up constantly once I build a real API in part 5 — handling concurrent requests, fanning out to multiple downstream services, and managing timeouts cleanly are exactly what channels and select are built for. It's also a big part of why Go shows up so often in infra and backend tooling: the concurrency primitives are cheap enough that you reach for them by default, not just when you've hit a performance wall.

Next up in part 4: error handling. I promised in part 1 that if err != nil would get its own post, and after spending real time with goroutines that return errors over channels, I have a lot more sympathy for Go's approach than I expected to.

If you've worked with both JVM threads and goroutines, I'd like to know — did the "share memory by communicating" mental shift come naturally, or did you find yourself reaching for locks out of habit at first?

Go's Type System — Structs, Interfaces, and Life Without Inheritance

mihir mohapatra — Fri, 19 Jun 2026 06:53:06 +0000

Go's Type System — Structs, Interfaces, and Life Without Inheritance

In part 1 of this series I talked about why I'm picking up Go after six years of Java and Kotlin, plus a recent deep dive into Rust. This time I want to get into the part that actually changed how I think about designing code: Go has no class inheritance at all.

Coming from the JVM world, that sentence sounded alarming the first time I read it. No extends. No abstract classes. No polymorphism through a class hierarchy. And yet Go backends at companies running serious scale seem to do just fine without it. After a few weeks living inside Go's type system, I get why.

Structs: Data, Nothing More

A Go struct is just a typed bag of fields. No constructors, no access modifiers in the Java sense, no inheritance:

type Order struct {
    ID       string
    Customer string
    Amount   float64
    Status   string
}

func NewOrder(id, customer string, amount float64) Order {
    return Order{
        ID:       id,
        Customer: customer,
        Amount:   amount,
        Status:   "pending",
    }
}

That NewOrder function is doing the job a constructor would do in Java — it's just a plain function by convention, not a language feature. Nothing stops you from building an Order{} directly with zero values either, which takes some adjusting to if you're used to constructors enforcing invariants.

Methods attach to structs separately, outside the type definition:

func (o Order) Total() float64 {
    return o.Amount
}

func (o *Order) MarkPaid() {
    o.Status = "paid"
}

That (o Order) vs (o *Order) distinction is the receiver type, and it trips up a lot of newcomers. A value receiver gets a copy of the struct; a pointer receiver can mutate the original. MarkPaid has to use a pointer receiver, or the status change would vanish the moment the method returns.

No Inheritance, So What Replaces It?

This is the part that took the most rewiring. In Java, if PremiumOrder needed everything Order had plus more, you'd write class PremiumOrder extends Order. Go simply doesn't have that mechanism. Instead, you embed:

type PremiumOrder struct {
    Order
    Priority int
}

Embedding Order inside PremiumOrder promotes its fields and methods up to the outer struct. So premiumOrder.Total() and premiumOrder.Status just work, as if they were defined directly on PremiumOrder. It looks like inheritance from the outside, but it's composition under the hood — PremiumOrder has an Order, it doesn't become one. That distinction matters the moment you need polymorphism, which is where interfaces take over.

Interfaces: The Part That Actually Surprised Me

Here's the genuinely different idea in Go's type system. In Java or Kotlin, a class declares upfront which interfaces it implements — class PaymentProcessor implements Refundable. The relationship is explicit and lives at the definition site.

In Go, interfaces are satisfied implicitly. There's no implements keyword at all:

type Refundable interface {
    Refund(amount float64) error
}

type CardPayment struct {
    TransactionID string
}

func (c CardPayment) Refund(amount float64) error {
    fmt.Printf("Refunding %.2f for transaction %s\n", amount, c.TransactionID)
    return nil
}

CardPayment never mentions Refundable anywhere. It just happens to have a Refund(amount float64) error method, so it satisfies the interface automatically. Any type, anywhere in the codebase — even one you didn't write — can satisfy an interface it never knew existed:

func processRefund(r Refundable, amount float64) error {
    return r.Refund(amount)
}

processRefund(CardPayment{TransactionID: "txn_123"}, 49.99)

The first time I saw this, it felt like it would lead to chaos — how do you know what implements what? But in practice it flips the design direction in a useful way. Interfaces in Go tend to be small and defined by the consumer, not the producer. You don't design a sprawling PaymentProcessor interface upfront and force every payment type to conform to it. You write the function that needs a Refund method, declare a one-method interface for exactly that, and let any matching type plug in. The standard library leans hard into this — io.Reader and io.Writer are each a single method, and half of Go's ecosystem is built by composing those tiny interfaces together.

Putting It Together: Composition Over Inheritance, For Real

The Java-world mantra "favor composition over inheritance" is something I nodded along to for years while still reaching for extends out of habit. Go doesn't give you the option to reach for it — composition is the only tool on the table, so you actually use it.

A small example of how this plays out: instead of a BasePaymentHandler abstract class with CardHandler and BankTransferHandler subclasses overriding behavior, you'd define the shape you need as an interface and write independent structs that satisfy it:

type PaymentHandler interface {
    Process(amount float64) error
}

type CardHandler struct{}
func (CardHandler) Process(amount float64) error {
    fmt.Println("Processing card payment:", amount)
    return nil
}

type BankTransferHandler struct{}
func (BankTransferHandler) Process(amount float64) error {
    fmt.Println("Processing bank transfer:", amount)
    return nil
}

func charge(h PaymentHandler, amount float64) error {
    return h.Process(amount)
}

No shared base class, no super.Process() calls, no fragile hierarchy to untangle later when a new payment type doesn't quite fit the existing tree. Each handler is independent; the interface is just a contract describing what charge needs.

Where This Still Feels Unfamiliar

A few things haven't fully clicked yet, and I'd rather be honest about that than pretend the transition was seamless:

Implicit satisfaction makes "what implements this?" harder to answer. In an IDE-heavy Java workflow, "find implementations" is one click. In Go, you're often grepping for method signatures, since there's no declared relationship to search for. Tooling helps, but it's a different mental habit.

Embedding isn't inheritance, and pretending otherwise causes bugs. It's tempting to treat PremiumOrder embedding Order as "PremiumOrder is an Order," but method promotion doesn't give you real polymorphism — you can't pass a PremiumOrder somewhere that explicitly expects an Order type. It works at the field/method-access level only.

Zero values mean "uninitialized" can silently look valid. A struct field you forgot to set isn't null and isn't an error — it's just the zero value ("", 0, false). That's elegant in some ways and a quiet source of bugs in others, especially coming from a world where constructors enforce required fields.

Up Next

This composition-first, implicit-interface design is going to matter a lot once concurrency enters the picture, since goroutines and channels lean on small interfaces too. In part 3, I'm digging into goroutines and channels — and where Go's concurrency model genuinely outperforms the thread-pool mental model I'm used to from the JVM, plus where it bites if you're not careful.

If you've made the jump from Java, Kotlin, or another OOP-heavy language to Go, I'm curious — did implicit interfaces feel natural to you immediately, or did it take a few weeks to stop reaching for implements?

Why I'm Learning Go in 2026 (A Java/Kotlin/Rust Engineer's Take)

mihir mohapatra — Thu, 18 Jun 2026 11:07:12 +0000

Why I'm Learning Go in 2026 (A Java/Kotlin/Rust Engineer's Take)

I've spent the better part of six years writing Java and Kotlin in production — FinTech systems, logistics platforms, EdTech backends. Somewhere in the last year I also went deep on Rust, building an AI gateway and a RAG server from scratch. So the obvious question is: why pick up another language?

Short answer: because every team I want to work for in 2026 is either already running Go in production, or wishes they were.

This post kicks off a series where I document learning Go properly — not "hello world and a for loop" learning, but the kind that ends with a real, production-grade service. If you're a backend engineer coming from the JVM world (or from Rust, or anywhere else) and curious whether Go is worth the detour, this series is for you.

The Itch That Started This

Most of my recent work has been in Rust, and I love it — the type system catches entire categories of bugs before they ship, and the performance ceiling is hard to beat. But Rust also makes you pay for that safety in compile times, borrow-checker fights, and a learning curve that scares away a chunk of teams who'd otherwise want those benefits.

Go takes a different bet: give up some of that compile-time rigor, and in exchange get a language so simple you can read a stranger's codebase on day one. After enough job descriptions and system design conversations mentioning Go in the same breath as "we need to move fast and onboard people quickly," I stopped arguing with the trend and decided to just learn it properly.

What Go Actually Optimizes For

Coming from Java and Kotlin, the first thing that struck me wasn't a feature — it was an absence. No class hierarchies to design upfront. No generics-heavy abstractions to reach for by default. No build tool wars (Maven vs Gradle vs whatever). Go's design philosophy is almost stubbornly opinionated: there's one way to format code (gofmt decides, not you), one way to manage dependencies (Go modules), and a standard library that already does most of what you'd otherwise reach for a framework to do.

A quick side-by-side of what's missing compared to the JVM world tells you a lot about Go's priorities:

JVM World	Go World
Class inheritance	Composition + interfaces
Checked/unchecked exceptions	Explicit `error` return values
Threads + thread pools	Goroutines + channels
Maven/Gradle build config	`go.mod`, almost no config
Annotations for DI, validation, etc.	Plain functions, explicit wiring

None of this means Go is "Java but simpler" — it's a genuinely different way of thinking about structuring a backend service. The interface system in particular is going to get its own post, because it inverts how I'm used to designing for abstraction.

Concurrency Was the Real Hook

If I'm honest, goroutines are why I stopped procrastinating on this. In Java, concurrency means thinking about thread pools, ExecutorService, and increasingly, virtual threads if you're on a recent JDK. In Go, concurrency is baked into the language itself:

func main() {
    results := make(chan string)

    go fetchFromService("orders", results)
    go fetchFromService("payments", results)
    go fetchFromService("inventory", results)

    for i := 0; i < 3; i++ {
        fmt.Println(<-results)
    }
}

func fetchFromService(name string, results chan<- string) {
    // simulate a network call
    time.Sleep(100 * time.Millisecond)
    results <- fmt.Sprintf("%s: done", name)
}

Three goroutines, a channel to coordinate them, no thread pool configuration, no Future<T> wrapping. That's the entire concurrency model for a huge percentage of real-world backend use cases. I'll go deeper into goroutines and channels — including where they bite you — in part 3.

Where I Think Go Will Push Back

I don't want this series to read like a sales pitch, so here's what's already bothering me a few weeks in:

Error handling feels repetitive. Coming from exceptions, writing if err != nil { return err } after every single call feels almost aggressively manual. I know there are good reasons for it (explicit control flow, no hidden stack unwinding), but my fingers haven't agreed yet.

No generics-first mindset (still adjusting). Generics arrived in Go 1.18, but the standard library and most idiomatic code still leans on interfaces and any in ways that feel like a step back from Kotlin's type system or Rust's traits.

Smaller batteries-included story for some things. No built-in ORM, no Spring-style dependency injection container, no Rocket/Axum-style ergonomic web framework baked into std (though Gin and Echo close that gap fast).

I expect most of these complaints to soften as I get further in — they usually do.

What This Series Will Actually Build

Rather than a string of disconnected toy examples, this series builds toward one real artifact: a production-grade Go backend service, the same way I approached rust-ai-gateway and BlazeRAG. Here's the planned route:

Part 2 — Go's type system: structs, interfaces, and composition over inheritance
Part 3 — Goroutines and channels, and where naive concurrency breaks
Part 4 — Error handling patterns that don't make your code unreadable
Part 5 — Building a real REST API with Gin
Part 6 — Testing: table-driven tests, benchmarks, and go test habits
Part 7 — Turning all of it into a production-grade service, ready to ship

If you're also coming from Java, Kotlin, or Rust and weighing whether Go is worth the time, follow along — I'll be honest about what's genuinely better, what's just different, and what I still miss from the languages I came from.

What's been your experience picking up Go from a JVM or Rust background? I'd genuinely like to know what surprised you, good or bad.