DEV Community: Ameer Hamza

Fine-Tuning LLMs for Backend Engineers

Ameer Hamza — Tue, 11 Aug 2026 15:27:37 +0000

Introduction

Fine-tuning does not teach your model new facts. It teaches your model a new behavior.

When you fine-tune an LLM, you adjust weights so the model responds in a specific style, format, or domain pattern. You are not uploading a knowledge base. The model still cannot reliably recall facts it was never trained on. It gets better at sounding like your use case.

This is why fine-tuning a support bot does not replace a knowledge base. The model learns to respond like a support agent. It does not learn your product documentation.

RAG gives the model facts at query time. Fine-tuning shapes how it uses them.

Why This Matters

Teams choose fine-tuning when prompt engineering and RAG are insufficient. That is valid. But choosing fine-tuning to "add knowledge" leads to stale answers, high retraining cost, and compliance risk when documentation changes weekly.

Backend engineers should treat fine-tuning as a behavior adapter in the system architecture, not a database replacement.

Prerequisites

This article assumes you have read Blog 001 and Blog 002. You should understand LLM inference and RAG for external knowledge.

The Problem

Common misconceptions:

"Fine-tune on our docs" to avoid building RAG. Docs change; weights do not update cheaply.
"Fine-tune once, done forever." Model behavior drifts; eval suites go stale.
"Bigger fine-tune is better." Full fine-tune on small data overfits and forgets general capability.
Ignoring data leakage between train and eval sets, inflating offline metrics.

Understanding the Core Concept

Behavior vs knowledge

Goal	Better approach	Why
Answer from current docs	RAG	Facts update by re-indexing
Consistent JSON output format	Fine-tune or constrained decoding	Shape response structure
Domain tone and terminology	Fine-tune	Style is behavioral
Tool-calling patterns	Fine-tune on trajectories	Teaches action selection
Reduce prompt length	Fine-tune	Bakes instructions into weights

Fine-tuning methods

Full fine-tuning updates all model weights. Highest flexibility, highest GPU memory and risk of catastrophic forgetting.

Parameter-efficient fine-tuning (PEFT) updates a small adapter (LoRA, QLoRA). Most production fine-tunes use this: train adapters on consumer or single-GPU setups, merge or hot-swap at serving time.

Instruction tuning is fine-tuning on (instruction, response) pairs to follow commands better. Alignment tuning (RLHF, DPO) shapes helpfulness and safety preferences.

Training data quality

Fine-tuning amplifies your dataset. Noisy examples become noisy behavior. Duplicates overweight certain patterns. Incorrect labels teach incorrect outputs. Invest in curation before GPUs.

How It Works Internally (High Level)

Start from a pretrained base model.
Prepare supervised examples (prompt, completion) in chat format.
Forward pass computes loss on completion tokens only (mask prompt tokens).
Backprop updates weights (full or LoRA adapters).
Evaluate on held-out set unrelated to training prompts.
Export merged weights or adapter checkpoints.
Deploy behind the same inference API with version tracking.

Step-by-Step Example

Goal: Support bot always responds in a structured format with empathy and escalation tags.

Collect 2,000 real ticket transcripts (redact PII).
Label ideal responses with {summary, action, escalate: bool}.
Fine-tune with LoRA on a 7B open model.
Keep RAG for product facts from the knowledge base.
At inference: retrieve docs, inject into prompt, fine-tuned model formats the answer.
Run eval: format validity, escalation accuracy, faithfulness to retrieved context.

Fine-tuning handles format and tone. RAG handles facts.

Architecture

Python Example

Illustrative LoRA training setup with Hugging Face PEFT (conceptual structure).

"""
Illustrative LoRA fine-tune setup.
Requires: pip install transformers peft datasets torch
"""
from datasets import Dataset
from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments, Trainer
from peft import LoraConfig, get_peft_model

BASE_MODEL = "meta-llama/Llama-3.2-1B-Instruct"

examples = [
    {
        "instruction": "Summarize the refund policy briefly.",
        "output": '{"summary": "14-day refund window for annual plans.", "escalate": false}',
    },
    {
        "instruction": "Customer threatens legal action over billing.",
        "output": '{"summary": "Acknowledge concern, escalate to legal.", "escalate": true}',
    },
]

def format_example(row: dict) -> dict:
    text = (
        f"<|user|>{row['instruction']}<|assistant|>{row['output']}"
    )
    return {"text": text}

dataset = Dataset.from_list(examples).map(format_example)

tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)
model = AutoModelForCausalLM.from_pretrained(BASE_MODEL)

lora_config = LoraConfig(
    r=8,
    lora_alpha=16,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.05,
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora_config)

def tokenize(batch):
    return tokenizer(batch["text"], truncation=True, max_length=512)

tokenized = dataset.map(tokenize, batched=True)

training_args = TrainingArguments(
    output_dir="./lora-out",
    num_train_epochs=3,
    per_device_train_batch_size=1,
    learning_rate=2e-4,
    logging_steps=1,
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized,
)
# trainer.train()  # Uncomment with GPU and model access
print(f"Trainable params: {model.print_trainable_parameters()}")

Run on GPU hardware with proper eval splits before production deployment.

Real-World Applications

JSON extraction with consistent schema adherence
Domain-specific tone (legal, medical chart notes) with compliance review
Function-calling fine-tunes for reliable tool selection
Distilling a large teacher model into a smaller student for routing tiers

Performance Considerations

Training cost: GPU hours scale with model size, dataset size, and epochs.
Serving cost: Merged fine-tunes share the same inference profile as the base model. Adapters add small overhead.
Update cadence: RAG re-index is hours. Fine-tune retrain is days and requires ML workflow.
Regression risk: Always run a golden eval suite before promoting a new checkpoint.

Common Mistakes

Fine-tuning to store facts that change frequently.
Training on synthetic data without human review.
No held-out eval set with production-like prompts.
Full fine-tune on tiny datasets.
Deploying without rollback to previous model version.

Interview Questions

Q1: What does fine-tuning change?

A: Model weights to adapt behavior, style, or format. Not a reliable store for volatile factual knowledge.

Q2: Fine-tuning vs RAG?

A: RAG injects external facts at query time. Fine-tuning shapes how the model responds. Often used together.

Q3: What is LoRA?

A: Low-Rank Adaptation: trains small additive matrices instead of all weights, reducing memory and cost.

Q4: What is catastrophic forgetting?

A: Fine-tuning degrades general capabilities when the dataset is narrow or training is aggressive.

Q5: How do you evaluate a fine-tuned model?

A: Held-out task metrics, format compliance, safety checks, and comparison against base model plus prompt baseline.

Q6: When is fine-tuning not worth it?

A: When prompt engineering plus RAG meets quality bars, or when you lack curated training data and eval infrastructure.

Data Pipeline for Fine-Tuning

Production fine-tuning requires MLOps discipline:

Source: Production logs (redacted), human-edited ideal responses, synthetic data (reviewed).
Format: Chat templates matching inference (system, user, assistant roles).
Split: Train/val/test with no document overlap for RAG-coupled eval.
Train: LoRA with early stopping on val loss.
Eval: Automated plus human review on safety and format.
Register: Model card with dataset hash, base model version, metrics.
Deploy: Canary with rollback to base plus adapter off.

Synthetic data risks

LLM-generated training data can teach the model its own failure modes. If synthetic, use a stronger teacher model, human spot-check, and diversity filters.

When to prefer prompts over fine-tuning

Fewer than 500 high-quality examples
Behavior changes weekly
You need explainability of instructions in the prompt
Multiple behaviors toggled per tenant (prompt flags beat N adapters)

Fine-tuning wins when you need consistent format, reduced prompt length, or domain tone at scale.

LoRA Hyperparameters (Starting Points)

Hyperparameter	Typical range	Notes
rank (r)	8-64	Higher rank, more capacity, more overfit risk
alpha	2x rank	Scaling factor
learning rate	1e-5 to 3e-4	Lower for larger bases
epochs	1-5	Stop early on val loss

Always evaluate on tasks different from training paraphrases to detect memorization.

Full Fine-Tune vs LoRA Decision

Factor	LoRA	Full fine-tune
GPU memory	Lower	High
Training time	Shorter	Longer
Behavior shift depth	Moderate	Deep
Catastrophic forgetting risk	Lower	Higher
Serving	Adapter merge or sidecar	Single weight blob

For most product teams, LoRA on an open base model covers format and tone needs.

Combining with RAG at Inference

system: policies + citation rules
user: question
retrieved: top chunks
model: fine-tuned for JSON + support tone

Fine-tune teaches how to format the answer. RAG supplies what the answer should reference.

Dataset Size Guidelines (Rules of Thumb)

Goal	Examples needed (order of magnitude)
Tone adjustment	500-2,000
Format compliance	1,000-5,000
Domain terminology	2,000-10,000 plus RAG
New factual domain	Prefer RAG, not fine-tune

Quality beats quantity. 500 expert-labeled examples outperform 10,000 noisy synthetic ones.

Legal and Compliance

Fine-tuning on customer data may implicate:

Consent for training use
Data retention policies
Right to deletion (can you unlearn?)

Document what data entered each training run. Adapters are smaller artifacts but still encode training data signals.

Rollback Strategy

Keep N-1 adapter checkpoint hot-swappable. Feature flag routes percentage traffic to new adapter. Automatic rollback if format error rate spikes.

Handoff Between ML and Platform Teams

ML team delivers:

Adapter checkpoint with eval report
Training data manifest hash
Known failure cases

Platform team owns:

Inference integration
Canary and rollback
Production monitoring

Without handoff checklist, fine-tunes ship without rollback paths.

Cost of Ownership

Include in fine-tune ROI:

GPU training hours
Labeling cost
Ongoing eval compute
Engineer review before each retrain

If sum exceeds prompt+RAG iteration cost over 12 months, delay fine-tune.

Storage and Artifact Management

Store adapters in object storage with metadata: base model hash, training commit, dataset version, eval scores, training hyperparameters. Inference servers load adapter by version tag on deploy. Never overwrite adapter blobs in place; immutable artifacts enable rollback.

For regulated industries, maintain training data lineage for audit: which customer data entered which run, with retention expiry.

Reference Appendix: Production FAQ

How do I know this is working in production?

Instrument the layer this article describes before changing models or prompts. Compare p50 and p95 latency, error rate, and task-specific quality scores week over week. AI regressions are subtle: flat aggregate uptime can hide wrong answers.

What is the first config change to try?

Reduce variability before increasing capability. Lower temperature for factual paths, shrink retrieval top-K, tighten context budgets, add output validation. Complexity is not a substitute for measurement.

What belongs in an on-call runbook?

Symptom, dashboard link, rollback lever (model version, feature flag, index snapshot), owner team, and customer communication template. LLM incidents need content rollback, not only service restart.

How do I explain tradeoffs to product managers?

Use dollars and seconds: cost per successful task, p95 time to first token, accuracy on golden set. Avoid debating model intelligence; debate measurable user outcomes and failure tolerance.

When should we retrain, re-index, or rewrite prompts?

Re-index when documents change. Rewrite prompts when behavior spec changes. Retrain or fine-tune when prompt plus RAG cannot meet format or tone requirements after eval iteration. Default order: prompt, RAG, fine-tune.

What is the common rollback path?

Keep previous model version, previous index snapshot, and previous prompt template addressable by version id for at least seven days. Rollback should be one feature flag or deploy revert, not a fire drill.

How does this interact with the rest of the handbook?

This topic is one layer in a stack. Read prerequisites listed in frontmatter. When debugging end-to-end failures, walk the request path from ingress through retrieval, inference, and output validation before concluding the model is wrong.

Summary

Change behavior with fine-tuning. Change facts with retrieval. Treat fine-tuning as a deployment artifact with versioning, eval gates, and clear ownership, not a one-time data upload.

Next in Series

Blog 006: LLM Quantization: Precision, Memory, and Production Tradeoffs

AI Agents Explained for Backend Engineers

Ameer Hamza — Tue, 11 Aug 2026 15:20:40 +0000

Introduction

AI agents are not intelligent. They are persistent.

An agent is a loop: the LLM thinks, picks a tool, executes it, reads the result, and thinks again. It continues until the task completes, a stop condition fires, or your budget runs out.

There is no built-in planning module. No memory of past failures unless you add it. No self-correction unless you engineer it. It is a while-loop wrapped around an API call.

That is what makes agents powerful and dangerous. They keep trying. They burn tokens. They call the wrong tool twice. They loop forever if you do not set exit conditions.

Why This Matters

Backend engineers know that unbounded loops are production incidents waiting to happen. Agents are unbounded loops with a non-deterministic decision function.

A runaway agent is not a research curiosity. It is a line item on your cloud bill and a reliability risk for downstream systems. Persistence without guardrails is an expensive infinite loop.

Prerequisites

This article assumes you have read Blog 001, Blog 002, and Blog 003. You should understand LLM inference, RAG for grounding, and MCP for standardized tool access.

The Problem

Teams ship agents by wrapping an LLM in a ReAct-style prompt and calling it done. Production breaks when:

No iteration cap: Agent calls tools indefinitely.
No token budget: A single user request costs dollars in API fees.
No timeout: Hung tool calls block the loop.
No idempotency: Retried tool calls double-charge or duplicate writes.
No human gate for irreversible actions (refunds, deletes, deployments).

Understanding the Core Concept

The agent loop

while not done:
    response = llm(messages, tools)
    if response.has_tool_call:
        result = execute_tool(response.tool_call)
        messages.append(result)
    else:
        return response.text

The LLM decides what to do next based on conversation history and tool results. Quality depends on prompts, tools, and guardrails, not on hidden reasoning.

Agent vs single LLM call

Aspect	Single LLM call	Agent loop
Latency	One round trip	Multiple round trips
Cost	Fixed per call	Scales with iterations
Capability	Text in, text out	Can act on external systems
Failure modes	Bad output	Bad output plus bad actions
Observability	Simple	Requires per-step tracing

Use an agent when the task requires multiple steps, external data, or actions. Use a single call when retrieval plus generation suffices.

Harness responsibilities

The harness (your code around the model) must provide:

Termination policy: max iterations, max tokens, wall-clock timeout
Tool dispatch: schema validation, auth, retries
State management: conversation history, scratchpad, episodic memory
Error handling: surface tool errors to the model or escalate
Human-in-the-loop: approval for high-risk operations

How It Works Internally (High Level)

User submits a goal.
Harness assembles system prompt, tool catalog, and user message.
LLM returns either a final answer or a tool call.
If tool call: harness validates, executes via MCP or direct API, appends result.
Loop until final answer or stop condition.
Harness logs full trace for debugging and billing.

Step-by-Step Example

Task: "Find the latest error rate for service checkout and post a summary to #incidents."

Iteration 1: Model calls metrics_query(service="checkout", window="1h").
Harness executes, returns error_rate: 4.2%.
Iteration 2: Model calls slack_post(channel="#incidents", message="...").
Harness checks approval policy, posts message.
Iteration 3: Model returns "Posted summary to #incidents."
Harness terminates. Total: 3 LLM calls, 2 tool calls.

Without a max iteration limit, a confused model might query metrics repeatedly.

Architecture

Exit conditions on the loop are not optional.

Python Example

Minimal agent harness with iteration cap and token budget.

"""
Minimal agent harness with guardrails.
"""
import json
from dataclasses import dataclass, field

MAX_ITERATIONS = 5
MAX_TOTAL_CHARS = 8000

@dataclass
class AgentState:
    messages: list[dict] = field(default_factory=list)
    iterations: int = 0
    total_chars: int = 0

def mock_llm(messages: list[dict], tools: list[dict]) -> dict:
    last = messages[-1]["content"] if messages else ""
    if "error rate" in last.lower() and not any("metrics" in str(m) for m in messages):
        return {
            "type": "tool_call",
            "name": "metrics_query",
            "arguments": {"service": "checkout", "window": "1h"},
        }
    if any("error_rate" in str(m) for m in messages):
        return {"type": "text", "content": "Checkout error rate is elevated at 4.2%."}
    return {"type": "text", "content": "I need more information."}

TOOLS = {
    "metrics_query": lambda args: {"error_rate": "4.2%", "service": args["service"]},
}

def run_agent(user_message: str) -> str:
    state = AgentState(messages=[{"role": "user", "content": user_message}])

    while state.iterations < MAX_ITERATIONS:
        state.iterations += 1
        response = mock_llm(state.messages, list(TOOLS.keys()))

        if response["type"] == "tool_call":
            name = response["name"]
            result = TOOLS[name](response["arguments"])
            state.messages.append({"role": "tool", "content": json.dumps(result)})
            state.total_chars += len(json.dumps(result))
        else:
            return response["content"]

        if state.total_chars > MAX_TOTAL_CHARS:
            return "Error: token budget exceeded"

    return "Error: max iterations exceeded"

if __name__ == "__main__":
    print(run_agent("What is the checkout error rate?"))

Replace mock_llm with your provider client. Add structured logging per iteration.

Real-World Applications

Code agents that read files, run tests, and open pull requests
Support agents that query CRM, search docs (RAG), and draft replies
Data agents that generate SQL, execute read-only queries, and summarize
DevOps agents that inspect logs and trigger approved runbooks

Performance Considerations

Latency: Each iteration adds a full LLM round trip plus tool execution time.
Cost: Token usage grows with history length. Summarize or prune old tool results.
Concurrency: Multiple agents sharing tools need per-tenant rate limits.
Reliability: Tool failures should be explicit in context, not swallowed.

Common Mistakes

No max iterations or timeout.
Giving agents write access without approval workflows.
Passing entire tool outputs into context without truncation.
Assuming the model will self-correct after repeated failures.
No distributed trace ID across LLM and tool calls.

Interview Questions

Q1: What is an AI agent in production terms?

A: A loop where an LLM repeatedly decides to call tools or return a final answer until a harness stop condition fires.

Q2: Why are agents expensive?

A: Each iteration is a full LLM inference call plus tool execution, with growing conversation history.

Q3: What guardrails are mandatory?

A: Max iterations, timeout, token budget, schema-validated tool calls, and human approval for irreversible actions.

Q4: Agent vs RAG: when to use which?

A: RAG grounds answers in documents. Agents take multi-step actions. Many systems use both.

Q5: What is the harness?

A: Your application code that manages the loop, tools, state, policies, and observability around the LLM.

Q6: How do you debug a bad agent outcome?

A: Inspect the per-iteration trace: which tools were called, with what arguments, and what the model saw at each step.

Termination Policies

Define explicit exit conditions in code, not in prompts alone:

Stop reason	Trigger	User experience
success	Model returns final answer	Normal completion
max_iterations	Loop count exceeded	Partial result plus explanation
budget_exceeded	Token or cost cap hit	Graceful degradation message
timeout	Wall clock limit	Retry suggestion
human_required	High-risk tool pending	Escalation UI
tool_failure_limit	N consecutive tool errors	Stop and log incident

Prompts that say "stop when done" are insufficient. Models do not reliably self-terminate under ambiguity.

State management options

Ephemeral: Full history in memory per request. Simple, no cross-session memory.

Session store: Redis or DB keyed by session ID. Required for multi-turn agents.

Scratchpad: Separate channel for intermediate reasoning and tool JSON, trimmed before user display.

Summarized memory: Compress turns older than K into a rolling summary to save context (Blog 012).

Cost model for agents

Approximate cost per task:

cost ≈ sum(iteration_i prompt_tokens + completion_tokens) × price_per_token

A five-iteration agent with 4K context per iteration is not five times a single 4K call if history grows each loop. Cap tool result size aggressively.

ReAct and Tool-Calling Patterns

ReAct interleaves reasoning text with tool calls in the transcript. The model emits natural language planning, then a tool invocation, then observes results.

Production tip: separate user-visible messages from internal tool traces. Users do not need raw JSON blobs; they need summaries.

Parallel vs sequential tools

Some APIs allow parallel tool calls. Rules:

Parallelize only independent reads
Serialize writes that touch the same resource
Define merge order when results conflict

Human-in-the-loop placement

Insert approval gates before:

Financial transactions
Data deletion
External emails
Production config changes

Return a pending state to the UI instead of blocking inside the model loop indefinitely.

Testing Agents

Unit test tool dispatch without the LLM:

Given mocked model output tool_call X, assert handler X runs with validated args.

Integration test with frozen model responses (record/replay) for deterministic CI.

Eval in staging with real models on golden tasks measuring success rate, average iterations, and cost per task.

Observability for Agent Traces

Structure logs as an ordered trace:

{
  "trace_id": "abc",
  "steps": [
    {"type": "llm", "tokens_in": 1200, "tokens_out": 45},
    {"type": "tool", "name": "search_docs", "latency_ms": 120},
    {"type": "llm", "tokens_in": 1400, "tokens_out": 200}
  ],
  "outcome": "success",
  "total_cost_usd": 0.004
}

Dashboards: success rate, avg steps, cost per task, tool error rate.

When Not to Use an Agent

Use workflow code when:

Steps are fixed (always fetch invoice, then email)
Logic is deterministic if data is known
Latency budget is tight

Use agents when:

Path depends on intermediate results
Tool set is large and choice is context-dependent
Exploration is required (within guardrails)

Over-agentifying simple pipelines adds cost and failure modes without benefit.

Incident Response Playbook

Symptom: Token spend 10x overnight.

Check: Agent traces for loops calling same tool, growing history, missing max_iterations.

Mitigation: Lower iteration cap globally, disable expensive tools via feature flag, hotfix harness.

Symptom: Wrong production data mutated.

Check: Write tools missing approval gate, idempotency, or staging environment separation.

Mitigation: Freeze write tools, require human approval, audit last 24h tool calls.

Capacity Planning

Estimate peak concurrent agents as:

concurrent_agents × avg_iterations × avg_tokens_per_call

Compare to provider TPM/RPM limits. Queue or shed load before hard failures.

Comparison: Workflow Engine vs Agent

Workflow engine	Agent
Fixed DAG	Dynamic tool choice
Predictable cost	Variable cost
Easy testing	Requires eval traces
Best for known processes	Best for exploratory tasks

Use Temporal or Step Functions when steps are known. Use agents when the path is data-dependent. Many products need both: workflow invokes agent only on fallback branch.

Reference Appendix: Production FAQ

How do I know this is working in production?

What is the first config change to try?

What belongs in an on-call runbook?

Symptom, dashboard link, rollback lever (model version, feature flag, index snapshot), owner team, and customer communication template. LLM incidents need content rollback, not only service restart.

How do I explain tradeoffs to product managers?

Use dollars and seconds: cost per successful task, p95 time to first token, accuracy on golden set. Avoid debating model intelligence; debate measurable user outcomes and failure tolerance.

When should we retrain, re-index, or rewrite prompts?

What is the common rollback path?

How does this interact with the rest of the handbook?

Summary

Agents are persistent loops, not autonomous minds. Engineering value is in the harness: termination, budgets, tool governance, and observability. Make agents stop safely before you make them smarter.

Next in Series

Blog 005: Fine-Tuning LLMs: Behavior vs Knowledge and When to Use It

Model Context Protocol (MCP) for Backend Engineers

Ameer Hamza — Tue, 11 Aug 2026 15:18:03 +0000

Introduction

Every AI agent needs tools. Before standardized protocols, every tool integration was a custom hack.

Model Context Protocol (MCP) standardizes how LLMs discover and call external tools. Instead of writing a bespoke integration for each API, you expose tools through a common protocol. The model sees a standard interface. Your backend handles authorization, execution, and error handling.

Think of MCP like OpenAPI for AI agents: one spec, many tools, consistent calling conventions.

Why This Matters

Backend engineers scale systems by standardizing interfaces. REST replaced ad-hoc RPC. OpenAPI replaced tribal knowledge about endpoints. MCP applies the same principle to agent tool access.

Before MCP, connecting an agent to Slack, a database, and a search API meant three different integration patterns, three auth flows, and three error-handling conventions. That does not scale when your tool catalog grows from 3 to 30.

Prerequisites

This article assumes you have read Blog 001 and Blog 002. You should understand LLM inference basics and how RAG provides external knowledge at query time.

The Problem

Custom tool integrations create:

N×M coupling: N agents times M tools means N×M integration code paths.
Schema drift: Tool signatures change; agent prompts break silently.
Inconsistent auth: Each tool handles credentials differently.
Poor observability: No uniform logging across tool calls.
Vendor lock-in: Switching agent frameworks means rewriting integrations.

Understanding the Core Concept

MCP defines a client-server model:

Role	Responsibility
MCP Host	The application (IDE, chat UI, agent runtime) that runs the LLM
MCP Client	Connects host to one or more MCP servers
MCP Server	Exposes tools, resources, and prompts over the protocol
Tools	Callable functions with typed input schemas
Resources	Readable data (files, records) the model can fetch

The LLM does not call HTTP endpoints directly. The host translates model tool requests into MCP messages, routes them to the correct server, and returns structured results.

Tools vs resources

Tools are actions: query a database, send a message, create a ticket. Resources are read-only context: file contents, configuration, documentation snippets. Separating reads from writes helps you apply different auth policies.

Discovery

MCP servers advertise available tools with JSON Schema descriptions. The host injects tool definitions into the model context. When the model emits a tool call, the host validates arguments against the schema before execution.

How It Works Internally (High Level)

Host starts MCP client and connects to configured servers (stdio, HTTP, or SSE transport).
Client requests tools/list from each server.
Host merges tool catalogs and presents them to the LLM.
Model generates a tool call with name and arguments.
Host validates, routes to the correct MCP server via tools/call.
Server executes, returns structured content or error.
Host feeds result back into the conversation for the next model turn.

Step-by-Step Example

Task: Agent looks up a customer's order status.

MCP server for orders exposes get_order(order_id: string).
User asks: "Where is order 48291?"
Model selects get_order with order_id: "48291".
Host validates schema, calls MCP server.
Server queries internal API, returns {"status": "shipped", "eta": "2026-07-08"}.
Model summarizes for the user.

If the server returns an error, the host should surface it to the model so it can retry or escalate.

Architecture

Standardize the interface layer first. Then scale the number of tools.

Python Example

Illustrative MCP-style tool registry pattern. Production MCP servers use the official SDK.

"""
Illustrative tool registry pattern (MCP-style).
Production: use the official MCP Python SDK.
"""
from dataclasses import dataclass
from typing import Any, Callable
import json

@dataclass
class Tool:
    name: str
    description: str
    parameters: dict
    handler: Callable[[dict], Any]

class ToolRegistry:
    def __init__(self):
        self._tools: dict[str, Tool] = {}

    def register(self, tool: Tool) -> None:
        self._tools[tool.name] = tool

    def list_tools(self) -> list[dict]:
        return [
            {
                "name": t.name,
                "description": t.description,
                "parameters": t.parameters,
            }
            for t in self._tools.values()
        ]

    def call(self, name: str, arguments: dict) -> str:
        if name not in self._tools:
            return json.dumps({"error": f"Unknown tool: {name}"})
        try:
            result = self._tools[name].handler(arguments)
            return json.dumps({"result": result})
        except Exception as exc:
            return json.dumps({"error": str(exc)})

def get_order_handler(args: dict) -> dict:
    order_id = args.get("order_id")
    if not order_id:
        raise ValueError("order_id required")
    return {"order_id": order_id, "status": "shipped", "eta": "2026-07-08"}

registry = ToolRegistry()
registry.register(Tool(
    name="get_order",
    description="Fetch order status by order ID",
    parameters={
        "type": "object",
        "properties": {"order_id": {"type": "string"}},
        "required": ["order_id"],
    },
    handler=get_order_handler,
))

if __name__ == "__main__":
    print("Tools:", json.dumps(registry.list_tools(), indent=2))
    print(registry.call("get_order", {"order_id": "48291"}))

Wrap handlers with auth checks, rate limits, timeouts, and audit logging.

Real-World Applications

IDE assistants accessing repositories, linters, and documentation via MCP servers
Enterprise agents querying internal databases through governed MCP gateways
Multi-tool agents where Slack, Jira, and search share one protocol layer
Local-first workflows exposing filesystem and CLI tools to a model host

Performance Considerations

Cold start: Each MCP server connection adds startup latency. Pool connections in long-running hosts.
Serialization: Large tool results bloat context. Truncate or summarize before feeding back to the model.
Concurrency: Parallel tool calls need idempotent handlers and clear ordering semantics.
Network hops: Remote MCP servers add RTT. Co-locate servers with data when possible.

Common Mistakes

Exposing destructive tools without confirmation gates.
Returning raw API responses instead of structured, model-friendly summaries.
No schema validation before execution.
Building MCP servers without auth scoped to the requesting user.
Treating MCP as a replacement for RAG (tools fetch live data; RAG indexes documents).

Interview Questions

Q1: What problem does MCP solve?

A: It standardizes how LLM applications discover and invoke external tools, reducing bespoke integration code.

Q2: How is MCP similar to OpenAPI?

A: Both define machine-readable interfaces. OpenAPI describes HTTP APIs; MCP describes tools and resources for LLM hosts.

Q3: What is the difference between an MCP tool and a resource?

A: Tools perform actions with side effects. Resources provide read-only context the model can fetch.

Q4: Who validates tool call arguments?

A: The MCP host validates against the tool schema before invoking the server.

Q5: When would you not use MCP?

A: For a single static integration with no growth in tool count, a direct API wrapper may be simpler.

Q6: How does MCP relate to function calling in chat APIs?

A: Chat APIs define how models emit tool calls. MCP defines how hosts connect to and execute those tools across servers.

MCP Transport and Deployment

MCP supports multiple transports:

Transport	Use case	Notes
stdio	Local tools, IDE plugins	Simple, single machine
HTTP/SSE	Remote servers	Network auth required
Hosted gateway	Enterprise tool bus	Central policy enforcement

Security model

Treat every MCP server like an internal API:

Authenticate the host to the server
Scope tools per user or tenant
Audit every tools/call with arguments and result hash
Rate limit destructive operations
Never expose raw SQL without read-only roles

Scaling tool catalogs

As tools grow past twenty, add:

Namespacing: billing.get_invoice vs crm.get_invoice
Discovery tiers: Load core tools always; load extended tools on demand
Schema registry: Version tool definitions; reject calls against stale schemas

MCP vs direct function calling

Provider function calling defines the model-facing schema. MCP standardizes server-side implementation. You can implement MCP servers behind OpenAI-compatible function routes so one tool implementation serves multiple agent hosts.

Extended Example: Multi-Server Host

A coding agent host connects to three MCP servers:

Filesystem server (stdio): read_file, list_dir scoped to workspace root.
Git server (HTTP): diff, commit with OAuth user token.
Docs server (HTTP): search_docs backed by your RAG index.

The host merges tool catalogs at startup. When the model calls search_docs, the host routes to server 3. When it calls read_file, server 1. Auth and rate limits are per server, not global defaults.

Comparison: MCP vs Ad-Hoc Integrations

Concern	Ad-hoc	MCP
Tool discovery	Hardcoded in app	Server advertisement
Schema versioning	Scattered	Central per server
Auth	Per integration	Per server policy
Reuse across hosts	Copy-paste	Same server binary
Testing	Mock each API	Mock MCP server

Failure Handling

When tools/call fails:

Return structured error to model (timeout, permission_denied, not_found).
Increment failure counter; trip circuit breaker after N failures.
Do not silently swallow errors; models loop on empty results.

Define idempotency keys for tools that mutate state.

Building Your First MCP Server

Minimal server responsibilities:

Implement tools/list returning name, description, JSON Schema parameters.
Implement tools/call executing the handler and returning text or structured content.
Validate inputs before side effects.
Return errors as structured JSON, not stack traces to the model.

Start with read-only tools. Add writes after auth and audit paths exist.

Governance Model

Central platform team owns:

Approved MCP server registry
Security review checklist per server
Shared client library in the agent host

Product teams own:

Domain-specific tool implementations
Business logic inside handlers

This mirrors API gateway governance for microservices.

Interop with OpenAI Function Calling

Map MCP tool schemas to provider function definitions at the host. When the provider returns a function call, translate to MCP tools/call. One MCP server can back multiple provider formats with a thin adapter layer.

Operations Runbook

Deploy: Version MCP servers independently from agent host. Pin server version in host config.

Rollback: If new tool schema breaks agents, revert server version; host rejects unknown tools gracefully.

Monitor: tools/call rate, error rate, p95 latency per tool, auth failure count.

Incident: On runaway tool loop, circuit-break the server at host level without redeploying the LLM.

Testing MCP Servers

Contract tests per tool:

Valid args return expected shape
Invalid args return structured error
Auth missing returns permission_denied
Timeout enforced at 30s default

Load test with parallel tools/list and tools/call matching peak agent traffic.

Enterprise Rollout Pattern

Phase 1: Read-only MCP servers (docs, search, metrics). Phase 2: Write tools with approval in staging. Phase 3: Production writes with audit and rate limits. Phase 4: Federated registry where teams publish servers to a central catalog with security sign-off. Skipping phases causes the same incidents as exposing raw admin APIs to junior scripts.

Document each tool with owner, on-call rotation, and deprecation policy. MCP without ownership becomes undeletable legacy surface area.

Reference Appendix: Production FAQ

How do I know this is working in production?

What is the first config change to try?

What belongs in an on-call runbook?

Symptom, dashboard link, rollback lever (model version, feature flag, index snapshot), owner team, and customer communication template. LLM incidents need content rollback, not only service restart.

How do I explain tradeoffs to product managers?

Use dollars and seconds: cost per successful task, p95 time to first token, accuracy on golden set. Avoid debating model intelligence; debate measurable user outcomes and failure tolerance.

When should we retrain, re-index, or rewrite prompts?

What is the common rollback path?

How does this interact with the rest of the handbook?

Summary

MCP is infrastructure for agent tool access. Design a standardized tool interface layer before your integration count explodes. The model sees a uniform catalog. Your backend enforces auth, limits, and observability behind each server.

Next in Series

Blog 004: AI Agents Explained: Loops, Guardrails, and Production Harnesses

Retrieval-Augmented Generation (RAG) for Backend Engineers

Ameer Hamza — Sun, 05 Jul 2026 20:00:37 +0000

Introduction

RAG does not make your LLM smarter. It gives your LLM a reference sheet.

Retrieval-Augmented Generation is simple in concept: search your documents, inject the relevant chunks into the prompt, and let the model answer using that context. In production, it is a distributed query pipeline where chunking, embedding model choice, index freshness, re-ranking, and context window limits each can silently degrade answer quality.

Most RAG failures are not generation failures. They are retrieval failures. The model answered correctly based on the wrong context you gave it.

Why This Matters

If you have debugged a slow SQL query or a stale cache, you already understand RAG failure modes. The generation step is the visible symptom. The retrieval step is often the root cause.

A support bot that cites the wrong policy version did not necessarily hallucinate. It may have retrieved an outdated chunk from a vector index that was never re-embedded after a docs update. Your job as a backend engineer is to treat RAG like any other data pipeline: measurable, observable, and testable at every stage.

Prerequisites

This article assumes you have read Blog 001. You should understand that LLMs are probabilistic token predictors without guaranteed grounding.

The Problem

The naive pattern:

User question -> LLM -> Answer

fails for domain-specific factual questions because the model has no access to your private, current documentation at inference time.

The naive RAG pattern:

User question -> Vector search -> Stuff top-K chunks -> LLM -> Answer

often fails silently because:

Chunks are too large or too small, splitting tables across boundaries or burying the answer in noise.
Embeddings miss semantic intent, especially for short queries or domain jargon.
The index is stale after documentation updates.
Top-K without re-ranking returns plausible but wrong passages.
Context overflow truncates the one chunk that contained the answer.

Understanding the Core Concept

RAG separates knowledge storage from language generation:

Component	Role	Failure mode
Ingestion	Parse, chunk, embed documents	Bad chunks, lost structure
Index	Store vectors for similarity search	Stale vectors, wrong metric
Retrieval	Find candidate passages for a query	Low recall, wrong neighbors
Re-ranking	Re-order candidates by relevance	Skipped step, latency spike
Generation	Synthesize answer from context	Ignores context, hallucinates beyond it

The LLM is the last mile. Search quality is the first mile.

Chunking strategies

Fixed-size chunks (for example, 512 tokens with overlap) are simple but may split sentences and tables. Semantic chunks split on paragraph or section boundaries for better coherence. Parent-child chunking retrieves small chunks for precision but injects larger parent context for generation.

Overlap, typically 10-20% of chunk size, reduces boundary artifacts where the answer spans two chunks.

Embedding model choice

The embedding model maps text to vectors where cosine similarity approximates semantic relatedness. A mismatch between embedding model and domain (legal, medical, code) hurts recall. For RAG quality, embedding choice often matters more than generator model choice.

Hybrid retrieval

Dense vector search alone struggles with exact identifiers (SKUs, error codes, function names). Combining BM25 keyword search with dense retrieval (hybrid search) improves recall on production workloads.

How It Works Internally (High Level)

Offline: Documents are parsed, chunked, embedded, and stored in a vector index with optional metadata filters.
Online: User query is embedded with the same model used at ingest time.
Search: Approximate nearest neighbor (ANN) search returns top-K candidates.
Re-rank (optional): A cross-encoder or lightweight reranker scores query-passage pairs.
Prompt assembly: System instructions plus retrieved passages plus user question.
Generation: LLM produces an answer constrained by provided context.

Step-by-Step Example

Question: "What is the refund window for annual plans?"

Embed the query.
Search index for top-5 chunks by cosine similarity.
Re-rank so the passage about annual billing rises to the top.
Assemble prompt with system rule: "Answer only from context. If unknown, say so."
Generate with low temperature (0.1-0.3).
Log query hash, chunk IDs, scores, and latency per stage.

If retrieval returns a chunk about monthly plans only, the model will answer confidently about monthly plans. Debug retrieval first.

Architecture

Highlight retrieval in your monitoring. That is where most failures happen.

Python Example

Minimal RAG retrieval loop using sentence embeddings and cosine similarity. For production, use a dedicated vector database.

"""
Minimal RAG retrieval demo.
Requires: pip install sentence-transformers numpy
"""
import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")

DOCUMENTS = [
    "Annual plans: refunds available within 14 days of purchase.",
    "Monthly plans: no refunds after billing cycle starts.",
    "Enterprise plans: custom refund terms per contract.",
]

def embed_texts(texts: list[str]) -> np.ndarray:
    vectors = model.encode(texts, normalize_embeddings=True)
    return np.array(vectors)

def retrieve(query: str, docs: list[str], top_k: int = 2) -> list[tuple[str, float]]:
    doc_vectors = embed_texts(docs)
    query_vector = embed_texts([query])[0]
    scores = doc_vectors @ query_vector
    ranked = sorted(zip(docs, scores), key=lambda x: x[1], reverse=True)
    return ranked[:top_k]

def build_prompt(query: str, contexts: list[str]) -> str:
    joined = "\n\n".join(f"- {c}" for c in contexts)
    return (
        f"Context:\n{joined}\n\n"
        f"Question: {query}\n\n"
        "Answer using only the context above."
    )

if __name__ == "__main__":
    query = "Can I get a refund on an annual subscription?"
    hits = retrieve(query, DOCUMENTS, top_k=2)
    for doc, score in hits:
        print(f"[{score:.3f}] {doc}")
    prompt = build_prompt(query, [d for d, _ in hits])
    print("\n--- Prompt ---\n", prompt)

Add chunking pipeline, metadata filters, reranker, and an eval set with recall@K metrics.

Real-World Applications

Internal documentation Q&A over wikis and PDFs
Customer support bots grounded in help center articles
Code assistants retrieving relevant files from a repository index
Compliance workflows requiring citations to source documents
Hybrid search combining BM25 keyword match with dense embeddings

Performance Considerations

Latency: Retrieval adds 50-300ms depending on index size, reranker, and filters. Budget it in your SLA.
Cost: Embedding at ingest time plus query-time embedding. Re-embedding an entire corpus on model swap is expensive.
Freshness: Event-driven re-index on document change beats nightly batch for accuracy.
Recall vs precision: Higher top-K improves recall but increases prompt tokens and noise.

Common Mistakes

Debugging prompts before debugging retrieval.
No eval set with labeled query-document pairs.
Chunking PDFs naively, destroying tables and lists.
Skipping re-ranking when top-K ANN results are noisy.
Assuming the LLM will ignore irrelevant context.

Interview Questions

Q1: What is RAG in one sentence?

A: A pattern that retrieves relevant documents at query time and injects them into the LLM prompt so answers are grounded in external knowledge.

Q2: Where do most RAG failures occur?

A: In retrieval: wrong chunks, stale index, poor embeddings, or insufficient recall.

Q3: How does RAG differ from fine-tuning for knowledge?

A: RAG injects facts at query time from an updatable index. Fine-tuning changes model weights and behavior but does not reliably store volatile facts.

Q4: What metrics would you track for a RAG pipeline?

A: Recall@K, MRR, retrieval latency, rerank latency, faithfulness, citation accuracy, and end-to-end answer correctness.

Q5: Why use chunk overlap?

A: To prevent answers from being split across chunk boundaries where neither chunk alone contains the full answer.

Q6: When would you add a re-ranker?

A: When ANN search returns semantically nearby but task-irrelevant passages, especially with short queries or large corpora.

Production Checklist

Before shipping RAG to production, verify each layer:

Ingestion idempotency: Re-running ingest on the same document produces the same chunk IDs or upserts cleanly.
Version tags: Store embedding model name and version on every vector.
Retrieval eval: Recall@5 above your threshold on a labeled set of at least 100 queries.
Faithfulness eval: Answers cite only retrieved text on a held-out Q&A set.
Latency budget: p95 retrieval under your SLA (often 200ms excluding LLM).
Failure logging: Log empty retrieval, low scores, and truncated context.

Index freshness patterns

Pattern	Freshness	Complexity
Nightly batch	Hours stale	Low
Event-driven on doc update	Minutes	Medium
Write-through on publish	Near real-time	High

For policy and pricing docs, event-driven re-index is usually worth the engineering cost.

When RAG is not enough

RAG handles lookup over static or slow-changing text. It struggles with:

Real-time transactional data (use tools or SQL instead)
Multi-hop reasoning across many documents (consider agent workflows or graph retrieval)
Computations (the model may hallucinate arithmetic; use a calculator tool)

Combine RAG with MCP tools (Blog 003) when answers require live system state.

Design Tradeoffs: Chunk Size

Chunk size	Pros	Cons
128-256 tokens	Precise retrieval	May lack surrounding context
512-1024 tokens	Good default for prose	Tables may split awkwardly
2000+ tokens	Full section context	Lower precision, higher noise

Tune on your eval set. Legal and API docs often need structure-aware chunking (by heading or OpenAPI operation), not fixed token windows.

Extended Walkthrough: Debugging a Wrong Answer

A user asks: "Do annual plans include phone support?"

Check generation: model cited "Priority email support for all plans."
Check retrieval logs: chunk support_tiers_v3.md#chunk-14 scored highest.
Open chunk: it describes monthly plans only; annual tier is chunk-22.
Root cause: embedding confused "annual" and "monthly" in short query.
Fix: add metadata filter plan_type=annual when query classifier detects billing intent; add reranker; expand eval queries.

Without retrieval logs, you would have tweaked the system prompt for days.

Observability Fields

Log per request:

query_text_hash
embedding_model_version
retrieved_chunk_ids with scores
rerank_scores if applicable
prompt_token_count
answer_faithfulness_score if automated

These fields make RAG debuggable like any distributed pipeline.

Eval Metrics Reference

Metric	Measures	Target direction
Recall@K	Relevant doc in top K	Higher
MRR	Rank of first relevant	Higher
nDCG	Graded relevance ranking	Higher
Faithfulness	Answer supported by context	Higher
Answer correctness	End-to-end vs gold	Higher
Latency p95	Retrieval plus generation	Lower

Run retrieval and generation evals separately before combining. A 10% retrieval recall gain often beats switching to a more expensive LLM.

Anti-Patterns in RAG Products

Dump entire wiki: Retrieval exists but pipeline sends 50 random pages.
No citations: User cannot verify answers; trust erodes on first error.
Single embedding for code and prose: Split indexes or use hybrid search.
Ignore ACLs: Vector index returns docs the user cannot access.
Skip ACL sync on delete: Removed permissions still retrievable until reindex.

Enforce document-level permissions at retrieval time, not only at the UI.

Summary

RAG is a search pipeline with an LLM at the end. Treat chunking, embedding, indexing, and retrieval as first-class engineering problems. Measure retrieval quality before tuning generation. Your RAG system is only as good as the search layer underneath it.

Next in Series

Blog 003: Model Context Protocol (MCP): Standardizing Tool Integration for AI Agents

LLMs Explained for Backend Engineers

Ameer Hamza — Sat, 04 Jul 2026 19:37:44 +0000

Introduction

If you have built APIs, databases, and distributed systems, you already have the mindset needed for AI engineering. The missing piece is a clear mental model of what a Large Language Model (LLM) actually is.

An LLM is not a search engine with better grammar. It is not a database of facts. It is a probabilistic token prediction engine trained on massive text corpora. You give it a sequence of tokens. It predicts the next token. Repeat that thousands of times and you get text that reads like an answer.

That single distinction explains both the power and the unreliability of modern AI applications.

Why This Matters

Backend engineers are comfortable with unreliable dependencies. Caches go stale. Third-party APIs return 500s. Queues back up. We design retries, circuit breakers, fallbacks, and observability around those failures.

LLMs are another unreliable dependency, but with a twist: they fail confidently. A wrong answer often looks as polished as a right one. There is no HTTP status code that says "this paragraph is hallucinated."

Production AI engineering is therefore the discipline of wrapping a non-deterministic core inside a deterministic system: retrieval, guardrails, parsers, eval gates, and human escalation paths.

Prerequisites

This is the first article in the AI Engineering Handbook. You should be comfortable with:

Basic Python
HTTP APIs and request/response flows
The idea of latency, throughput, and error rates in production services

No prior machine learning coursework is required.

The Problem

Teams often treat the LLM as the entire product:

User -> LLM -> Answer

That mental model breaks in production:

No grounding: The model invents facts not present in training data for your domain.
No memory contract: Stateless APIs do not remember prior sessions unless you build memory.
No permission boundary: The model will attempt any completion the prompt allows.
Unbounded cost: Token usage scales with input and output length.
Variable latency: Time to first token and total generation time depend on model size and load.

The model is the easy part. The system around it is where engineering begins.

Understanding the Core Concept

Tokens, not words

LLMs operate on tokens, subword units produced by a tokenizer. The phrase "unhappiness" might be one token or three depending on the tokenizer. This matters for:

Billing: API pricing is per token.
Context limits: Windows are measured in tokens, not characters.
Domain failures: Rare product names may split into many tokens or map to unknown tokens, hurting quality and cost.

Next-token prediction

At each step, the model outputs a probability distribution over the vocabulary. A decoding strategy (greedy, temperature sampling, top-p) selects the next token. The process repeats autoregressively until a stop condition.

There is no separate "fact checking" step. There is no guaranteed retrieval from a knowledge base unless you add retrieval (covered in Blog 002).

Training vs inference

Phase	What happens	Engineering concern
Pretraining	Learn language patterns from huge corpora	Model choice, license, capability ceiling
Fine-tuning / alignment	Adapt behavior to instructions or domain	Data quality, forgetting, eval
Inference	Generate tokens for your prompt	Latency, cost, guardrails

As a backend engineer building applications, you mostly live in inference. You choose models, assemble context, and enforce policies around the call.

How It Works Internally (High Level)

Tokenization: Raw text becomes integer token IDs.
Embedding: Token IDs map to dense vectors.
Transformer layers: Self-attention lets each token attend to others; feed-forward layers transform representations.
Output head: Final layer projects to vocabulary logits.
Sampling: Decoding strategy picks the next token.
Repeat: Append token, update KV cache (Blog 013), continue until stop.

You do not need to implement a transformer to ship a product. You do need to know that latency and memory grow with context length and output length.

Step-by-Step Example

User question: "What is our refund policy for annual plans?"

Naive approach: Send the question directly to the LLM.

Likely failure: The model produces a plausible refund policy that does not match your actual terms.

Production approach:

Authenticate the user.
Retrieve policy chunks from your knowledge base (RAG, Blog 002).
Assemble a prompt with system rules, retrieved context, and the user question.
Call the model with temperature appropriate for factual tasks (low).
Parse structured output if needed.
Run output guardrails (no legal commitments beyond retrieved text).
Log prompt hash, retrieval IDs, latency, and token counts.

The LLM generates language. Your system decides what it is allowed to say.

Architecture

The LLM is one box in a larger service. Treat it like you would treat an external API with no SLA on correctness.

Python Example

Minimal illustration: call an OpenAI-compatible chat API and measure tokens. This is inference-only, no retrieval.

"""
Minimal LLM inference wrapper with token usage logging.
Requires: pip install openai
Set OPENAI_API_KEY in environment.
"""
import os
from openai import OpenAI

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

SYSTEM_PROMPT = (
    "You answer using only the context provided. "
    "If the context is insufficient, say you do not know."
)

def complete(user_message: str, context: str = "") -> dict:
  messages = [
      {"role": "system", "content": SYSTEM_PROMPT},
      {
          "role": "user",
          "content": f"Context:\n{context}\n\nQuestion:\n{user_message}",
      },
  ]
  response = client.chat.completions.create(
      model="gpt-4o-mini",
      messages=messages,
      temperature=0.2,
      max_tokens=512,
  )
  choice = response.choices[0].message.content
  usage = response.usage
  return {
      "text": choice,
      "prompt_tokens": usage.prompt_tokens,
      "completion_tokens": usage.completion_tokens,
      "total_tokens": usage.total_tokens,
  }

if __name__ == "__main__":
  result = complete(
      user_message="Summarize the refund window.",
      context="Annual plans: refunds within 14 days of purchase.",
  )
  print(result["text"])
  print(f"Tokens: {result['total_tokens']}")

In production, add: timeouts, retries with idempotency keys, structured logging, and budget caps per user.

Real-World Applications

Customer support assistants with retrieval over help docs
Code assistants with repo context and sandboxed execution (Blog 004)
Document Q&A over internal PDFs and wikis
Classification and extraction with constrained output schemas
Agent workflows that call tools via standardized protocols (Blog 003)

Performance Considerations

Latency: Dominated by model size, context length, and output length. Measure TTFT and tokens per second (Blog 015).
Cost: prompt_tokens + completion_tokens at model-specific rates. Long system prompts are not free.
Concurrency: GPU memory limits concurrent sequences. Queue or route when saturated.
Caching: Identical prefix prompts may benefit from prompt caching on some providers (later in handbook).

Common Mistakes

Treating the model as source of truth for dynamic business data.
Omitting observability on prompts, retrieval IDs, and token usage.
Using high temperature for factual tasks.
Ignoring tokenizer effects on domain-specific vocabulary.
No failure mode when the model refuses or returns empty output.

Interview Questions

Q1: What is an LLM in one sentence?

A: A neural network trained to predict the next token in a sequence, used at inference time to generate text autoregressively.

Q2: How is an LLM different from a search engine?

A: Search retrieves existing documents by matching queries. An LLM generates new text from learned patterns without guaranteed grounding unless you add retrieval.

Q3: Why do LLMs hallucinate?

A: They optimize for plausible continuations, not verified truth. Without external grounding or constraints, they may invent facts.

Q4: What belongs in the system around the model?

A: Retrieval, auth, rate limits, guardrails, parsers, evals, logging, and escalation paths.

Q5: What drives LLM API cost?

A: Total tokens processed (input + output), model tier, and optional features like tool calls or vision.

Q6: When should you not use an LLM?

A: When deterministic rules suffice, when strict correctness is required without verification, or when latency and cost cannot tolerate probabilistic generation.

Summary

An LLM is a probabilistic token predictor, not an oracle. Backend engineers succeed with LLMs when they design systems: context assembly, retrieval, policy enforcement, and observability. The model generates language. Your architecture decides whether that language is safe, grounded, and useful.

Next in Series

Blog 002: Retrieval-Augmented Generation (RAG): Architecture and Tradeoffs

HTTP Protocol Deep Dive: Everything Every Backend Engineer Must Know

Ameer Hamza — Wed, 13 May 2026 10:27:35 +0000

This post is an in-depth explanation of HTTP (Hypertext Transfer Protocol), the core foundation of the internet. Whenever your frontend (browser or mobile app) communicates with a backend server, it happens through HTTP. The post explains that HTTP is inherently stateless — meaning the server does not remember previous interactions and treats every new request as independent. It also covers the different parts of an HTTP message (method, URL, headers, body) and their roles in detail. Headers are compared to address labels on a courier parcel that carry important extra information (metadata). The post further discusses API design best practices, including the correct use of HTTP methods (GET, POST, PUT, PATCH), CORS (Cross-Origin Resource Sharing), the importance of status codes (200, 404, 500), and techniques like caching and compression to improve server performance.

Key Concepts Breakdown

1. Statelessness

Simple Explanation: The server does not store any memory of previous requests. Every request must be self-contained, meaning it should include all necessary information (such as authentication tokens) to be processed.
Why It Matters: This makes backend architecture simple and highly scalable. If one server goes down, another server can handle the request without losing any session data.

2. HTTP Headers (Metadata)

Simple Explanation: Headers are key-value pairs that provide additional information about the request or response. Examples include which browser the client is using (User-Agent), the expected response format (Accept: application/json), or whether the user is authenticated (Authorization).
Why It Matters: They make communication between frontend and backend flexible without modifying the actual data in the body.

3. Idempotent vs Non-Idempotent Methods

Simple Explanation: A method is idempotent if calling it once or multiple times produces the same result on the server (e.g., GET, PUT, DELETE). If each call creates a different result, it is non-idempotent (e.g., POST).
Why It Matters: When the network fails and the client retries a request, you need to know whether retrying is safe or if it might create duplicate entries.

4. CORS & Pre-flight Requests (OPTIONS)

Simple Explanation: For security reasons, browsers block direct API calls from one domain to another. For complex requests (such as those with JSON data or custom headers like Authorization), the browser first sends an OPTIONS request (pre-flight) to ask the server for permission.
Why It Matters: Without proper CORS setup, browsers will reject third-party API calls — one of the most common issues in frontend-backend integration.

5. HTTP Status Codes

Simple Explanation: These are three-digit numbers that indicate the result of a request: 2xx (Success), 3xx (Redirection), 4xx (Client errors like 400 Bad Request or 404 Not Found), and 5xx (Server errors).
Why It Matters: Frontend applications can use these codes to show appropriate messages or update the UI without always parsing the response body.

6. Caching (E-Tags & 304 Not Modified)

Simple Explanation: If the data has not changed, the server sends a 304 Not Modified status instead of the full response. The browser then uses its locally cached version.
Why It Matters: This saves bandwidth and can improve application speed by up to 10 times.

Real Job Scenario

Problem Context:

You are working on an AI-driven Document Intelligence platform called LexAI. Your frontend (React/Vite) is running on http://localhost:5173 and your backend API (FastAPI) is on http://localhost:8000. When a user uploads a PDF, the frontend sends a POST request with Authorization: Bearer and Content-Type: application/json, but the request fails. The browser console shows a red "CORS Error", while the backend logs show no incoming request.

Why It’s Difficult:

The frontend engineer thinks the backend is down, while the backend engineer insists the code is correct and no request is reaching the server. Hours are wasted in debugging.

How This Concept Helps:

Understanding CORS and pre-flight requests solves this issue directly. Since the frontend and backend are on different ports (5173 to 8000) and the request includes non-simple headers (Authorization) and content type (application/json), the browser sends an OPTIONS request first to check permissions.

Step-by-Step Solution:

Check Network Tab: Open browser developer tools and look at the Network tab. You will see an OPTIONS request failing before the actual POST request.
Identify the Block: The backend is not properly handling the OPTIONS request or not returning the correct Access-Control-Allow-Origin headers.
Backend Fix: Configure CORS middleware in your backend (FastAPI, Django, etc.).
Configure Origins & Headers: Add http://localhost:5173 to the allowed origins and explicitly allow Authorization and Content-Type in Access-Control-Allow-Headers.
Cache Pre-flight: Set the Access-Control-Max-Age header so the browser caches the pre-flight response (e.g., for 24 hours) and avoids sending OPTIONS on every request.

Final Result:

The pre-flight check passes (returning 204 No Content), the original POST request succeeds, and the blocker between the API and UI teams is resolved.

Practical Implementation Guide

Step 1: Always use the correct HTTP methods. Use POST to create new records, PATCH for partial updates, and PUT only when completely replacing a resource.
Step 2: Standardize status codes. Return 401 Unauthorized for unauthenticated users, 403 Forbidden for permission issues, and 500 for server errors. Avoid sending all errors as 200 OK with custom JSON messages.
Step 3: Enable data compression. For APIs returning large JSON responses, turn on Gzip or Brotli compression at the server level (NGINX or API Gateway). This can reduce a 25MB payload down to 3MB.
Step 4: Handle large file uploads properly. For images, videos, or PDFs, use multipart/form-data instead of JSON to allow streaming and prevent server crashes.

Technical Insights & Tradeoffs

HTTP’s stateless nature is perfect for horizontal scaling and load balancing because no server needs to remember user sessions. The tradeoff is that every request becomes larger as you must send authentication tokens (like JWTs) with each call.

Common Mistake: Many developers use PUT for partial updates. PUT means complete replacement and is idempotent. For updating just one field (like phone number), PATCH should be used.

When NOT to use HTTP Caching: Avoid it for real-time dashboards or AI-generated streaming content (like ChatGPT). In such cases, WebSockets or Server-Sent Events (SSE) are better choices.

Workplace & Career Impact

Deep knowledge of HTTP helps you make better technical decisions in system design and confidently justify your choices during architecture discussions. Status codes act as a universal language that improves collaboration between frontend and backend teams. Mastering these fundamentals moves you from being just a framework developer to a true Software Architect.

Quick Recap

HTTP is a stateless protocol that manages data transfer between client and server.
Every request is self-contained and carries important metadata through headers.
HTTP methods tell the server the intended action, while status codes define the outcome.
Browsers enforce CORS and pre-flight requests for security.
Caching, compression, and proper use of methods and codes are essential for building efficient and scalable backends.

Understanding Check Questions

You are building a profile page backend. The user wants to update only their phone number. Should you use PUT or PATCH for this action, and why?
Your system is moving from a monolith to a microservices architecture with 5 different API servers. How does HTTP’s stateless property help you manage user login sessions?
A mobile app (non-browser client) and a web frontend are both sending the same POST request (with JSON and auth token) to the backend. The web app gets blocked while the mobile app works fine. What is the reason behind this difference in the context of CORS and OPTIONS requests?

Thank you for reading!

I hope this deep dive helped you build a stronger understanding of HTTP and how it powers modern applications. If you found it useful, feel free to share it with other developers.

Now it’s your turn — try answering the Understanding Check Questions in the comments below. I’d love to read your answers and discuss them with you. This is one of the best ways to truly internalize these concepts.

Happy coding! 🚀

Authentication vs. Authorization: A Deep Dive Every Backend Engineer Must Know

Ameer Hamza — Wed, 13 May 2026 07:19:07 +0000

This post is an in-depth breakdown of Authentication (Who are you?) and Authorization (What are you allowed to do?). In the early days, identity was based on simple trust. In modern web applications, we rely on complex and secure systems. As a backend engineer, it is essential to understand the key differences between stateful (Sessions) and stateless (JWTs) authentication, when to use API keys, and exactly how “Sign in with Google” (OAuth 2.0 / OIDC) works behind the scenes. The post also covers practical security risks such as how hackers exploit timing attacks and detailed error messages, along with ways to keep your systems secure.

Key Concepts Breakdown

1. Authentication vs. Authorization

Simple Explanation: Authentication (AuthN) is the process of verifying identity (like showing your ID card). Authorization (AuthZ) is the process of checking permissions (like whether that ID card allows you to enter the server room).
Why It Matters: Mixing up the two leads to insecure systems. Always verify who the user is first, then decide what they can access.

2. Stateful Authentication (Sessions)

Simple Explanation: The server stores the user’s session data (whether they are logged in, their user info, etc.) in a database or cache like Redis. The browser only receives a small Session ID stored in a cookie.
Why It Matters: You retain full control. You can instantly log out any user by revoking their session.

3. Stateless Authentication (JWT)

Simple Explanation: JSON Web Tokens (JWTs) are self-contained ID cards that include user data and a cryptographic signature. The server doesn’t need to query a database. It only verifies the signature.
Why It Matters: When you have thousands of users and multiple microservices, this approach allows your system to scale easily without slowing down due to database lookups.

4. OAuth 2.0 & OpenID Connect (OIDC)

Simple Explanation: OAuth 2.0 is a protocol that lets one application access another application’s data without sharing passwords (delegation). OpenID Connect (OIDC) builds on top of it to provide user identity and authentication (e.g., “Sign in with Google”).
Why It Matters: Users don’t want to create new passwords for every app. This is the industry standard for secure third-party logins and integrations.

5. API Keys

Simple Explanation: These are passwords used for communication between servers and machines. No human or UI is involved. It’s pure machine-to-machine communication.
Why It Matters: When your backend needs to talk to third-party services (like OpenAI, Stripe, etc.), API keys are the standard way to authenticate.

6. Role-Based Access Control (RBAC)

Simple Explanation: Assign each user a role (Admin, Editor, Viewer, etc.) and grant permissions based on that role.
Why It Matters: It keeps your authorization logic clean. Instead of writing if-else checks in every API route, you can handle role validation in middleware.

7. Security Flaws: Error Leaks & Timing Attacks

Simple Explanation: Never give hackers hints. Always return generic error messages (“Invalid credentials”) whether the email is wrong or the password is incorrect. Password verification should also take the same amount of time for every input (constant-time comparison).
Why It Matters: Small details can reveal whether an account exists, allowing attackers to focus on cracking the password.

Real Job Scenario

Problem Context:

You are an SDE at a fast-growing e-commerce startup. The team is migrating from a monolith to microservices (User Service, Cart Service, Order Service, etc.).

Why It’s Difficult:

Previously, the User Service stored sessions in Redis. Now, every time a user interacts with the Cart Service, it has to call the User Service to verify if the user is logged in. This cross-service communication dramatically increased latency.

Solution Using Concepts from This Post:

Shift from Stateful (Sessions) to Stateless (JWT) authentication.

Step-by-Step Implementation:

Login: User sends email/password to the User Service.
JWT Creation: After verification, the User Service generates a JWT (containing user_id, role, etc.), signs it with a secret key, and sends it to the browser.
Subsequent Requests: The browser includes the JWT in the Authorization header when calling the Cart Service.
Local Verification: The Cart Service uses the shared secret (or public key) to verify the token’s signature and extract the user_id and role locally. No need to call the User Service or Redis.

Final Result:

Internal network calls and database queries are drastically reduced. APIs respond extremely fast, and adding new microservices becomes much easier because authentication is now fully decentralized and stateless.

Practical Implementation Guide

Step 1: For simple web apps (e.g., internal admin dashboards), start with Stateful Sessions. They are secure and easy to manage.
Step 2: For mobile apps or microservices architectures, implement JWTs, but always store secret keys securely (.env files, AWS Secrets Manager, etc.).
Step 3: Always return generic error messages during login/signup: return res.status(401).json({ error: "Invalid email or password" });
Step 4: Protect routes using RBAC middleware (e.g., @require_role('admin') in Django or authorizeRole(['admin']) in Express).
Step 5: Never store plain-text passwords. Always hash them using bcrypt or argon2.

Technical Insights & Tradeoffs

JWTs are fast and highly scalable, but they come with a major challenge: Revocation. Since the token is stateless, you cannot instantly log out a user until the token expires. Sessions allow instant revocation by simply deleting the session record.

Use Sessions for B2B SaaS web applications where tight security control and immediate revocation are critical.
Use JWTs for high-traffic mobile backends and microservices architectures.
Never share usernames/passwords with third parties. Use API Keys or OAuth instead.
Common Mistake: Storing JWTs in localStorage (vulnerable to XSS attacks). Prefer HTTPOnly + Secure cookies when possible.

Workplace & Career Impact

Understanding these concepts deeply helps you make confident architectural decisions in system design discussions. Security and access control directly impact revenue and company reputation. Engineers who master these fundamentals progress faster to Senior and Staff-level roles.

You’ll also be able to push back effectively against Product Managers who request user-unfriendly but insecure flows (like revealing whether an email exists), explaining clearly why security tradeoffs matter.

Quick Recap

Authentication verifies identity. Authorization determines permissions.
Sessions (Stateful) keep data on the server. JWTs (Stateless) are self-contained, scalable tokens.
OAuth 2.0 & OIDC enable secure third-party logins without sharing passwords.
API Keys are for machine-to-machine communication.
Always use generic errors and constant-time operations to prevent information leaks and timing attacks.

Understanding Check Questions

Scenario: A user’s account is compromised and you’re using JWT (Stateless Authentication). How do you immediately revoke/block that specific user’s session without logging out everyone else?
Scenario: You’re designing a B2B SaaS dashboard with strict compliance requirements. Admins must be able to revoke any user’s access in milliseconds. Would you choose JWT or Session-based auth? Why?
Scenario: Your frontend developer wants to show “Account not found, please sign up” if an email doesn’t exist in the database. As a backend engineer, how would you explain the security implications from a technical and security perspective?

Thank you for reading!

I hope this deep dive helped you strengthen your understanding of Authentication and Authorization. If you enjoyed the post, feel free to share it with fellow developers.

Now it’s your turn — try answering the Understanding Check Questions in the comments below. I’d love to read your responses and discuss them with you. This is the best way to truly internalize these concepts.

Happy coding! 🚀

DEV Community: Ameer Hamza

Fine-Tuning LLMs for Backend Engineers

Introduction

Why This Matters

Prerequisites

The Problem

Understanding the Core Concept

Behavior vs knowledge

Fine-tuning methods

Training data quality

How It Works Internally (High Level)

Step-by-Step Example

Architecture

Python Example

Real-World Applications

Performance Considerations

Common Mistakes

Interview Questions

Data Pipeline for Fine-Tuning

Synthetic data risks

When to prefer prompts over fine-tuning

LoRA Hyperparameters (Starting Points)

Full Fine-Tune vs LoRA Decision

Combining with RAG at Inference

Dataset Size Guidelines (Rules of Thumb)

Legal and Compliance

Rollback Strategy

Handoff Between ML and Platform Teams

Cost of Ownership

Storage and Artifact Management

Reference Appendix: Production FAQ

Summary

Further Reading

Next in Series

AI Agents Explained for Backend Engineers

Introduction

Why This Matters

Prerequisites

The Problem

Understanding the Core Concept

The agent loop

Agent vs single LLM call

Harness responsibilities

How It Works Internally (High Level)

Step-by-Step Example

Architecture

Python Example

Real-World Applications

Performance Considerations

Common Mistakes

Interview Questions

Termination Policies

State management options

Cost model for agents

ReAct and Tool-Calling Patterns

Parallel vs sequential tools

Human-in-the-loop placement

Testing Agents

Observability for Agent Traces

When Not to Use an Agent

Incident Response Playbook

Capacity Planning

Comparison: Workflow Engine vs Agent

Reference Appendix: Production FAQ

Summary

Further Reading

Next in Series

Model Context Protocol (MCP) for Backend Engineers

Introduction

Why This Matters

Prerequisites

The Problem

Understanding the Core Concept

Tools vs resources

Discovery

How It Works Internally (High Level)

Step-by-Step Example

Architecture

Python Example

Real-World Applications