DEV Community: Suryadeepsinh Gohil

The Context Window Paradox: Why Bigger Isn't Always Better in AI

Suryadeepsinh Gohil — Wed, 21 Jan 2026 18:16:28 +0000

A Story About Our Obsession with More

Imagine you're a chef preparing a meal for a food critic. You have a beautiful dining table, but only so much space. Adding seventeen competing dishes doesn't necessarily make the meal better. Sometimes it makes the best dish harder to notice.

That is a useful analogy for an LLM's context window, with an important limit: more context can help, and sometimes it is necessary. What matters is whether the model can use the information you include.

A larger window gives us capacity. It doesn't automatically give us relevance, reliable reasoning, or a good cost per answer. This guide is about making those trade-offs explicit.

Revision note: this article now distinguishes published research from illustrative examples. The earlier unsourced model-accuracy table and company anecdotes have been removed; the cost arithmetic and parent-child token accounting are corrected. The examples below are not production benchmark results.

1. Capacity Is Only One Constraint

It is tempting to treat a long-context model as a replacement for retrieval: paste the documents into the prompt and let the model find the answer.

For a small, stable document set, that can be a sensible baseline. For broad synthesis, removing too much context can make the answer worse. But as the corpus, conversation history, and number of requests grow, there are several separate constraints:

Constraint	Engineering question
Capacity	Does the complete request fit the model's input and combined-context limits?
Coverage	Did we include the evidence needed to answer, including exceptions and counterexamples?
Selection	Can the model identify the relevant evidence among the other material?
Latency	How much time goes to retrieval, reranking, prefill, and generation?
Cost	What do generation, retrieval, reranking, and failed attempts cost together?

The goal is sufficient evidence within a measured budget. A short prompt that omits the answer is no improvement over a long, noisy one.

2. What "Lost in the Middle" Actually Shows

The 2023 paper Lost in the Middle: How Language Models Use Long Contexts, by Nelson F. Liu and colleagues, tested multi-document question answering and key-value retrieval. In their experiments, performance often depended on where the relevant information appeared: the beginning and end could work better than the middle.

The paper's main evaluations included models such as GPT-3.5-Turbo and Claude 1.3; it also reports GPT-4 experiments in an appendix. It does not provide the Claude 3/Gemini 1.5 accuracy table that appeared in the earlier version of this article.

The useful conclusion is scoped: position and distracting context can affect retrieval from a prompt. It is not proof that every current model ignores the middle, that every longer prompt is worse, or that reranking eliminates the problem.

For your own system, test the same question with the supporting passage at the beginning, middle, and end. Keep the answer, distractors, model version, and prompt template fixed. Repeat across multiple questions and runs. A single successful answer is a demonstration, not an accuracy estimate.

3. Do the Cost Math Before Choosing an Architecture

Let's use an illustrative input-token rate of $10 per million tokens, not a current quote for any particular provider.

Input per request	Input cost per request	10 requests per conversation	10,000 such conversations
50,000 tokens	$0.50	$5.00	$50,000
2,000 tokens	$0.02	$0.20	$2,000

That is a 25× difference in this input-cost component. It assumes each request has the stated input size and rate. It excludes output tokens, caching discounts, retrieval, reranking, retries, and infrastructure. It also says nothing about answer quality: the 2,000-token version must still contain enough evidence.

For a separate example, reducing total cost from $0.50 to $0.15 per query saves:

($0.50 - $0.15) × 10,000 queries = $3,500 per month
($0.50 - $0.15) / $0.50 = 70% reduction

The earlier $35,000 figure was ten times too high.

There is also a computational reason to watch context length. Standard dense self-attention forms pairwise interactions that grow quadratically with sequence length, as described in Attention Is All You Need. Increasing the sequence length tenfold means roughly a hundredfold increase in that component; not exponential growth, and not necessarily a hundredfold increase in total request latency. Kernels, attention variants, caching, batching, and the rest of the model change the actual serving cost.

4. Budget the Complete Request

"Retrieve the top five chunks" is not a token budget. Five short passages and five large parent documents can have very different costs.

Start with the full request:

context capacity
  - reserved output/reasoning capacity where applicable
  - safety margin
  - system instructions, tool schemas, query and message overhead
  = room for history and evidence

The following Python example allocates that remaining space. The history fractions are starting heuristics, not empirically optimal settings. Supply counts from the target model's tokenizer and message format; character counts or word counts are not interchangeable with tokens.

from math import floor


def allocate_budget(*, context_limit, output_reserve, fixed_input,
                    history_needed, intent="factual", safety_margin=256):
    """Allocate integer token caps; does not tokenize or truncate messages."""
    counts = (context_limit, output_reserve, fixed_input,
              history_needed, safety_margin)
    if any(type(value) is not int or value < 0 for value in counts):
        raise ValueError("Token counts must be non-negative integers")
    if context_limit == 0:
        raise ValueError("context_limit must be positive")

    history_fraction = {
        "factual": 0.10,
        "analytical": 0.15,
        "conversational": 0.50,
    }
    if intent not in history_fraction:
        raise ValueError("Unknown intent")

    available = (context_limit - output_reserve
                 - safety_margin - fixed_input)
    if available < 0:
        raise ValueError("Fixed request plus reserves exceeds context limit")

    history = min(history_needed, floor(available * history_fraction[intent]))
    return {"history": history, "retrieval": available - history}


budget = allocate_budget(
    context_limit=8192, output_reserve=1024, fixed_input=900,
    history_needed=1200, intent="analytical", safety_margin=256,
)
assert budget == {"history": 901, "retrieval": 5111}
assert 900 + 1024 + 256 + sum(budget.values()) == 8192

This is only the allocator. The prompt assembler must enforce the returned caps, preserve whole evidence units, and re-count the serialized final request. Provider-specific input/output limits still apply. If essential evidence does not fit, use another retrieval step, a staged analysis, a larger suitable model, or an explicit insufficient-evidence response. Silently cutting the passage that contains the exception is a bad optimization.

Unused capacity is fine. A budget is a ceiling, not a requirement to fill the window.

5. Chunk for Meaning, Then Count What You Actually Send

Start with document structure

Fixed-size chunks are a useful baseline. They become problematic when they separate a definition from its exception, a table from its header, or code from the explanation of its inputs.

Try section-aware splitting first. Preserve document IDs, headings, version dates, and source locations. Semantic splitting can help identify topic boundaries, but it adds indexing cost and still needs size limits. Compare it against the simpler baseline on your questions rather than assuming it is better.

Parent-child retrieval has two different token counts

With parent-child indexing, we search small child chunks and expand the matches to larger parent passages for generation. That separates search granularity from reading granularity. It does not mean the generator pays only for the children.

Here is an illustrative example:

Stage	Count
Retrieved children	10 × 128 = 1,280 tokens
Distinct parent passages after deduplication	3 × 1,000 = 3,000 tokens
Evidence actually sent to generation	3,000 tokens
Comparison baseline	5 × 1,000 = 5,000 tokens

Under those assumptions, evidence input falls by 40%, not 75%. System instructions, query, history, and output are additional.

If only three of the ten children share a parent and the other seven each have a different parent, expansion produces eight parents: 8,000 tokens. That is larger than the baseline. Deduplicate first, rank the expanded passages, and enforce the generation budget after expansion.

Propositional indexing takes this further by extracting individual factual statements. It may improve precision for some questions, but extraction can lose qualifiers or introduce errors. Keep every proposition linked to its source and evaluate extraction fidelity. There is no general 80% token-savings guarantee.

6. Build a Pipeline You Can Inspect

Here is a practical starting point, with optional stages added only when measurements justify them:

Question + authorized corpus scope
    ↓
Budget and retrieval strategy
    ↓
Keyword / vector candidates
    ↓
Optional reranking → parent expansion → deduplication
    ↓
Evidence selection within budget
    ↓
Prompt assembly and final token check
    ↓
Answer with source references, or an insufficient-evidence response
    ↓
Trace, evaluation and cost accounting

Hybrid search combines lexical matching with semantic similarity. It can recover different kinds of matches, but it cannot guarantee complete coverage. Apply access constraints to candidate retrieval and again before serving evidence; a relevant document is not necessarily one the current user may read.

A reranker can improve ordering at additional cost. Candidate count and reranking depth are workload parameters. Test their effect on both evidence recall and latency. Do not assume a smaller reranked set is always sufficient for questions that require several documents.

Compression deserves the same scrutiny. Removing an apparently irrelevant sentence can remove a negation or exception. For high-consequence answers, preserve source passages and citations so that summaries can be checked against the original evidence.

Retrieved text is also untrusted input. Keep instructions separate from evidence, constrain tool permissions, and test document-borne prompt injection. Better retrieval does not remove that boundary.

7. Caching Can Hide Work; It Cannot Make All Latency Disappear

Next-turn prefetching is an optimization to test after you have a measured baseline. Predict a small number of likely follow-ups, retrieve their evidence asynchronously, and reuse it only when the actual request matches closely enough.

A valid cache hit may reduce retrieval time. It does not eliminate request handling, model prefill, generation, or network latency. Wrong predictions consume extra resources. Track the hit rate, useful-prefetch rate, and wasted work alongside latency percentiles.

Cache entries need a corpus/version identifier, an access scope, an expiry policy, and a way to invalidate stale evidence. A cached result must not cross a user's authorization boundary.

Likewise, a normal streaming generation request cannot simply accept new passages halfway through because a background process fetched them. Use the fetched material in a subsequent model/tool step, or use a serving architecture explicitly designed for incremental retrieval. Treat that architecture as a separate implementation problem, not a free property of streaming.

8. Evaluate Before Optimizing

Start with a fixed set of representative questions and expected supporting evidence. Include multi-document questions, unanswerable questions, stale/conflicting sources, and permission boundaries.

Compare a long-context baseline with a simple retrieval baseline. Then change one component at a time: chunking, reranking, parent expansion, compression, or prefetching. Keep model version and generation settings recorded so a later comparison means something.

Measure	What it tells you	Common trap
Evidence recall	Whether the required supporting material was retrieved	High precision can hide missing evidence
Evidence precision	How much retrieved material is relevant under your rubric	"Mentioned in the answer" is not the same as relevant
Answer correctness	Whether the answer satisfies the question and reference criteria	A fluent or cited answer can still be wrong
Faithfulness	Whether claims are supported by the supplied evidence	Faithful repetition of a stale source is not current truth
p50/p95 latency and time to first token	Typical and slower-user experience	Streaming does not remove prefill time
Cost per successful query	Total measured cost divided by queries meeting a declared success rule	Dividing by all queries measures cost per query instead

An LLM judge can help scale evaluation, but its decisions are not ground truth. Use a written rubric, inspect disagreements, and calibrate against human-reviewed examples. Report sample size, failures, and variability instead of a universal target such as "90% faithfulness is enough."

Output/input token ratio is a workload descriptor, not a quality score. A short answer may legitimately require reading a long contract. Avoid optimizing that ratio at the expense of evidence coverage.

A small experiment worth running

For each evaluation question, record:

question_id, corpus_version, model_version, prompt_version
retrieval_strategy, retrieved_source_ids, evidence_positions
input_tokens, output_tokens, retrieval_ms, rerank_ms, ttft_ms
total_latency_ms, total_cost, correctness, faithfulness, failure_reason

Plot cost against correctness and inspect failures by question type. Do not assume the chart will have a particular shape. Choose a configuration that meets your quality and latency requirements at an acceptable cost, then verify it on held-out questions.

9. Agents Need the Same Discipline at Every Step

In an agent workflow, context can accumulate through tool results, retries, intermediate summaries, and previous decisions. A budget for one call is not a budget for the task.

Keep durable state outside the prompt: source references, completed actions, unresolved questions, and results that must not be recomputed. Assemble the evidence needed for the next step while retaining the information required to justify earlier decisions. Summaries should point back to originals rather than silently replacing them.

Set limits on steps, retries, tool output size, and total cost. Test recovery from partial failure. A compact prompt does not help if the agent repeats an unsuccessful action twenty times.

10. A Practical Implementation Order

Measure a baseline. Create evaluation questions and trace the current system before changing retrieval.
Enforce a complete-request budget. Count all inputs and reserve output capacity. Test overflow and insufficient-evidence paths.
Improve selection. Compare chunking and retrieval approaches, then add reranking if it earns its cost.
Test the difficult cases. Evidence in the middle, multiple required sources, outdated documents, and access boundaries.
Optimize the measured bottleneck. Add caching or compression only when you can show a useful trade-off.

The interesting engineering question is not "How much can this model fit?" It is "What evidence does this answer require, what did we actually send, and how do we know it worked?"

Long context and retrieval are tools we can combine. The judgment lies in knowing what to include, what to leave out, and when the system does not yet have enough evidence.

→ Connect with me on LinkedIn

After months of iterating, scaling, and breaking things (on purpose), I’ve published a detailed blog on building scalable cloud-native infrastructure from scratch using Kubernetes, Terraform, and multi-cloud

Suryadeepsinh Gohil — Wed, 09 Apr 2025 22:20:31 +0000

Suryadeepsinh Gohil

Apr 9 '25

From Idea to Infra: Building Scalable Systems with Kubernetes, Terraform & Cloud (Detailed)

#architecture #devops #kubernetes #terraform

10 min read

From Idea to Infra: Building Scalable Systems with Kubernetes, Terraform & Cloud (Detailed)

Suryadeepsinh Gohil — Wed, 09 Apr 2025 21:53:55 +0000

Introduction: Scale the System You Actually Have

An MVP has two jobs: help you learn whether the product is useful, and leave you a workable path when demand grows. Building a distributed platform before you understand the workload can make both jobs harder.

The difficult part is deciding which constraints deserve attention now. A slow query, an unreliable deployment, and a queue that never catches up are different problems. They do not all need Kubernetes, more services, or another cloud.

This guide follows six phases from architecture to operations. It uses an illustrative analytics API and a separate AdTech design exercise. These are engineering examples, not claims about a client deployment or measured company results. The earlier unsourced company performance table has been removed.

The code is a starting point with explicit prerequisites. Local syntax checks are useful, but they do not prove cloud permissions, networking, recovery, or production capacity. Do not apply infrastructure examples to an account without reviewing their cost and access requirements.

Phase 1: Turn Product Requirements into Constraints

Before drawing service boxes, write down:

The critical user operation and its acceptable latency/error rate.
Expected steady and burst traffic, payload sizes, and data retention.
Which work must finish synchronously and which can be queued.
What must remain correct during retries, duplicate events, and partial failure.
Who will operate the system, including deployments and incidents.

A modular monolith can support substantial traffic. Microservices become useful when there is a concrete reason for independent deployment, ownership, scaling, or isolation. They also introduce network failures, distributed tracing, version compatibility, and data-consistency work. Team size alone is not a reliable decision rule.

An illustrative analytics application

Start with one API, a relational database, object storage for uploads, and a worker queue for long-running analysis. Give background jobs stable identifiers and make retries safe for the operations they perform. Add a streaming platform when you need its ordering, retention, replay, or consumer model, not merely because the architecture diagram looks more scalable.

Client → API → database
           ↓
      job queue → workers → object storage / results

The first scaling question might be whether workers need more capacity. Splitting authentication into a new service would not address that bottleneck.

Phase 2: Make Infrastructure Changes Reviewable

Terraform is useful because infrastructure changes can be versioned, compared, and reviewed. It does not make a configuration correct by itself.

Keep environment roots distinct from reusable modules:

infra/
  modules/
    network/
    database/
  environments/
    dev/
    production/
k8s/
  deployment.yaml
  service.yaml
  hpa.yaml

Use separate state and credentials for environments that need isolation. CLI workspaces are a state-selection mechanism; they are not an access-control boundary.

For an S3 backend, provision the state bucket separately, enable versioning, restrict access, and configure locking. HashiCorp now documents S3 lockfiles through use_lockfile; DynamoDB-based locking is deprecated. State and saved plans can contain sensitive values, so treat their storage and retention accordingly.

An EKS module boundary with explicit inputs

This fragment targets the EKS module's v21 interface. It pins the module so the example is reviewable. Before real deployment, review release/security changes and choose supported Kubernetes and add-on versions for your region. The existing VPC, private subnets, DNS, routing, egress, and operator/CI access are prerequisites, not resources created by this fragment.

terraform {
  required_version = ">= 1.10, < 2.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 6.0"
    }
  }
}

variable "region" {
  type = string
}

variable "kubernetes_version" {
  type = string
}

variable "vpc_id" {
  type = string
}

variable "private_subnet_ids" {
  type = list(string)
}

provider "aws" {
  region = var.region
}

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "21.0.0"

  name                    = "analytics-example"
  kubernetes_version      = var.kubernetes_version
  vpc_id                  = var.vpc_id
  subnet_ids              = var.private_subnet_ids
  endpoint_private_access = true
  endpoint_public_access  = false

  addons = {
    vpc-cni = {
      before_compute = true
    }
    kube-proxy = {}
    coredns    = {}
  }

  eks_managed_node_groups = {
    workers = {
      instance_types = ["m6i.large"]
      min_size       = 2
      desired_size   = 2
      max_size       = 6
    }
  }
}

The node-group bounds are illustrative capacity settings, not a benchmark. Setting a maximum does not install a node autoscaler. Define how node capacity will grow, how workloads spread across failure domains, and how upgrades are tested. Configure EKS access entries and Kubernetes permissions deliberately; creating a cluster is not the same as granting a deployment runner access to it.

For a new local validation directory, terraform init -backend=false downloads modules/providers without initializing remote state, and terraform validate checks configuration consistency. These commands do not provision a cluster. A useful plan still requires real reviewed inputs, provider access, and the intended backend. Apply only the reviewed plan through your controlled release process.

Phase 3: Make Kubernetes Objects Agree

Three objects need to point at the same workload: the Deployment owns matching Pods, the Service selects those Pods, and the HPA targets the Deployment.

The following files assume a pre-existing production namespace. Replace example.invalid/analytics-api:replace-me with your tested image, preferably pinned by digest. The application must listen on port 8080 and implement /readyz and /livez. These placeholders deliberately cannot be mistaken for a supplied application image.

k8s/deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-server
  namespace: production
spec:
  selector:
    matchLabels:
      app: api-server
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: api-server
    spec:
      containers:
        - name: api
          image: example.invalid/analytics-api:replace-me
          ports:
            - name: http
              containerPort: 8080
          resources:
            requests:
              cpu: "500m"
              memory: "256Mi"
            limits:
              cpu: "1"
              memory: "512Mi"
          readinessProbe:
            httpGet:
              path: /readyz
              port: http
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /livez
              port: http
            initialDelaySeconds: 20
            periodSeconds: 10

The Deployment selector must match its Pod template labels. That required relationship was missing from the earlier example. CPU and memory values here are starting values to measure, not production sizing advice.

Readiness controls whether a Pod receives Service traffic; liveness can trigger container restarts. Design the endpoints accordingly. Making liveness fail whenever a shared database is unavailable can turn a dependency outage into a restart storm. Use a startup probe if initialization needs a separate allowance. See the Kubernetes probe documentation.

k8s/service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: api-server
  namespace: production
spec:
  selector:
    app: api-server
  ports:
    - name: http
      port: 80
      targetPort: http
  type: ClusterIP

The Service makes the application reachable inside the cluster. It does not create public ingress, DNS, or TLS.

k8s/hpa.yaml:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-hpa
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api-server
  minReplicas: 3
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

CPU utilization is measured against CPU requests, and the cluster needs a working resource-metrics API, commonly supplied by Metrics Server. The Deployment omits replicas because the HPA manages that value. Without an HPA, set an explicit replica count instead.

HPA scales Pods, not databases or node capacity. Load-test the full path and watch connection pools, queue age, downstream limits, and scheduling failures. More replicas can overload a dependency faster.

Phase 4: Show the Authentication and Deployment Steps

A fresh CI job does not inherit another job's checked-out files or Kubernetes context. A deployment step needs both, plus an authorized identity and a reachable API endpoint.

Keep infrastructure provisioning and application rollout as separate reviewed workflows. The example below deploys an application to an existing EKS cluster. It does not create the cluster or automatically apply Terraform on every push.

Prerequisites:

A runner labeled eks-deploy with AWS CLI v2 and a compatible kubectl, plus network access to the private EKS endpoint. Restrict that runner to trusted deployment workflows.
GitHub's AWS OIDC provider and a least-privilege deployment role. Match the trust policy to the repository/environment and its actual subject-claim format.
An EKS access entry and Kubernetes authorization for that role in production. AWS authentication alone is insufficient.
A GitHub production environment with appropriate deployment protection rules, and environment variables AWS_REGION, AWS_DEPLOY_ROLE_ARN, and EKS_CLUSTER_NAME.
The three checked-in manifests above with a real image and tested health endpoints. Registry access must work from the cluster.

name: Deploy existing EKS application
on:
  workflow_dispatch:

permissions:
  contents: read
  id-token: write

concurrency:
  group: production-application
  cancel-in-progress: false

jobs:
  deploy:
    environment: production
    runs-on: [self-hosted, linux, eks-deploy]
    timeout-minutes: 15
    env:
      AWS_REGION: ${{ vars.AWS_REGION }}
      EKS_CLUSTER_NAME: ${{ vars.EKS_CLUSTER_NAME }}
    steps:
      - uses: actions/checkout@v4

      - name: Assume deployment role through OIDC
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ vars.AWS_DEPLOY_ROLE_ARN }}
          aws-region: ${{ vars.AWS_REGION }}

      - name: Create a job-specific Kubernetes context
        shell: bash
        run: |
          set -euo pipefail
          export KUBECONFIG="$RUNNER_TEMP/eks-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}.config"
          trap 'rm -f "$KUBECONFIG"' EXIT
          aws eks update-kubeconfig \
            --region "$AWS_REGION" \
            --name "$EKS_CLUSTER_NAME" \
            --kubeconfig "$KUBECONFIG" \
            --alias article-production
          kubectl --context article-production -n production apply --dry-run=server -f k8s/
          kubectl --context article-production -n production apply -f k8s/
          kubectl --context article-production -n production rollout status deployment/api-server --timeout=180s

GitHub documents the OIDC permissions and trust configuration, and AWS documents update-kubeconfig and its prerequisites. id-token: write permits requesting an OIDC token; it does not itself grant AWS or Kubernetes access. The example uses readable action version tags; production workflows should pin reviewed commit SHAs and maintain them.

Server-side dry-run exercises admission and validation against the target cluster, but rollout readiness is still not an end-to-end application test. Add a smoke test, watch error/latency changes, and define a rollback decision. A failed rollout command reports failure; it does not automatically restore the previous release.

Choose one owner for application reconciliation

Direct CI deployment and GitOps are alternatives for this workload. With Argo CD automated sync, have CI publish the image and update the desired manifest revision; let Argo CD reconcile the application. Do not also have an unrelated CI job continually apply different desired state to the same objects.

Git history supports review and rollback, but it cannot undo an incompatible database migration or restore lost data. Helm can package reusable manifests, but rendered output still needs validation. Keep plaintext secrets out of Git; if using SOPS, control decryption permissions and validate the encrypted files as part of the workflow.

Phase 5: Treat Recovery as a Tested Procedure

Multi-cloud adds dependencies and operational work. It is not automatically a disaster-recovery plan. A second Kubernetes cluster is of little use if the application cannot reach its database, decrypt data, obtain credentials, or serve the right DNS name.

For an illustrative GKE-to-EKS recovery design, write down:

Dependency	What the recovery test must establish
Compute	Compatible images, capacity, configuration, and network policies
Database	An engine-compatible restore or replication path, tested for consistency
Object storage	Replication lag, version history, access permissions, and deletion behavior
Identity and secrets	Working authentication and decrypt permissions in the recovery environment
Traffic	Health checks, DNS behavior, TLS, and safe failover/failback

Aurora backups and BigQuery exports are not interchangeable database restore mechanisms. Choose a recovery procedure for the actual engine and workload. Likewise, copying objects nightly leaves a possible data-loss window and does not by itself prove recoverability.

Define RPO as the acceptable data-loss window and RTO as the acceptable restoration time. Restore into an isolated environment, run consistency and application checks, and record elapsed time and failure points. A backup job succeeding is only the beginning of that evidence.

Phase 6: Observe User Outcomes and Saturation

Begin with request rate, errors, latency distributions, and saturation. Add queue age and retry/dead-letter counts for asynchronous work. Trace IDs should connect an incoming request to the relevant jobs and downstream calls.

Prometheus, Grafana, and OpenTelemetry are useful options. Choose retention and cardinality deliberately; adding a user ID or request ID as a metric label can make the monitoring system itself expensive.

Here is a complete illustrative Prometheus alert-rule file, not a Grafana provisioning document. It assumes your counter is named http_requests_total and has service and status labels. Adjust the metric names, exclusions, traffic floor, and threshold to your service.

groups:
  - name: api-example
    rules:
      - alert: ApiHighErrorRatio
        expr: |
          (
            sum(rate(http_requests_total{service="api",status=~"5.."}[5m]))
            /
            sum(rate(http_requests_total{service="api"}[5m]))
          ) > 0.05
          and
          sum(rate(http_requests_total{service="api"}[5m])) > 1
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "API error ratio exceeds the illustrative threshold"

The traffic floor avoids treating a single low-volume failure as the same situation as a sustained incident. Missing metrics still need their own alert; no data is not proof of health. See Prometheus alerting-rule syntax and validate with promtool check rules in your monitoring project.

For an SLO, first define the eligible operation, what counts as good, and the measurement window. An illustrative request-based availability objective of 99.95% permits a bad-request fraction of 0.05%. Track budget consumption over multiple windows; a fixed 5% error alert is not a complete error-budget policy.

Design Exercise: An AdTech Platform Under Bursty Load

Consider a hypothetical bidding API with campaign management, asynchronous analytics, and fraud checks. This section is a design exercise, not an account of Magnite's architecture or results.

Separate latency-critical decisions from work that can finish later. A cached campaign record may accelerate the read path, but budget updates still need concurrency control. A local check like BudgetLeft > bidPrice cannot prevent two simultaneous requests from overspending the same budget. Decide where reservation, expiry, and reconciliation happen, and test retries and races.

For the bidding workload, use the same Deployment selector/Pod-label relationship shown earlier, with app: bidding-engine. Add measured CPU and memory requests before using utilization-based autoscaling. Benchmark the full request path rather than asserting a latency based on the programming language or Redis alone.

The evidence worth collecting is concrete:

Question	Test and record
Does the API meet its latency target?	Traffic shape, payload distribution, p50/p95/p99, error rate, warm-up, and test duration
Does autoscaling help?	Burst arrival, scale-up lag, queue age, pending Pods, and downstream saturation
Are budget decisions correct?	Concurrent requests, duplicate delivery, timeout/retry cases, and reconciliation mismatches
Can a release be recovered?	Failed rollout, rollback time, schema compatibility, and user-visible impact
Is the design affordable?	Cost per defined workload unit, including idle capacity and observability

Publish before/after numbers only when you can explain the workload, infrastructure, measurement window, and the change that produced them. A table of impressive figures without that context is not engineering evidence.

A Release Checklist That Earns Its Place

Validate configuration and render the exact artifacts that will be deployed.
Review the infrastructure plan, identities, network reachability, and cost implications.
Exercise application health checks, dependency failure, and representative load in a suitable test environment.
Verify deployment, smoke-test behavior, and observe the service against its acceptance criteria.
Practice recovery, including data and configuration, before depending on it during an incident.

Treat infrastructure as a product: versioned, tested, documented, and owned. Add complexity when a measured requirement justifies it. The strongest architecture is one the team can explain, operate, and recover, with evidence for the claims it makes.

→ Connect with me on LinkedIn