DEV Community: Srijan Verma

Stop Letting Flaky APIs Crash Your AI Agents

Srijan Verma — Sun, 30 Aug 2026 12:22:39 +0000

How to combine exponential backoff, circuit breakers, and graceful fallbacks for production-grade agentic workflows.

The Bottleneck in Production

AI agents are only as reliable as the tools they invoke. When an LLM decides to search the web, scrape a URL, or fetch database records, it depends entirely on network stability.

In production, external APIs fail constantly. A sudden surge causes 429 rate limits, a third-party microservice throws a 504 timeout, or a target endpoint goes down entirely. The naive approach—executing raw tool calls directly inside the agent loop—is a ticking time bomb:

# The Naive Anti-Pattern: Fragile Tool Execution
def execute_agent_tool(tool_name: str, payload: dict):
    # One 500 error here kills the entire multi-step reasoning chain
    response = requests.post(f"https://api.service.internal/{tool_name}", json=payload)
    return response.json()

When this call breaks, the unhandled exception crashes the runtime. You lose the entire reasoning graph, waste LLM tokens, and degrade the user experience.

The System Architecture: Layered Tool Defense

To keep multi-step agents alive, you need a defensive execution pipeline wrapped around every tool. Instead of allowing errors to bubble up and kill the agent, we handle failures across three distinct layers:

Exponential Backoff: Mitigate transient network glitches and minor rate spikes by retrying with increasing delays.
Circuit Breaker: Detect persistent downtime. If an API fails three times consecutively, trip the breaker to stop sending doomed requests.
Graceful Fallbacks & Partial Degradation: When a primary service is down, route the query to a replica, cached store, or lightweight fallback (e.g., cached search index instead of a live browser scrape).

[ Agent Core ] 
      │
      ▼
┌───────────────────────────────┐
│     Circuit Breaker Check     │
│   (Is Primary Service Up?)    │
└──────────────┬────────────────┘
       OPEN    │   CLOSED (Healthy)
       ┌───────┴────────┐
       ▼                ▼
┌─────────────┐  ┌───────────────────────────┐
│  Fallback   │  │ Retry Engine (Backoff)    │
│  Provider   │  │ └──> Primary API Endpoint │
└──────┬──────┘  └──────────────┬────────────┘
       │                        │ SUCCESS
       │ (Degraded)             │
       ▼                        ▼
┌────────────────────────────────────────────┐
│      Structured Response + Metadata        │
│   (LLM receives context of degradation)    │
└────────────────────────────────────────────┘

By returning a degraded result accompanied by metadata (e.g., status: "degraded", source: "fallback_cache"), the LLM can adjust its downstream reasoning rather than hallucinating over missing data.

The Implementation

We combine tenacity for retry logic with circuitbreaker to isolate failing services. The following production-ready pattern ensures failures are caught and handled before reaching the LLM orchestrator.

from tenacity import retry, stop_after_attempt, wait_exponential
from circuitbreaker import circuit, CircuitBreakerError

# 1. Protect external call with circuit breaker and exponential backoff
@circuit(failure_threshold=3, recovery_timeout=60)
@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=2, max=10))
def fetch_primary_data(query: str) -> dict:
    resp = requests.get("https://api.flaky-service.com/v1/search", params={"q": query}, timeout=3)
    resp.raise_for_status()
    return {"data": resp.json(), "source": "primary", "degraded": False}

# 2. Resilient fallback wrapper for the agent runtime
def execute_tool_safely(query: str) -> dict:
    try:
        return fetch_primary_data(query)
    except (CircuitBreakerError, Exception) as err:
        # Fallback to internal cache or secondary tool
        cached_result = local_cache.get(query) or "No fresh data available."
        return {
            "data": cached_result,
            "source": "cache_fallback",
            "degraded": True,
            "error_context": str(err)
        }

This snippet ensures three critical guarantees:

Idempotent Retry Safety: The request backs off exponentially up to 10 seconds.
Fail-Fast Protection: Once the breaker opens, execution drops straight to the fallback without waiting for timeouts.
Agent Loop Continuity: The orchestrator receives a structured dictionary containing error metadata instead of an uncaught exception.

Production Lessons & Takeaways

Isolate Circuit Breakers Per Tool: Never use a single global circuit breaker. Wrap each external integration independently so a failing weather API doesn't disable your payment tool.
Pass Degradation Metadata to the Prompt: When serving fallback data, explicitly inform the model via context ("Note: Live search is offline. Using cached results from 2 hours ago."). This keeps the model's responses

Stop Choosing Between BM25 and Vector Search: Implement Hybrid Search with RRF

Srijan Verma — Fri, 28 Aug 2026 22:54:53 +0000

Combine dense embeddings and sparse keyword search using Reciprocal Rank Fusion to eliminate retrieval failure modes in production RAG systems.

The Bottleneck in Production

Most production RAG pipelines start with pure vector search. It works reliably during initial demos, but fails quietly once users begin searching for exact product IDs, error codes, or domain-specific identifiers.

Vector search translates text into semantic coordinate spaces. Because dense models optimize for high-level concepts, they tend to blur fine-grained details. A search for ERR-502-BAD-GATEWAY might retrieve general network troubleshooting docs instead of the exact runbook for error 502.

Conversely, relying purely on keyword search (BM25) breaks when users paraphrase. A query for "reduce database memory footprint" completely misses an article titled "Mitigating PostgreSQL RAM Saturation" if there is no direct keyword overlap.

# The Naive Approach: Semantic-only retrieval misses exact tokens
def get_context(query: str, vector_db) -> list[str]:
    # Fails when query contains exact SKUs, UUIDs, or specific error logs
    return vector_db.similarity_search(query, k=5)

Relying on a single retrieval strategy creates hard blind spots that degrade downstream LLM generation quality.

The Architecture: Concurrent Hybrid Search & RRF

The fix is running sparse (BM25) and dense (vector) searches concurrently and merging their ranked outputs using Reciprocal Rank Fusion (RRF).

                     ┌───> [ BM25 Keyword Search ] ───> Sparse Ranked List ───┐
[ User Query ] ──────┤                                                         ├───> [ RRF Fusion ] ───> Top-K Documents ───> LLM
                     └───> [ Dense Vector Search ] ───> Dense Ranked List  ───┘

RRF avoids the core challenge of hybrid search: score normalization. BM25 produces unbounded positive floating-point scores, while vector search usually outputs cosine similarities between -1 and 1. Merging raw scores requires fragile heuristic scaling factors.

Instead of comparing raw scores, RRF scores documents based strictly on their relative rank position across both search lists:

$$RRF(d) = \sum_{m \in M} \frac{1}{k + r_m(d)}$$

Where:

$M$ is the set of retrieval systems (BM25 and Vector).
$r_m(d)$ is the 1-based rank position of document $d$ in system $m$.
$k$ is a smoothing constant (typically set to 60) that prevents low-ranking outliers from disproportionately skewing results.

The Implementation

Here is a minimal, production-ready implementation combining BM25, dense embeddings, and RRF in under 25 lines of Python:

import numpy as np
from rank_bm25 import BM25Okapi
from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity

def hybrid_rrf_search(query: str, corpus: list[str], top_n: int = 3, k: int = 60) -> list[str]:
    # 1. Sparse BM25 scoring & ranking
    bm25 = BM25Okapi([doc.lower().split() for doc in corpus])
    bm25_rank = np.argsort(bm25.get_scores(query.lower().split()))[::-1]

    # 2. Dense Vector scoring & ranking
    model = SentenceTransformer('all-MiniLM-L6-v2')
    doc_embs, query_emb = model.encode(corpus), model.encode([query])
    dense_rank = np.argsort(cosine_similarity(query_emb, doc_embs)[0])[::-1]

    # 3. Reciprocal Rank Fusion
    rrf_scores = {}
    for rank_idx, doc_idx in enumerate(bm25_rank):
        rrf_scores[doc_idx] = rrf_scores.get(doc_idx, 0.0) + (1.0 / (k + rank_idx + 1))
    for rank_idx, doc_idx in enumerate(dense_rank):
        rrf_scores[doc_idx] = rrf_scores.get(doc_idx, 0.0) + (1.0 / (k + rank_idx + 1))

    sorted_docs = sorted(rrf_scores.items(), key=lambda x: x[1], reverse=True)
    return [corpus[doc_idx] for doc_idx, _ in sorted_docs[:top_n]]

This pattern guarantees resilience. If a user inputs a technical keyword match, BM25 pushes the correct document to the top. If a user describes a high-level conceptual problem, dense

Stop Using Regex: Building Zero-Crash LLM JSON Pipelines in Production

Srijan Verma — Thu, 27 Aug 2026 11:50:37 +0000

Replace brittle string parsing with schema-enforced validation and self-healing repair layers.

The Bottleneck in Production

Most LLM pipelines don't break because the model generated bad logic. They break because the model violated your backend's JSON parsing contract.

When running LLMs in production services, you will inevitably hit:

Trailing commas, missing closing brackets, and unescaped quotes.
Truncated strings caused by token limit exhaustion.
Safety refusals that return plain text instead of the requested JSON schema.

Engineers usually respond with fragile regex hacks and nested try/except blocks:

# The classic production anti-pattern
try:
    match = re.search(r"\{.*\}", response.content, re.DOTALL)
    data = json.loads(match.group(0))
except Exception:
    logger.error("LLM failed to return valid JSON")  # Pipeline crashes here

This pattern guarantees a steady 5% to 15% error rate at scale. You cannot treat probabilistic token generators like deterministic REST APIs without an enforcement boundary.

The System Architecture & Fix

To achieve 99.9% pipeline reliability, you must shift from post-hoc string manipulation to a Three-Layer Validation Pattern:

Pre-Sanitization: Strip unexpected control characters, BOM markers, and markdown code fences before parsing.
Strict Schema Binding: Bind the output directly to a Pydantic model at the API layer using native SDK structured outputs.
Targeted Repair Fallback: If schema validation fails due to token truncation or malformed keys, route the raw output to a cheap, high-speed repair step instead of dropping the request.

[ Raw User Input ]
        │
        ▼
[ Pre-Cleaner (Strip Control Chars / Markdown) ]
        │
        ▼
[ LLM Provider + Pydantic Schema Enforcement ]
        │
        ├──► Valid Schema? ────► [ Downstream Backend Services ]
        │
        └──► Parse Error / Truncated?
                    │
                    ▼
        [ Lightweight Repair Adapter (Fast/Cheap LLM) ]
                    │
                    ▼
        [ Validated Pydantic Object ]

By offloading the schema enforcement to the provider's constrained decoding and validating with Pydantic, your backend code deals strictly with typed objects.

The Implementation

Here is the modern, production-grade pattern using native Pydantic parsing with OpenAI's structured outputs:

from pydantic import BaseModel, Field
from openai import OpenAI

class UserProfile(BaseModel):
    user_id: str
    email: str
    confidence_score: float = Field(ge=0.0, le=1.0)
    tags: list[str] = []

client = OpenAI()

def extract_profile(raw_input: str) -> UserProfile:
    completion = client.beta.chat.completions.parse(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "Extract user metadata from input."},
            {"role": "user", "content": raw_input}
        ],
        response_format=UserProfile,
    )
    # Automatically parsed and validated against the Pydantic schema
    return completion.choices[0].message.parsed

Why This Pattern Works

Zero Regex: The API forces the engine's token sampling to follow the JSON Schema derived directly from UserProfile.
Compile-Time Type Safety: Your IDE and downstream services receive an instantiated UserProfile instance, not a generic dict.
Deterministic Latency: Eliminates the overhead of writing custom retry loops for common syntax errors.

Production Lessons & Takeaways

Check finish_reason First: Always verify choice.finish_reason == "stop". If it equals "length", your payload was cut off by max_tokens, and attempting to parse the JSON string will fail every time.
Never Write Manual Extraction Regex: Regex fails on nested JSON arrays, escaped quotes, and newlines inside strings. Use native SDK structured outputs (.parse()) or tools like instructor.
Use Cheap Fallback Adapters: When working with open-source models that lack native constrained decoding, pipe failed JSON through a sub-second model (like Gemini Flash or Claude 3.5 Haiku) with the prompt: "Fix this invalid JSON to match schema X. Output raw JSON only."

Surviving the 429 Storm: Building Resilient LLM Fallbacks in Production

Srijan Verma — Wed, 26 Aug 2026 11:50:42 +0000

How to eliminate cascading failures from LLM rate limits using exponential backoff, circuit breakers, and tiered fallback chains.

The Bottleneck in Production

Most production LLM integrations start with a direct SDK call wrapped in a naive try/except block. During quiet periods, it works fine. But when traffic spikes, your application hits provider Token-Per-Minute (TPM) or Request-Per-Minute (RPM) limits, throwing HTTP 429 errors.

The immediate reaction is often an uncontrolled retry loop. That is an anti-pattern:

# Anti-pattern: The self-inflicted DDoS
for _ in range(5):
    try:
        return openai_client.chat.completions.create(...)
    except Exception:
        time.sleep(0.5)  # Thrashes the API and worsens rate limits

When 100 concurrent workers hammer an already-throttled provider with instant retries, you trigger a cascading failure. Your API workers block, thread pools exhaust, upstream quotas remain locked, and raw JSON errors leak to your users.

The System Architecture & Fix

Rate limits and transient outages are not anomalies; they are environmental constraints. A production-grade backend requires a layered defensive strategy rather than raw retries.

A resilient LLM architecture relies on a 3-layer safety net:

Jittered Exponential Backoff: Retries transient network glitches and brief 429/503 spikes over increasing intervals (e.g., 2s, 4s, 8s) to let upstream token buckets refill.
Dynamic Fallback Routing: If retries exhaust or provider error rates cross a defined threshold, automatically route the payload to a secondary, cheaper, or alternative provider (e.g., from GPT-4o to Claude 3.5 Haiku or a self-hosted vLLM instance).
Graceful Degradation: If all models fail, serve a semantic cache match or a structured, user-friendly fallback response instead of failing the request.

[Incoming User Request]
         │
         ▼
 ┌──────────────┐    HTTP 429/5xx     ┌───────────────────────┐
 │ Primary LLM  │ ──────────────────► │ Exponential Backoff   │
 └───────┬──────┘   (Retries Exhaust) └───────────┬───────────┘
         │                                        │
         │ Success                                ▼
         │                             ┌───────────────────────┐
         │                             │ Secondary / Fast Model│
         │                             └───────────┬───────────┘
         │                                         │
         ▼                                         ▼ Failed
  [Final Response] ◄─────────────────── ┌───────────────────────┐
                                        │ Cache / Static Helper │
                                        └───────────────────────┘

The Implementation

Using tenacity, we can implement exponential backoff targeting specific HTTP status codes, paired with an automated fallback handler.

Here is a minimal, robust implementation:

import openai
from tenacity import retry, stop_after_attempt, wait_exponential, retry_if_exception

def is_retryable_error(exception: BaseException) -> bool:
    status_code = getattr(exception, "status_code", None)
    return status_code in {429, 500, 502, 503, 504}

def fallback_completion(prompt: str) -> str:
    # Secondary model fallback or cached response
    client = openai.OpenAI()
    fallback = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
    )
    return fallback.choices[0].message.content

@retry(
    stop=stop_after_attempt(3),
    wait=wait_exponential(multiplier=1, min=2, max=8),
    retry=retry_if_exception(is_retryable_error),
    reraise=False,
)
def generate_response(prompt: str) -> str:
    try:
        client = openai.OpenAI()
        response = client.chat.completions.create(
            model="gpt-4o",
            messages=[{"role": "user", "content": prompt}],
        )
        return response.choices[0].message.content
    except Exception:
        return fallback_completion(prompt)

This pattern isolates failures. If the primary model encounters a 429, it backs off cleanly without monopolizing compute. If the rate limit persists past three attempts, the execution transparently transfers to the fallback model without failing the upstream pipeline.

Taming Context Bloat: How to Scale AI Agent Memory Without Breaking the Token Bank

Srijan Verma — Tue, 25 Aug 2026 12:59:20 +0000

Stop dumping raw message arrays into LLMs and start using structured state with sliding windows.

The Bottleneck in Production

The most common mistake when deploying AI agents is treating chat history as an append-only log. In early prototypes, appending every user turn, tool response, and raw JSON blob directly into the messages array works fine.

In production, this pattern collapses after twenty turns. Token usage scales linearly with conversation depth, driving up API latency and inference costs. Worse, models experience "lost-in-the-middle" degradation, forgetting early constraints or crashing altogether due to token limit errors.

# The naive anti-pattern: unbounded list growth
messages.append({"role": "user", "content": user_input})
messages.append({"role": "assistant", "content": llm_response})
# 30 turns later: 15,000 tokens wasted on stale tool payloads
response = client.chat.completions.create(model="gpt-4o", messages=messages)

Dumping unpruned histories into your LLM turns your database into an expensive latency trap.

The System Architecture & Fix

The solution is decoupling ephemeral dialogue from persistent conversational state.

Instead of forcing the LLM to re-parse the entire conversation history on every turn to understand what happened ten minutes ago, we split context into two distinct layers:

Sliding Window Buffer: Retain only the last $N$ turns to preserve immediate conversational rhythm and flow.
Structured State JSON: Maintain a compact, deterministic key-value store of critical facts (e.g., active user goals, collected form values, resolved errors).

   Incoming User Turn
           │
           ▼
┌─────────────────────────────────────────┐
│            Context Assembler            │
│ ─────────────────────────────────────── │
│ 1. Static System Prompt (Identity)      │
│ 2. Current State JSON (Facts & Goals)   │
│ 3. Sliding Window Buffer (Last N Turns) │
└─────────────────────────────────────────┘
           │
           ▼
     LLM Inference (Bounded & Predictable)

This ensures your token payload stays flat whether a session lasts 3 turns or 300 turns.

The Implementation

Here is a lightweight context manager you can drop directly into your backend service pipeline.

from typing import Any, Dict, List

def build_bounded_context(
    system_prompt: str,
    raw_history: List[Dict[str, str]],
    state_payload: Dict[str, Any],
    max_turns: int = 6
) -> List[Dict[str, str]]:
    """Assemble a token-bounded context payload with structured state."""
    # Enforce strict sliding window on ephemeral chat history
    trimmed_history = raw_history[-max_turns:] if len(raw_history) > max_turns else raw_history

    # Inject current state directly as a system-level context injection
    state_injection = {
        "role": "system",
        "content": f"CURRENT_SESSION_STATE: {state_payload}"
    }

    return [{"role": "system", "content": system_prompt}, state_injection] + trimmed_history

This pattern provides deterministic context bounds. Your backend guarantees that the context size passed to the provider never exceeds your calculated budget:

If an agent needs to update persistent state (like a shipping address or user intent), extract that state asynchronously or via tool calls, store it in your database, and inject the clean JSON dictionary on the next invocation.

Production Lessons & Takeaways

State belongs in your database, not the context window: Treat the LLM context like RAM and your database like a hard drive. Extract facts into structured fields instead of making the model re-read 40 turns of history.
Keep $N$ between 4 and 8 turns: In practice, immediate dialogue context rarely requires more than the last 3-4 round trips. Anything older is usually noise.
Make state updates idempotent: Update your structured state schema using explicit tool/function calls rather than free-form LLM summaries to avoid hallucinated state drift.

Stop Passing Full Chat Transcripts: Build a Two-Tier AI Memory System

Srijan Verma — Mon, 24 Aug 2026 09:29:54 +0000

Decouple ephemeral conversation context from persistent user preferences with a hybrid cache and KV store.

Most developer tutorials show you how to build AI agents by appending every user message to a local Python list. While this works in a notebook, it immediately falls apart in production backends.

The Bottleneck in Production

The naive approach stores chat histories directly in process memory or dumps entire conversation logs back into the LLM context window on every turn.

# The Anti-Pattern: Unbounded memory growth & volatile storage
chat_history.append({"role": "user", "content": prompt})
response = openai.chat.completions.create(model="gpt-4o", messages=chat_history)

This pattern breaks for three reasons:

Volatile Lifecycle: If your container restarts or scales down, the user's entire interaction state vanishes.
Context Bloat & Cost: Sending 30 turns of raw conversation spikes latency and blows through token budgets without adding meaningful signal.
Loss of Critical Facts: When the message list inevitably hits context limits, naive truncation drops early messages—erasing core details like dietary preferences, user IDs, or system constraints.

The System Architecture & Fix

To build a production-grade agent, you must decouple conversational context (short-term) from durable user preferences (long-term).

Instead of maintaining a massive array of raw dialogue, implement a two-tier memory architecture:

Tier 1 (L1 - Short-Term Session Cache): A fast, in-memory store (or Redis) that tracks the active session's recent dialogue turns and immediate state.
Tier 2 (L2 - Long-Term Persistent KV Store): A durable key-value store (DynamoDB, Postgres JSONB, or S3) containing structured, long-lived facts about the user.

                  +-----------------------+
                  |      User Request     |
                  +-----------+-----------+
                              |
                              v
                  +-----------------------+
                  |    Agent Controller   |
                  +-----+-----------+-----+
                        |           |
       1. Read Session  |           | 2. Read Preferences
                        v           v
             +------------+       +-------------+
             |  L1 Cache  |       | L2 KV Store |
             |  (Redis)   |       |  (Database) |
             +------------+       +-------------+
                        |           |
                        +-----+-----+
                              |
                              v
                  +-----------------------+
                  | Merged Prompt Context |
                  +-----------+-----------+
                              |
                              v
                  +-----------------------+
                  |     LLM Provider      |
                  +-----------------------+

When a request arrives, the agent loads structured facts from Tier 2, merges them with the recent context from Tier 1, and passes an optimized

Bulletproofing AI Agents: How to Prevent $2,000 Infinite API Loops

Srijan Verma — Sat, 22 Aug 2026 12:38:17 +0000

Implement multi-layer circuit breakers, payload hashing, and financial cutoffs before an autonomous agent drains your backend.

The Bottleneck in Production

Autonomous AI agents running in tool-use loops fail unpredictably. When an LLM encounters an unexpected schema, a transient network error, or an ambiguous prompt, it often enters a hallucinated retry storm.

In standard web apps, a runaway loop hits a rate limit or returns a 500 Internal Server Error. In agentic architectures, an unconstrained ReAct loop executes external API calls continuously, burning tokens, exhausting upstream quotas, and running up massive cloud bills in minutes.

Here is the anti-pattern running in far too many codebases:

# Anti-pattern: Unbounded autonomous agent loop
while not task_complete:
    action = llm.decide_action(state)
    result = external_api.call(action.endpoint, action.params)
    state = update_state(result)

If the LLM fails to transition state due to an unparseable response, this loop runs indefinitely. Cloud providers do not issue refunds for self-inflicted API usage.

The System Architecture & Fix

To make AI agent tool execution production-safe, never allow direct API calls from agent code. Route every external request through an isolated API Safety Wrapper implementing three distinct layers of defense:

Deterministic Request Firewall: A hard cap on execution count per task session (Time-To-Live counter).
Sliding-Window Loop Detector: Hashing outgoing request payloads to catch repetitive or oscillating tool invocations.
Financial Kill Switch: A pre-flight budget validator that cuts credentials immediately if projected cost exceeds session limits.

[ AI Agent Engine ]
        │
        ▼
[ API Safety Wrapper ]
   ├── 1. Call Counter Check (Limit < N)
   ├── 2. Hash Duplicate Detector (Window: last 3 calls)
   └── 3. Pre-flight Cost Estimator (Budget < Limit)
        │
   ┌────┴──────────────────────────┐
[ Passed ]                    [ Tripped ]
   │                               │
   ▼                               ▼
[ External Upstream API ]     [ Emergency Kill Switch ]
                              (Revoke Token & Abort)

This ensures that even if an agent hallucinates or crashes, the blast radius is strictly confined to a single session budget.

The Implementation

Here is a lightweight, production-ready safety wrapper that you can wrap around any HTTP client or SDK.

import hashlib

class APISafetyWrapper:
    def __init__(self, client, max_calls: int = 50, budget_limit: float = 5.0):
        self.client = client
        self.max_calls = max_calls
        self.budget_limit = budget_limit
        self.history = []
        self.total_cost = 0.0

    def execute(self, endpoint: str, payload: dict, estimated_cost: float = 0.02):
        sig = hashlib.md5(f"{endpoint}:{sorted(payload.items())}".encode()).hexdigest()

        if len(self.history) >= self.max_calls:
            raise RuntimeError(f"Circuit Breaker: Hard limit ({self.max_calls}) reached.")

        if self.history[-3:].count(sig) >= 2:
            raise RuntimeError(f"Loop Detected: Repeating payload sent to {endpoint}.")

        if (self.total_cost + estimated_cost) > self.budget_limit:
            self.client.revoke_credentials()  # Emergency shutdown
            raise PermissionError("Budget Exceeded: Financial kill-switch triggered.")

        self.history.append(sig)
        self.total_cost += estimated_cost
        return self.client.call(endpoint, payload)

Stop Choosing Between RAG, Fine-Tuning, and Agents: Build the Hybrid Trio Instead

Srijan Verma — Fri, 21 Aug 2026 20:07:21 +0000

How to combine vector search, fine-tuned lightweight models, and targeted agent tooling into a resilient production backend.

Most production AI failures happen because engineering teams treat RAG, Fine-Tuning, and AI Agents as mutually exclusive choices.

They pick one hammer and try to solve every problem with it.

THE BOTTLENECK IN PRODUCTION

When you rely solely on RAG, you end up stuffing 40-page PDFs and massive prompt instructions into a single context window. Your latency climbs past 4 seconds, your token bill explodes, and the model still fails to return valid JSON.

Conversely, if you try to fine-tune your way out of the problem, your model bakes in stale data. The moment your pricing or API contracts change next week, you are stuck retraining weights.

And if you build an unconstrained multi-agent loop to orchestrate everything dynamically, you invite runaway token consumption and unpredictable execution loops.

Here is the anti-pattern running in too many production backends right now:

# The Naive Anti-Pattern: Monolithic prompt bloat
response = openai.ChatCompletion.create(
    model="gpt-4",
    messages=[
        {"role": "system", "content": "You are an agent. Follow this 50-rule schema: {...}. Here are 10 docs: [...]"},
        {"role": "user", "content": user_query}
    ]
)

This brute-force approach collapses under real user loads.

THE SYSTEM ARCHITECTURE & FIX

The solution is a hybrid architecture where each component does exactly one job well:

RAG acts as the Fact Engine: It retrieves volatile, real-time context (pricing, policy docs, inventory).
Fine-Tuning acts as the Format Engine: A small, fine-tuned model (like Llama 3 or Mistral) guarantees deterministic JSON formatting and brand voice without multi-shot prompt overhead.
The Agent acts as the Action Engine: It parses the verified schema and calls downstream internal APIs safely.

               ┌───────────────────────┐
               │      User Query       │
               └──────────┬────────────┘
                          │
                          ▼
               ┌───────────────────────┐
               │  Vector DB Retrieval  │  <-- Dynamic Facts (RAG)
               └──────────┬────────────┘
                          │
                          ▼
               ┌───────────────────────┐
               │ Fine-Tuned Small LLM  │  <-- Strict Schema & Voice
               └──────────┬────────────┘
                          │
                          ▼
               ┌───────────────────────┐
               │   API / Tool Runner   │  <-- Deterministic Action
               └───────────────────────┘

THE IMPLEMENTATION

Here is a clean, reliable pattern in Python that separates knowledge retrieval from structured action execution:

from typing import Dict, Any

class HybridPipeline:
    def __init__(self, vector_store, fine_tuned_llm, api_client):
        self.vector_store = vector_store
        self.llm = fine_tuned_llm
        self.api_client = api_client

    def execute_query(self, user_query: str) -> Dict[str, Any]:
        # 1. Fetch real-time facts
        context = self.vector_store.similarity_search(user_query, k=3)

        # 2. Generate structured payload via fine-tuned model
        prompt = f"Context:\n{context}\n\nTask: {user_query}"
        structured_action = self.llm.generate_json(prompt)

        # 3. Deterministic tool execution
        if structured_action.get("action") == "trigger_api":
            return self.api_client.post(
                endpoint=structured_action["endpoint"], 
                payload=structured_action["params"]
            )

        return {"status": "success", "data": structured_action["response"]}

Why This Pattern Works

Token Efficiency: The prompt does not need 1,000 tokens of schema instructions because the fine-tuned model already knows its exact output schema.
Data Freshness: Dynamic variables remain in the vector database, eliminating the need to retrain when docs change.
Fail-Safe Execution: The agent doesn't write arbitrary code; it simply triggers predefined internal API contracts using parsed parameters.

PRODUCTION LESSONS & TAKEAWAYS

Separate Style from Facts: Fine-tune for formatting, grammar, and schema compliance. Use RAG for anything that changes more frequently than your deployment cycle.
Distill Down to Smaller Models: Generate training sets using GPT-4 to fine-tune 8B-parameter open-source models. You get sub-second latency and cut inference costs by up to 90%.
Constrain Agent Scope: Never start with an open-ended autonomous agent loop. Start with single-tool determinism (e.g., direct CRM lookup or refund dispatch) before layering complex agent chains.

Stop Leaking Data to Cloud APIs: Self-Hosting DeepSeek Locally with Ollama

Srijan Verma — Fri, 21 Aug 2026 11:22:54 +0000

Cut inference bills to zero and eliminate third-party data leaks by running quantized LLMs on your own hardware.

The Bottleneck in Production

Most backend teams start their LLM journey by piping sensitive user data directly into cloud APIs. While convenient for day-one prototypes, this approach falls apart under real-world production constraints.

Every outbound API call introduces network latency (often 800ms+ per round-trip), tight rate limits, and uncontrollable per-token billing spikes. More critically, sending internal system logs, proprietary code, or PII to third-party endpoints violates basic security baselines and compliance requirements (GDPR, SOC2, HIPAA).

# The Naive Approach: Leaking raw internal logs to a remote cloud API
import openai

def analyze_trace(error_trace: str):
    # Sends proprietary code & internal stack traces across the public web
    return openai.ChatCompletion.create(
        model="gpt-4",
        messages=[{"role": "user", "content": f"Analyze: {error_trace}"}]
    )

When your microservice processes millions of requests or handles air-gapped data, remote APIs are an operational liability.

The System Architecture & Fix

The solution is to decouple your pipeline from the cloud by deploying a local inference engine. Ollama manages model lifecycles, memory allocation, and quantization under a unified C++ runtime with a clean HTTP interface.

Instead of paying a 30-second penalty to load 70GB unquantized weights, Ollama runs 4-bit/8-bit quantized models (GGUF format). This allows high-reasoning models like DeepSeek-R1 (1.5B to 8B parameters) to execute entirely within unified memory (Apple Silicon) or standard consumer VRAM.

[Client Application / Microservice]
                │
                ▼ (Local HTTP / gRPC via localhost:11434)
┌──────────────────────────────────────────────┐
│  OLLAMA RUNTIME (Local Daemon)               │
│  ┌────────────────────────────────────────┐  │
│  │ Model Manager & Context Cache          │  │
│  └───────────────────┬────────────────────┘  │
│                      ▼                       │
│  ┌────────────────────────────────────────┐  │
│  │ Quantized Inference (GGUF Engine)      │  │
│  └───────────────────┬────────────────────┘  │
└──────────────────────┼───────────────────────┘
                       ▼
       [Local Hardware: VRAM / RAM]
         (Zero External Network Calls)

Requests hit your local socket, inference runs directly on your silicon, and zero bytes leave your machine.

The Implementation

Below is a robust, production-ready pattern using the official Python client. It streams responses locally to maintain a fast time-to-first-token (TTFT) while handling runtime exceptions.

import ollama

def stream_local_inference(prompt: str, model: str = "deepseek-r1:1.5b") -> str:
    """Executes local LLM inference without outbound network calls."""
    try:
        response_stream = ollama.generate(
            model=model,
            prompt=prompt,
            stream=True,
            options={"temperature": 0.2, "num_ctx": 4096}
        )
        full_text = []
        for chunk in response_stream:
            token = chunk.get("response", "")
            print(token, end="", flush=True)
            full_text.append(token)
        return "".join(full_text)
    except ollama.ResponseError as err:
        return f"Inference failed: {err.error}"

if __name__ == "__main__":
    result = stream_local_inference("Optimize this query: SELECT * FROM users;")

Why This Works

Zero Egress: All data processing is bound to localhost:11434.
Streaming by Default: Eliminates perceived latency by yielding tokens immediately as the GPU generates them.
Deterministic Context: The num_ctx parameter explicitly bounds local memory usage, preventing out-of-memory (OOM) crashes under high loads.

Production Lessons & Takeaways

Size Models to Your VRAM Budget: Start with smaller quantized builds (e.g., deepseek-r1:1.5b or 7b). Running a model that spills out of VRAM into system swap memory degrades performance by up to 10x.
Always Set Context Windows (num_ctx): The default context size can dynamically balloon your RAM footprint. Lock it down to the exact size your workload requires.
Implement Local Circuit Breakers: If local inference latency exceeds your SLA during sudden traffic spikes, route non-sensitive fallback traffic to a cold replica before failing over to the cloud.

Deterministic Guardrails: Stop LLM Tool Calling Failures at Runtime

Srijan Verma — Thu, 20 Aug 2026 11:46:40 +0000

How to prevent malformed agent payloads and execution bugs using strict pre-execution schema enforcement.

The Bottleneck in Production

Most production failures in AI agent pipelines happen at the boundary between probabilistic LLM generation and deterministic system execution.

When you ask an LLM to generate parameters for a database query, file operation, or external API call, it often hallucinates invalid fields, emits wrong primitive types (e.g., string integers instead of real numbers), or truncates the JSON payload.

The naive approach lets these raw payloads hit production APIs, relying on downstream try/catch blocks:

# The Anti-Pattern: Blind Execution
raw_json = json.loads(llm_response)
# If the LLM hallucinated 'user_id' as a string or omitted 'limit', this explodes in production
db.execute_query(raw_json["query"], raw_json["params"])

Post-execution error handling is too late. It pollutes backend logs with unhandled exceptions, risks partial state mutations, and leaves execution pathways vulnerable to prompt injection attacks.

The System Architecture & Fix

To build reliable agents, you must treat LLM outputs as untrusted user inputs. Introduce a deterministic, pre-execution validation gateway that enforces strict schema validation and isolates runtime execution.

Instead of passing the LLM payload directly to internal microservices, route the output through a validation barrier before dispatching to a sandboxed worker.

+----------------+      +----------------+      +-------------------------+
|  User Prompt   | ---> |  LLM Inference | ---> | Pre-Execution Guardrail |
+----------------+      +----------------+      | (Pydantic / Strict JSON)|
                                                +-------------------------+
                                                             |
                                      [ Invalid Payload ]    |   [ Valid Payload ]
                                      +----------------------+-----------------------+
                                      |                                              |
                                      v                                              v
                         +--------------------------+                   +--------------------------+
                         | Fail Closed & Auto-Retry |                   | Isolated Sandbox Exec    |
                         | (Return schema error)    |                   | (Docker / gRPC Worker)   |
                         +--------------------------+                   +--------------------------+

Strict Parsing: Validate the LLM output against explicit type-safe schemas (Pydantic in Python or Zod in TypeScript).
Fail-Closed Behavior: If schema validation fails, immediately drop execution and feed the structural error back to the model context for automatic self-healing.
Sandboxed Execution: Run approved tool calls inside ephemeral or restricted execution environments to contain unexpected side effects.

The Implementation

Here is a clean, production-ready pattern using Pydantic to validate tool schemas deterministically before dispatching:

from typing import Any, Literal
from pydantic import BaseModel, Field, ValidationError

class DatabaseQueryTool(BaseModel):
    tool_name: Literal["execute_sql"]
    query_type: Literal["SELECT", "INSERT", "UPDATE"]
    table: str = Field(..., pattern=r"^[a-zA-Z_][a-zA-Z0-9_]*$")
    limit: int = Field(default=10, le=100)

def dispatch_tool_call(raw_payload: dict[str, Any]) -> dict[str, Any]:
    try:
        validated_call = DatabaseQueryTool.model_validate(raw_payload)
        # Safe to execute inside an isolated execution container
        return {"status": "success", "result": f"Executed {validated_call.query_type} on {validated_call.table}"}
    except ValidationError as e:
        # Fail closed: Do not execute, return structured feedback
        return {"status": "error", "reason": e.errors(include_url=False)}

Why This Works

Strict RegEx & Limits: Field validation prevents common injection attacks and accidental resource starvation (e.g., capping limit to 100).
Zero Leaked Exceptions: Uncaught crashes are eliminated at the boundary, returning predictable error payloads to your orchestration layer.
Self-Healing Ready: The e.errors() object can be directly passed back to the LLM as system context: "Your previous output failed validation: {reason}. Fix the arguments."

Production Lessons & Takeaways

Never Trust Raw JSON: Always run LLM outputs through a schema parser before calling internal APIs or running code.
Fail Closed by Default: If a field is missing or malformed, abort execution immediately instead of attempting to guess the model's intent.
Isolate the Execution Layer: Run tools in isolated worker processes (like Docker containers or gRPC sidecars) to limit blast radius in case of unexpected parameters.

Stop Using Naive Hugging Face Pipelines: Scale LLM Serving with vLLM

Srijan Verma — Wed, 19 Aug 2026 15:54:44 +0000

Eliminate GPU memory fragmentation and 10x your inference throughput using continuous batching and PagedAttention.

THE BOTTLENECK IN PRODUCTION

Wrapping a standard Hugging Face pipeline or raw PyTorch model inside a FastAPI wrapper is the fastest way to bring down your production backend.

In naive deployments, every concurrent request allocates contiguous GPU memory for its Key-Value (KV) cache. Because generated sequence lengths are unpredictable, up to 60-80% of your VRAM sits completely wasted due to internal and external memory fragmentation. Worse, static batching forces the server to wait for the slowest generation to complete before processing new incoming requests.

# ❌ THE ANTI-PATTERN: Blocking, fragmented, static execution
from transformers import pipeline

# Loads weights naively into VRAM; blocks worker threads per request
pipe = pipeline("text-generation", model="mistralai/Mistral-7B-Instruct-v0.2", device=0)

def handle_request(prompt: str):
    # Static batching causes high latency tails; unmanaged KV cache risks OOM
    return pipe(prompt, max_new_tokens=256)

Under even modest traffic spikes, this architecture leads to compute underutilization, soaring latency tails, and sudden Out-Of-Memory (OOM) crashes.

THE SYSTEM ARCHITECTURE & FIX

To hit enterprise-grade scale, you need specialized inference engines like vLLM or NVIDIA Triton.

The core unlock is PagedAttention. Similar to how an OS manages virtual memory with paging, PagedAttention stores KV caches in non-contiguous physical memory blocks. This virtually eliminates memory waste, dropping fragmentation overhead to under 4%.

Pair this with Continuous (Iteration-Level) Batching. Instead of waiting for an entire batch to finish generating, the engine dynamically injects new requests at every token generation step.

[ Incoming Client Requests ]
            │
            ▼
┌──────────────────────────────────────────────┐
│           vLLM Engine Architecture           │
│                                              │
│  ┌──────────────────┐  ┌──────────────────┐  │
│  │ Continuous       │  │ PagedAttention   │  │
│  │ Dynamic Batcher  │  │ KV Cache Manager │  │
│  └────────┬─────────┘  └────────┬─────────┘  │
└───────────┼─────────────────────┼────────────┘
            │                     │
            ▼                     ▼
┌──────────────────────────────────────────────┐
│            GPU Compute Execution             │
│       (FlashAttention Kernels / CUDA)        │
└──────────────────────────────────────────────┘
            │
            ▼
[ Low-Latency Streamed Responses ]

Requests leave the queue immediately upon completion, and newly arrived prompts are batched into the very next forward pass without stalling existing generation cycles.

THE IMPLEMENTATION

Here is a clean, production-ready pattern to initialize high-throughput LLM serving using vLLM's Python interface:

from vllm import LLM, SamplingParams

# Configure deterministic token generation constraints
sampling_params = SamplingParams(
    temperature=0.2,
    top_p=0.95,
    max_tokens=256
)

# Initialize engine with bounded GPU allocation and tensor parallelism
llm = LLM(
    model="mistralai/Mistral-7B-Instruct-v0.2",
    tensor_parallel_size=1,        # Scale across GPUs if needed
    gpu_memory_utilization=0.90,   # Reserve 90% VRAM for weights + KV cache
    max_model_len=4096             # Strict context boundary
)

def generate_text(prompts: list[str]):
    # vLLM continuously batches this list internally at the C++ kernel level
    outputs = llm.generate(prompts, sampling_params)
    return [output.outputs[0].text for output in outputs]

This setup gives you deterministic memory controls via gpu_memory_utilization, hardware acceleration via custom CUDA kernels, and seamless continuous batching out of the box.

PRODUCTION LESSONS & TAKEAWAYS

Never manage KV memory manually: Offload caching and scheduling to dedicated engines (vLLM, TensorRT-LLM) that operate at the CUDA kernel level.
Adopt OpenAI API Compatibility: Most modern inference engines can run as standalone microservices with an OpenAI-compatible HTTP interface, allowing zero-friction swaps in existing application layers.
Profile Prefill vs. Decode Bottlenecks: Prefill (prompt processing) is compute-bound, while decode (token generation) is memory-bandwidth-bound. Tune your batch sizes and max_num_batched_tokens based on your system's prompt-to-response length ratio.

Stop Wasting Tokens: Building a Sub-50ms Semantic Cache with Redis

Srijan Verma — Wed, 19 Aug 2026 14:08:14 +0000

Slash LLM latency by 90% and cut API bills by caching query intent instead of raw strings.

The Bottleneck in Production

Direct LLM calls are the slowest, most expensive component in modern backend stacks. A typical round-trip to an LLM provider takes between 2,000ms and 4,000ms and costs fractions of a cent per request. At scale, this destroys both user experience and infrastructure budgets.

Traditional caching fails completely for conversational AI and search interfaces. If you hash the raw query string into a standard key-value store, your cache hit rate rarely exceeds 15–20%.

# The Naive Approach: Breaks on minor variations
cache_key = hashlib.sha256(user_query.strip().lower().encode()).hexdigest()
cached_response = redis_client.get(cache_key) # Fails on synonyms or typos

A user asking "How do I reset my password?" and another asking "Forgot my credentials, need a reset link" represent identical user intent. Yet, exact-match string hashing treats them as two distinct misses, triggering two redundant, expensive model evaluations.

The System Architecture & Fix

The solution is Semantic Caching: caching queries based on vector embeddings rather than raw text strings.

Instead of an exact key lookup, we convert incoming queries into dense vector representations and run an approximate nearest neighbor (ANN) search in Redis using cosine distance. If the distance to a cached query is within our confidence threshold (e.g., similarity $\ge$ 0.90), we serve the cached answer immediately.

[ User Query ]
      │
      ▼
[ Embedding Model ] (e.g., bge-small-en-v1.5, ~10ms)
      │
      ▼
[ Redis Vector Search ] ── (Cosine Sim ≥ 0.90?)
      │                               │
    [ YES ]                         [ NO ]
      │                               │
      ▼                               ▼
[ Return Cached ]             [ Call LLM API ] (~3000ms)
    (~50ms)                           │
                                      ▼
                              [ Save Vector + Text ]

This cuts downstream latency from ~3,000ms to under 50ms for cache hits while protecting your upstream rate limits.

The Implementation

Below is a production-ready pattern using a fast local embedding model (BAAI/bge-small-en-v1.5) and Redis Vector Search.

import numpy as np
import redis
from redis.commands.search.query import Query
from sentence_transformers import SentenceTransformer

encoder = SentenceTransformer("BAAI/bge-small-en-v1.5")
r = redis.Redis(host="localhost", port=6379, decode_responses=False)

def get_semantic_answer(query: str, threshold: float = 0.90) -> str:
    query_vector = encoder.encode(query).astype(np.float32).tobytes()

    # Search for nearest vector within index
    q = Query("(*)=>[KNN 1 @vector $vec AS score]").return_fields("response", "score").dialect(2)
    results = r.ft("idx:cache").search(q, query_params={"vec": query_vector})

    if results.docs and (1 - float(results.docs[0].score)) >= threshold:
        return results.docs[0].response.decode("utf-8")

    # Cache Miss: Fetch from LLM and store vector
    llm_response = call_llm(query)  # Replace with actual LLM provider call
    doc_id = f"cache:{hash(query)}"
    r.hset(doc_id, mapping={"vector": query_vector, "response": llm_response})

    return llm_response

Why This Works in Production:

Low Overhead: Running a lightweight embedding model locally takes ~10ms on CPU, keeping total cache-hit round trips under 50ms.
Fail-Open Design: If vector search fails or encounters network degradation, the pipeline gracefully falls back to the direct LLM call without crashing the request cycle.

Production Lessons & Takeaways

Tune Your Similarity Threshold Conservatively: Start with a strict cosine similarity threshold between 0.88 and 0.92. Anything lower than 0.85 introduces false positives where slightly different questions get wrong answers.
Co-locate Embeddings with Workers: Run your embedding model directly on the application runtime container or within the same VPC subnet as Redis. Avoid using external embedding APIs for the cache lookup, as network latency defeats the purpose of the cache.
Enforce TTLs on Dynamic Knowledge: Set explicit Time-To-Live (TTL) policies on cached vector keys. LLM answers that depend on changing backend data must expire automatically to avoid serving stale hallucinations.