DEV Community: Salmon Joy

Multi-hop retrieval

Salmon Joy — Mon, 26 Oct 2026 04:54:32 +0000

Multi‑hop Retrieval for Enterprise Knowledge‑Driven Q&A

📋 Concise Diagnosis

Symptom	Typical Cause	Immediate Remedy
Answer stops after a single paragraph	Query spans multiple knowledge islands; single‑vector lookup returns only the first island	Decompose into sub‑questions and chain retrieval hops
Stale regulatory numbers appear in the response	Context window contains cached documents, but the live value is governed by an external API	Add a “live‑value” hop that calls the authoritative service
Latency spikes when depth > 3	Each hop launches a separate retrieval round; network and model inference accumulate	Profile per‑hop latency, enforce a cumulative budget, and consider early‑stop criteria
Business rule is violated despite correct evidence	Rule logic lives in deterministic code, but the LLM is asked to “reason” about it	Encode the rule as executable Python (or a policy engine) and feed its result as a system prompt

⏩ Decision Flow

flowchart TD
    A["Receive user question"]
    B["Parse & Detect Multi‑hop Need"]
    C["Generate First Sub‑query"]
    D["Retrieve Evidence #1"]
    E["Run Deterministic Rule #1"]
    F["Iterate to Next Sub‑query"]
    G["Assemble Final Answer"]
    H["Return to User"]
    A --> B
    B --> C
    C --> D
    D --> E
    E --> F
    F --> D
    F --> G
    G --> H

The diagram shows a single loop for readability; real deployments may iterate up to a configured depth limit.

📐 Architecture & Responsibility Matrix

Component	Core Responsibility	Owner (Typical)
Front‑end API	Accepts raw user query, returns streamed answer	Product / Platform
Question Analyzer	Detects whether the query requires multi‑hop processing (e.g., multiple entities, conditional clauses)	NLP Engineering
Sub‑query Generator	Emits a focused retrieval prompt for each hop (e.g., “What is the Basel III definition of capital requirement?”)	Prompt Engineering
Retriever	Executes a vector or keyword search against the enterprise corpus; returns ranked passages	Retrieval Team
Evidence Chainer	Persists retrieved passages, deduplicates, orders them, and makes them available to subsequent hops	Knowledge Engineering
Rule Engine	Executes deterministic business rules or policy checks (e.g., “capital requirement must be ≥ 8 %”)	Compliance / Engineering
Live‑Value Adapter	Calls external authoritative APIs for up‑to‑date numeric values (e.g., latest stress‑test factor)	Integration Team
Response Composer	Merges chained evidence, rule outcomes, and live values into a coherent natural‑language answer; enforces stop‑criteria	LLM Ops
Monitoring & Guardrails	Tracks per‑hop latency, token usage, and compliance violations; triggers fallback if limits exceeded	SRE / Observability

🧑‍💻 Teaching‑Only Python Helper (Standard Library Only)

# teaching-only: helper that decides whether to continue hopping
import time
from typing import List, Tuple

MAX_HOPS = 5           # illustrative ceiling; adjust per business policy
MAX_LATENCY_SEC = 12   # illustrative cumulative budget; adjust per SLA

def should_continue(
    hop_counter: int,
    elapsed_sec: float,
    evidence: List[Tuple[str, float]],
) -> bool:
    """
    Decide if another retrieval hop is allowed.

    Parameters
    ----------
    hop_counter: int
        Number of hops already executed.
    elapsed_sec: float
        Total wall‑clock time spent on previous hops.
    evidence: List[Tuple[str, float]]
        List of (doc_id, relevance_score) collected so far.

    Returns
    -------
    bool
        True → launch next hop; False → stop and compose answer.
    """
    # Hard stop on depth
    assert hop_counter >= 0, "hop_counter cannot be negative"
    if hop_counter >= MAX_HOPS:
        return False

    # Hard stop on latency
    assert elapsed_sec >= 0.0, "elapsed time cannot be negative"
    if elapsed_sec > MAX_LATENCY_SEC:
        return False

    # Soft stop: diminishing returns (example heuristic)
    if len(evidence) >= 2:
        # compare last two relevance scores
        _, prev_score = evidence[-2]
        _, last_score = evidence[-1]
        # stop if improvement is negligible
        if last_score - prev_score < 0.02:
            return False

    return True

📚 Illustrative Architectural Decision Records

ADR 1 – Choice of Retrieval Backend

Context

Enterprise knowledge bases are stored in heterogeneous formats (PDF, Confluence, relational tables). The team evaluated pure dense‑vector search (FAISS) versus a hybrid dense + BM25 approach (Elastic) for the first hop.

Decision

Adopt a hybrid backend that first filters with BM25 on metadata (document type, confidentiality label) and then re‑ranks with dense embeddings.

Consequences

Improves recall for sparse regulatory terms that dense models under‑represent.
Adds modest per‑hop latency due to the two‑stage pipeline; this overhead is acceptable because early hops dominate the overall latency budget.

Acceptance Criteria (illustrative)

The hybrid approach must demonstrably increase recall on the internal compliance benchmark.
Per‑hop latency should remain within the organization’s acceptable latency budget as measured on a production‑size index.

ADR 2 – Handling Live Numerical Values

Context

Regulatory answers often need the most recent numeric factor (e.g., “latest stress‑test multiplier”). Storing such numbers in the vector store quickly becomes stale.

Decision

Introduce a dedicated Live‑Value Adapter that detects sub‑queries matching the pattern latest * and triggers an HTTP call to the official regulator API. The retrieved value replaces any cached occurrence before answer composition.

Consequences

Guarantees freshness for regulated numbers, satisfying audit requirements.
Adds external‑service latency; a circuit‑breaker will abort the live call after a short timeout and fall back to the most recent cached value with a “⚠️ may be outdated” flag.

Acceptance Criteria (illustrative)

The live‑value path must return values that match the regulator API in the overwhelming majority of cases.
The fallback flag should appear only when the live call fails, and such failures should be rare enough to meet compliance expectations.

✅ Evaluation & Maintenance Checklist

✅ Item	How to Verify	Frequency
Depth‑Latency Trade‑off – cumulative latency respects SLA	Automated load test that exercises the max‑depth path and records end‑to‑end time	CI on every code merge
Evidence Freshness – no stale regulatory doc beyond an acceptable age	Compare timestamps of retrieved passages against source metadata	Weekly audit
Rule Engine Correctness – deterministic outputs for known inputs	Unit tests with exhaustive edge cases (e.g., capital = 7 % should be rejected)	Nightly
Live‑Value Success Rate – high success ratio for API calls	Monitoring of HTTP 200 counters vs. total live‑value attempts	Real‑time alert on sustained degradation
Retrieval Recall – meets internal benchmark expectations	Run the benchmark suite after any index rebuild or embedding model change	Per index refresh
Security / Permissions – each hop respects document ACLs	Integration test that simulates a user lacking access to a confidential doc	Monthly
Model Hallucination Guardrail – answer always cites at least one retrieved passage	Regex check that every answer contains a citation marker (e.g., `[DocID]`)	CI
Documentation Sync – ADRs, architecture diagram, and config files are version‑controlled	Repo scan for drift between `architecture.md` and actual code constants	Every release

📖 References

Microsoft RAG concepts – Azure Foundry documentation on Retrieval‑Augmented Generation

https://learn.microsoft.com/en-us/azure/foundry/concepts/retrieval-augmented-generation
LoRA: Low‑Rank Adaptation of Large Language Models – original paper introducing parameter‑efficient fine‑tuning

https://arxiv.org/abs/2106.09685

Query rewriting and HyDE

Salmon Joy — Sun, 25 Oct 2026 04:54:31 +0000

Query Rewriting & HyDE in Retrieval‑Augmented Generation

Diagnosis Table

Symptom	Likely Root Cause	Immediate Mitigation
Low relevance but correct intent	Query too terse or uses domain jargon unknown to the retriever	Run an Intent Analyzer and consider a safe rewrite (add missing qualifiers).
High relevance, unintended facts	HyDE synthetic document mismatches real sources	Verify HyDE output against a deterministic Live‑Value Resolver before composition.
Compliance flags on answer	Original query contains regulated entities (e.g., ticker symbols) that were altered	Preserve the exact token sequence; skip rewrite and apply rule‑based guardrails.
Sudden latency spikes	Both rewrite and HyDE pipelines fire for every request	Add a Decision Helper that selects a single strategy per query.
Repeated hallucinations on numeric topics	Retrieval fell back to stale cache instead of live API	Route numeric sub‑queries to an authoritative tool/API.

Decision Flow

flowchart TD
    A["User query"] --> B["Is query domain‑critical (e.g., regulator‑required tokens)?"]
    B -->|yes| C["Keep original, enforce guardrails"]
    B -->|no| D["Is intent confidence high?"]
    D -->|yes| E["Use original query for retrieval"]
    D -->|no| F["Can a safe rewrite be generated?"]
    F -->|yes| G["Rewrite + retrieve"]
    F -->|no| H["Generate HyDE document, then retrieve"]
    C --> I["Retrieve + compose answer"]
    E --> I
    G --> I
    H --> I

Architecture & Responsibilities

Component	Primary Responsibility
Input Handler	Normalise Unicode, enforce length limits, flag regulated identifiers.
Intent Analyzer	Produce a confidence indicator using a frozen‑base LLM; does not modify model weights.
Rewriter	Generates a longer, context‑rich query while preserving identified regulated tokens.
HyDE Generator	Constructs a short answer, embeds it as a synthetic document, and returns the embedding to the Retriever.
Retriever	Performs dense/sparse search over the knowledge base; falls back to vector store if lexical match fails.
Live‑Value Resolver	Calls authoritative APIs (e.g., market data) for any numeric entity detected after retrieval.
Answer Composer	Merges retrieved passages, HyDE‑generated text, and live values; applies a template that inserts guardrail checks.
Guardrails (Business Rules Engine)	Executes deterministic code that validates compliance, rate limits, and forbidden content before the final answer is emitted.
Monitoring & Logging	Captures latency, cost, intent‑drift metrics, and flags hallucinations for human review.

Decision Helper (Python – standard library only)

# teaching‑only
import re
from typing import Literal

def decide_strategy(query: str) -> Literal["original", "rewrite", "hyde"]:
    """
    Choose a retrieval‑augmentation strategy.

    Rules
    -----
    1. Non‑empty, printable ASCII required.
    2. If a regulated token (ticker, SSN, etc.) is present → keep original.
    3. If the Intent Analyzer would likely indicate high confidence → original.
    4. Otherwise, attempt a safe rewrite; if rewriting fails → HyDE.

    The function never contacts external services; it only analyses the string.
    """
    # 1. basic validation
    assert isinstance(query, str) and query.strip(), "query must be non‑empty"
    assert all(32 <= ord(ch) <= 126 for ch in query), "query must be printable ASCII"

    # 2. detect regulated tokens (very simple regex for illustration)
    regulated = re.search(r"\b([A-Z]{1,4}\d{0,2})\b", query)  # e.g., ticker XYZ or XYZ12
    if regulated:
        return "original"

    # 3. mock intent confidence (placeholder – real system uses a frozen LLM)
    #    Here we treat the presence of a question mark as a proxy for lower confidence.
    high_confidence = query.count("?") == 0
    if high_confidence:
        return "original"

    # 4. safe rewrite feasibility (e.g., length > 5 words → likely expandable)
    if len(query.split()) > 5:
        return "rewrite"

    # 5. fallback
    return "hyde"

ADR 1 – When to Apply Query Rewriting

Context

User queries often omit qualifiers that are essential for downstream compliance (e.g., “stock ticker XYZ price” must stay literal, while “latest mortgage rates” benefits from added temporal scope).

Decision

Apply a safe rewrite only when:

No regulated tokens are detected.
The Intent Analyzer suggests low confidence and the query is long enough to allow meaningful expansion.

Consequences

Positive

Broader lexical match improves recall.
Enables downstream retrieval of related contextual information without manual prompt engineering.

Negative

Potential intent drift if the rewrite injects assumptions (e.g., adding “30‑day” when the user asked for “today”).

Acceptance Criteria

Demonstrated improvement in relevance metrics on a held‑out test set compared with the original‑only pipeline.
Human reviewers confirm that rewritten queries retain the original intent in the majority of cases.
Measured latency remains within the operational budget for the deployment environment.

ADR 2 – When to Use HyDE

Context

Certain domains (legal, medical, fintech) contain concise factual answers that are rarely present verbatim in the corpus, but a short synthetic answer can act as a strong anchor for retrieval.

Decision

Trigger HyDE when:

The Intent Analyzer indicates low confidence and the query is a brief factual request (single sentence, limited token count).
No regulated tokens are present.
Retrieval on the original query returns an insufficient number of high‑scoring candidates.

Consequences

Positive

Synthetic document provides a dense embedding that bridges phrasing gaps.
Early relevance boost without additional external calls.

Negative

Generated text may hallucinate unsupported facts, increasing reliance on the Guardrails engine.
Additional CPU cycles for the HyDE LLM, though limited to a single forward pass on a frozen model.

Acceptance Criteria

Relevance metrics (e.g., nDCG) show a measurable lift for HyDE‑enabled queries relative to a baseline without HyDE.
Post‑guardrail factual error rate, as verified by human auditors, remains acceptably low.
Overall system latency for the HyDE path does not exceed the limits appropriate for the target user experience.

Evaluation & Maintenance Checklist

[ ] Unit Tests for decide_strategy covering regulated token detection, confidence branches, rewrite eligibility, and HyDE fallback.
[ ] Integration Tests that mock the Intent Analyzer and verify the correct pipeline branch executes.
[ ] Regression Suite: run relevance benchmarks on a stable validation set after any LoRA update.
[ ] Compliance Audits: automatically validate that every answer containing regulated identifiers passed through the Guardrails engine.
[ ] Live‑Value Validation: for all numeric sub‑queries, confirm that values originate from authorized APIs (e.g., Bloomberg, Federal Reserve).
[ ] Latency Monitoring: record per‑stage latency; raise alerts if any stage exceeds the service‑level objective defined for the deployment.
[ ] Cost Tracking: aggregate token usage per strategy and ensure it stays within allocated budgets.
[ ] Model Drift Review: periodically compare intent confidence distributions with the latest data; adjust thresholds only after a documented evaluation.

References

Microsoft RAG concepts – https://learn.microsoft.com/en-us/azure/foundry/concepts/retrieval-augmented-generation
LoRA: Low‑Rank Adaptation of Large Language Models – https://arxiv.org/abs/2106.09685

Two-stage retrieval and reranking

Salmon Joy — Thu, 22 Oct 2026 04:54:29 +0000

Two‑Stage Retrieval & Reranking: A Technical Guide

Recall first, precision second. This guide explains how to separate candidate generation from relevance refinement, how to tune the funnel, and which metrics keep the system both fast and trustworthy.

📋 Concise Diagnosis Table

Symptom	Likely Root Cause	First‑Stage Check	Second‑Stage Check
Low answer relevance (many off‑topic sentences)	Top‑N too small → insufficient recall	Verify vector index recall on a held‑out query set	Confirm reranker model is loaded and scoring correctly
High latency (≈ 2 s)	Top‑N too large or expensive reranker	Measure index latency per query (ms)	Profile reranker inference time per candidate (ms)
Prompt token budget exceeded (≈ 1 500)	Top‑k too high → too many tokens passed to generator	Count tokens returned from index (sum of passage lengths)	Count tokens after reranker cut‑off
Missed regulatory clause (false‑negative)	Recall gap – missing relevant doc in index	Run BM25 / dense recall test against known clause IDs	Ensure reranker does not drop the clause by rank

🧭 Decision Flow

flowchart TD
    A["Start: User Query"]
    B["Dense / BM25 Search → Top‑N"]
    C["Apply Filters (date, jurisdiction)"]
    D["Reranker LLM scores candidates"]
    E["Select Top‑k"]
    F["Assemble Prompt for Generator"]
    G["Generate Answer"]
    H["Return to User"]
    A --> B
    B --> C
    C --> D
    D --> E
    E --> F
    F --> G
    G --> H

🏗️ Architecture & Responsibility Matrix

Component	Primary Responsibility	Deterministic Business Rule	Typical Latency Target*
Index Service (e.g., Pinecone, Elasticsearch)	Retrieve up to Top‑N document chunks via vector similarity or BM25	If `query.metadata.source == "authoritative"` then restrict to indices flagged `authoritative == true`	≤ 150 ms per query
Filter Layer	Enforce compliance filters (date, jurisdiction, confidentiality)	If `doc.date < policy.effective_date` then drop the document	≤ 30 ms
Reranker (LLM inference with a lightweight prompt)	Score the Top‑N candidates and produce a ranking	If `score < rerank.threshold` then discard before Top‑k selection	≤ 300 ms for N = 200
Prompt Assembler	Concatenate selected Top‑k snippets, add system prompt, enforce token budget	If `total_tokens > max_prompt_tokens` then truncate lowest‑ranked snippets	≤ 20 ms
Generator (chat‑style LLM)	Produce the final answer using the assembled prompt	If `generated_text` contains any `disallowed_terms` then flag for human review	≤ 800 ms
Monitoring & Logging	Capture recall, precision, latency, token counts per request	If `latency > SLA.latency` or `precision < SLA.precision` then raise an alert	Real‑time

*Latency targets are illustrative; actual Service‑Level Agreements must be defined per deployment.

🐍 Teaching‑Only Python Helper

# teaching-only: helper to decide top‑N / top‑k based on policy constants
import os
from typing import Tuple

# ---- Configuration (replace with env‑vars or config files in production) ----
MAX_PROMPT_TOKENS = 1024          # hard limit for downstream generator
MAX_LATENCY_MS    = 1500          # end‑to‑end budget
DEFAULT_TOP_N    = 200
DEFAULT_TOP_K    = 5

def decide_funnel_sizes(
    estimated_index_tokens: int,
    estimated_rerank_latency_ms: int,
) -> Tuple[int, int]:
    """
    Return (top_n, top_k) that respects MAX_PROMPT_TOKENS and MAX_LATENCY_MS.
    Assertions guarantee deterministic behaviour for unit testing.
    """
    # 1️⃣ Start with defaults
    top_n = DEFAULT_TOP_N
    top_k = DEFAULT_TOP_K

    # 2️⃣ Adjust top_n if the token budget would be exceeded even before reranking
    if estimated_index_tokens > MAX_PROMPT_TOKENS:
        # Reduce the pool size proportionally, but never below 1
        top_n = max(1, top_n * MAX_PROMPT_TOKENS // estimated_index_tokens)

    # 3️⃣ Adjust top_k to keep the final prompt under the token limit
    #    Assume an average snippet length of 200 tokens (empirical, not arbitrary)
    avg_snippet_len = 200
    max_k = MAX_PROMPT_TOKENS // avg_snippet_len
    top_k = min(top_k, max_k)

    # 4️⃣ Ensure the overall latency budget is not violated
    #    +500 ms is a placeholder for generator inference; replace with real estimate in production
    total_latency = estimated_rerank_latency_ms + 500
    assert total_latency <= MAX_LATENCY_MS, (
        f"Latency {total_latency} ms exceeds budget {MAX_LATENCY_MS} ms"
    )

    # 5️⃣ Return a deterministic tuple
    return top_n, top_k

# Example (for notebooks or documentation):
# top_n, top_k = decide_funnel_sizes(estimated_index_tokens=1200,
#                                   estimated_rerank_latency_ms=250)
# print(top_n, top_k)

📜 Architectural Decision Records (ADRs)

ADR 1 – Choosing a Dense Vector Index

Context	The corpus contains roughly twelve thousand compliance paragraphs. Queries are frequently phrased in natural language, requiring semantic matching.
Decision	Adopt a dense vector index (e.g., Pinecone) as the primary first‑stage retriever, supplemented by BM25 as a fallback for exact phrase matches.
Alternatives	1. Pure BM25 – deterministic but limited semantic recall. 2. Hybrid (BM25 + dense) – higher engineering cost.
Consequences	Improves recall for paraphrased regulatory language, but introduces vector‑search latency that must be monitored.
Acceptance Criteria	Target recall on a held‑out set of 200 clause queries ≥ 0.85 and 95th‑percentile latency ≤ 150 ms on the target hardware.

ADR 2 – Reranker Scoring Threshold

Context	The reranker LLM returns a raw relevance score in the range [0, 1]. Some low‑scoring candidates may still contain useful context.
Decision	Introduce a configurable rerank.threshold (default 0.60). Candidates scoring below the threshold are discarded before Top‑k selection.
Alternatives	1. Fixed Top‑k without a score filter – risk of low‑quality snippets. 2. Dynamic, per‑query thresholds – adds complexity and non‑determinism.
Consequences	Guarantees a minimum relevance baseline, reduces token waste, and keeps reranker latency predictable.
Acceptance Criteria	Target Precision@k (k = 5) on a validation set ≥ 0.78 and token count reduction relative to raw Top‑N ≥ 30 %. The threshold may be tuned but must stay ≥ 0.55 to satisfy the precision target.

✅ Evaluation & Maintenance Checklist

Recall Test: Run a quarterly BM25 + dense recall benchmark on a labelled query set (≥ 200 samples). Record the percentage of known relevant passages retrieved in Top‑N.
Precision Test: After reranking, compute Precision@k (k = 5, 10) on the same set. Verify against the targets in ADR 2.
Latency SLA: Measure end‑to‑end latency (index → rerank → generator) under a representative load (≥ 100 QPS). Alert if the 95th‑percentile exceeds MAX_LATENCY_MS.
Token Budget Audits: Log prompt_tokens_before and prompt_tokens_after. Ensure average reduction meets the > 20 % target.
Compliance Filter Validation: Unit‑test all deterministic filter rules (date, jurisdiction, confidentiality) with a deterministic test harness; any failure must block deployment.
Model Drift Monitoring: Track hallucination rate using a synthetic fact‑checking prompt. If the rate rises above 5 %, trigger a prompt‑engineering review (do not automatically fine‑tune).
Documentation Sync: Verify that any change to MAX_PROMPT_TOKENS, MAX_LATENCY_MS, or rerank.threshold is reflected in this guide and the corresponding ADRs.
Backup & Rollback: Archive the exact vector index snapshot and reranker prompt version used for each production release. Enable deterministic rollback if a new release violates any acceptance criterion.

📚 References

Microsoft RAG concepts – Azure Foundry documentation: https://learn.microsoft.com/en-us/azure/foundry/concepts/retrieval-augmented-generation
LoRA: Low‑Rank Adaptation of Large Language Models – research paper: https://arxiv.org/abs/2106.09685

(The LoRA paper is cited for understanding lightweight adaptation; it is not the original RAG publication.)

Editorial Note

A symptom listed in the diagnosis table is a hypothesis, not proof of the root cause.
RAG does not guarantee factual accuracy, instant compliance, or complete coverage; deterministic business rules and live‑value APIs must be used where up‑to‑date facts are required.
LoRA updates freeze the base model weights and add low‑rank matrices; they do not inherently preserve all original behaviours—regression testing remains essential.
When a request requires a live value (e.g., a current market price), retrieve it from an authoritative API rather than a cached document.

End of guide.

Hybrid search: lexical + vector

Salmon Joy — Wed, 21 Oct 2026 04:54:39 +0000

Hybrid Search in Enterprise Retrieval: Lexical + Vector Fusion

Quick Diagnosis

Symptom	Likely Root Cause	Lexical (BM25) Impact	Dense Vector Impact	Immediate Action
Exact phrase returns no hit	Query terms missing in index or stop‑word removal	Low recall – exact match fails	Irrelevant – semantic similarity may surface loosely related docs	Verify analyzer pipeline, re‑index with phrase‑aware tokenizer
Relevant documents missing despite synonyms	Vocabulary mismatch, no semantic layer	Zero score (no token overlap)	May retrieve but low rank if weight is too low	Increase vector weight or add synonym expansion to index
Legal compliance audit returns extra unrelated hits	Over‑aggressive vector weighting	Still deterministic for exact hits	Introduces noise	Reduce vector contribution, add stricter metadata filter
Entity identifiers (e.g., “ACC‑12345”) are not boosted	No entity‑aware boosting rule	Matches exact token, but lower than surrounding text	Vector similarity may ignore numeric patterns	Add entity boost factor in scoring function
Search latency spikes after hybrid enablement	Unoptimized fusion step or large candidate set	Fast BM25 lookup	Cosine similarity on many vectors is expensive	Limit candidate pool, cache embeddings, tune top‑k per branch

Decision Flow (Mermaid)

flowchart TD
    A["Start"]
    B["Is query short & exact?"]
    C["Is entity pattern detected?"]
    D["Set lexical weight"]
    E["Set vector weight"]
    F["Execute hybrid search"]
    A --> B
    B -->|Yes| D
    B -->|No| C
    C -->|Yes| D
    C -->|No| E
    D --> F
    E --> F

The diagram guides weight selection before the hybrid request is issued.

Architecture & Responsibility Matrix

Component	Primary Responsibility	Owner (role)	Key Interfaces
Lexical Index (BM25)	Tokenization, posting list creation, deterministic scoring	Search Index Engineer	Query API, Document Ingestion Pipeline
Vector Store	Embedding generation (offline), approximate nearest‑neighbor (ANN) lookup, cosine similarity	ML Retrieval Engineer	Embedding Service, ANN Index
Fusion Engine	Combine BM25 score & vector similarity, apply entity boost, enforce metadata filters	Retrieval Architect	Scoring Service, Filter Service
Metadata Filter Service	Apply user‑defined constraints (date, department, product) before ranking	Data Governance Lead	Filter API, Access Control Layer
Observability Layer	Log latency, score distribution, drift in recall/precision	SRE / Analytics	Metrics Exporter, Alerting Rules
Evaluation Harness	Periodic QA against labeled queries, regression detection	QA Engineer	Test Suite, Benchmark Dashboard

Core Distinctions for Hybrid Retrieval

Current Knowledge – Stored in the lexical index and vector store; it reflects the state of the document corpus at ingestion time.
Live Values – Anything that must be obtained from an authoritative system (e.g., pricing API, compliance service) should be fetched via a dedicated tool or API at query time, not from the retrieved text.
Enforceable Business Rules – Deterministic logic such as entity boosts, mandatory metadata filters, or compliance flags must live in code that guarantees execution for every request.
Tuning for Stable Behavior – Adjustments to weight ratios, boost factors, or retrieval thresholds should be performed after systematic evaluation with labeled data and regression testing; they are not a substitute for the business rules above.

Python Decision Helper (standard library only)

# teaching-only: illustrative helper to choose lexical/vector weights
import re
from typing import Dict

def decide_weights(query: str, has_entity: bool, metadata_filter: bool) -> Dict[str, float]:
    """
    Returns a weight map for hybrid search.

    - lexical weight + vector weight must sum to 1.0.
    - entity presence pushes lexical weight up.
    - metadata filtering favours lexical precision.

    Raises:
        AssertionError: if inputs are malformed or resulting weights are out of bounds.
    """
    # Basic sanity checks
    assert isinstance(query, str) and query.strip(), "query must be a non‑empty string"
    assert isinstance(has_entity, bool), "has_entity must be bool"
    assert isinstance(metadata_filter, bool), "metadata_filter must be bool"

    # Heuristic: short exact queries → strong lexical emphasis
    short_exact = len(query.split()) <= 3 and not re.search(r"\b(what|how|why)\b", query, re.I)

    # Base weights
    lexical = 0.6
    vector = 0.4

    if short_exact:
        lexical = 0.8
        vector = 0.2
    if has_entity:
        lexical = min(lexical + 0.1, 0.9)   # boost exact token match
        vector = 1.0 - lexical
    if metadata_filter:
        lexical = min(lexical + 0.05, 0.95) # filters favour deterministic hits
        vector = 1.0 - lexical

    # Final validation
    assert 0.0 <= lexical <= 1.0, "lexical weight out of range"
    assert 0.0 <= vector <= 1.0, "vector weight out of range"
    assert abs((lexical + vector) - 1.0) < 1e-6, "weights must sum to 1.0"

    return {"lexical": lexical, "vector": vector}

Architecture Decision Record 1 – Weighting Strategy Selection

Context – Enterprises need a reproducible way to decide the lexical‑vs‑vector ratio per query type.

Decision – Adopt a rule‑based heuristic (see decide_weights) that adjusts weights based on query length, presence of identifiable entities, and whether a metadata filter is active.

Uncertainty – Heuristics may not capture domain‑specific jargon, multilingual nuances, or evolving business terminology; the mapping could be sub‑optimal for newly introduced vocabularies.

Acceptance Criteria

The hybrid system must demonstrate at least the same mean average precision as the prior baseline on a held‑out validation set of enterprise queries.
95th‑percentile latency must remain within the service‑level agreement defined for the production environment.
Adjusting the vector weight upward should not cause an unacceptable drop in recall as judged by the evaluation harness.

Architecture Decision Record 2 – Entity Boosting Policy

Context – Many enterprise queries contain identifiers (account numbers, SKU codes, regulation IDs). Lexical match already surfaces them, but they can be drowned by higher‑scoring semantic hits.

Decision – Introduce a deterministic boost factor applied to the BM25 score whenever a regular‑expression pattern matches a known identifier schema. The boost is applied before fusion with the vector score.

Uncertainty – Over‑boosting may hide newer, semantically relevant content that does not repeat the identifier verbatim.

Acceptance Criteria

Precision on identifier‑centric queries (top‑3 results) must meet or exceed the established baseline.
The boost should not noticeably degrade recall for queries that do not contain identifiers.
The additional computation for the boost must not materially increase average query latency.

Evaluation & Maintenance Checklist

[ ] Document Refresh – Periodically re‑ingest updated documents and recompute embeddings using the same frozen base model.
[ ] Regression Test Suite – After any index or weight change, run a representative set of labeled queries; reject changes if mean average precision degrades beyond an acceptable margin.
[ ] Metadata Schema Review – Verify that filter fields (e.g., effective_date, department) remain aligned with governance policies; update the filter service if the schema drifts.
[ ] Observability Review – Confirm that latency histograms and score distribution dashboards are healthy; set alerts for deviations from expected operational ranges.
[ ] Entity Pattern Audit – Regularly validate that regular expressions cover newly introduced identifier formats; add missing patterns as needed.
[ ] Weight Tuning Cadence – Conduct controlled experiments after major business cycles to recalibrate lexical/vector ratios based on fresh evaluation results.

References

Microsoft RAG concepts – https://learn.microsoft.com/en-us/azure/foundry/concepts/retrieval-augmented-generation
LoRA: Low‑Rank Adaptation of Large Language Models – https://arxiv.org/abs/2106.09685

Managed model API vs self-hosted inference

Salmon Joy — Thu, 15 Oct 2026 04:54:22 +0000

Managed Model API vs Self‑Hosted Inference

Quick‑Diagnosis Table

Decision Factor	Managed API (Provider‑Hosted)	Self‑Hosted Inference (On‑Prem / Cloud‑Owned)
Operational ownership	Provider runs GPU fleet, scaling logic, OS patches, and health checks.	Your ops team provisions hardware, runs schedulers, applies patches, and monitors.
Data control & compliance	Data leaves your environment; you must rely on provider‑defined egress contracts.	Data stays in your network; you can enforce any regulatory boundary you control.
Hardware utilization	Shared capacity; batch sizes are limited by provider’s quota and pricing tier.	You can size batches precisely, over‑subscribe GPUs, or reserve dedicated instances.
Customization	Prompt‑only or limited parameter tweaks (temperature, top‑p).	Ability to swap model checkpoints, add adapters, change tokenizers, or run LoRA updates.
Reliability model	SLA defined by provider (e.g., high uptime); fail‑over is automatic.	You must design redundancy (multi‑zone, active‑passive, health‑checks) yourself.
Skill requirements	Minimal; integration via HTTP/SDK.	Requires Kubernetes, GPU driver management, model‑server knowledge (e.g., NVIDIA Triton).
Latency profile	Depends on provider’s multi‑tenant queue; generally higher than a tightly tuned local stack.	Can be tuned to lower latency by colocating near downstream services and controlling batch timing.
Total cost of ownership	Predictable usage‑based pricing plus any data‑egress fees; no capital expense.	Capital expense for GPUs plus variable OPEX (power, staffing, maintenance).

Note: The table is a diagnostic aid. Real projects should measure latency, throughput, and cost for their specific workloads before committing.

Decision Flow (Mermaid)

flowchart TD
    A["Start"] --> B["Require strict data residency?"]
    B -->|Yes| C["Need very low latency?"]
    B -->|No| D["Team has GPU‑ops expertise?"]
    C -->|Yes| E["Choose Self‑Hosted"]
    C -->|No| F["Choose Managed API"]
    D -->|Yes| E
    D -->|No| F
    E --> G["Implement self‑hosted stack"]
    F --> G

The diagram condenses the most common trade‑offs: data residency, latency, and operational skill set.

Architecture Responsibility Matrix

Layer	Managed API Owner	Self‑Hosted Owner
Model hosting & hardware	Provider (fleet mgmt, GPU health)	Internal SRE / Platform team (provisioning, driver updates)
Inference service (REST / gRPC)	Provider (API gateway, auth)	Your model server (Triton, vLLM, or custom)
Scaling & autoscaling	Provider (elastic load)	Your orchestration (K8s HPA, custom scaler)
Monitoring & alerting	Provider (service health page)	Your observability stack (Prometheus, Grafana, alerts)
Security & access control	Provider IAM, API keys	Zero‑trust network policy, mTLS, internal RBAC
Data governance	Provider‑defined contracts	In‑house DLP, audit logs, compliance checks
Versioning & updates	Provider releases new model versions	Your CI/CD pipeline for model artifact promotion
Cost tracking	Provider billing dashboard	Internal chargeback tags (GPU‑hours, storage)

Decision Helper (Python, std‑lib only)

# teaching‑only example
import os
from dataclasses import dataclass

@dataclass(frozen=True)
class InferenceContext:
    """All inputs that influence the operating model decision."""
    data_residency_required: bool
    latency_target_ms: int | None   # None means “no explicit target”
    gpu_ops_expertise: bool
    expected_monthly_token_volume: int

def decide_inference_mode(ctx: InferenceContext) -> str:
    """
    Return the recommended operating mode: ``"managed"`` or ``"self_hosted"``.
    The logic mirrors the Mermaid diagram but adds defensive checks.
    """
    # Basic sanity checks – raise early if required fields are missing.
    assert isinstance(ctx.data_residency_required, bool), "Residency flag must be bool"
    assert ctx.expected_monthly_token_volume >= 0, "Token volume cannot be negative"
    if ctx.latency_target_ms is not None:
        assert ctx.latency_target_ms >= 0, "Latency target must be non‑negative"

    # Decision tree
    if ctx.data_residency_required:
        # Residency forces control of the stack; latency becomes secondary.
        return "self_hosted"
    if ctx.latency_target_ms is not None and ctx.latency_target_ms < 100:
        # Very low latency is generally only achievable with local inference.
        return "self_hosted"
    if ctx.gpu_ops_expertise:
        return "self_hosted"
    # Default to managed when none of the constraints push toward ownership.
    return "managed"

# Example (illustrative, not executed):
# ctx = InferenceContext(True, None, False, 5_000_000)
# mode = decide_inference_mode(ctx)
# print(mode)  # → "self_hosted"

The helper is deliberately simple: it does not embed any provider‑specific pricing or SLA numbers, keeping the logic portable across clouds.

Architectural Decision Records (ADRs)

ADR 001 – Data Residency vs Latency

Context

A regulated financial services team processes personally identifiable information (PII) and must keep all raw data behind its firewall. The same service also requires very low response time for real‑time fraud scoring.

Decision

Deploy a self‑hosted inference stack inside the private data center, using a GPU‑accelerated model server with a LoRA‑based domain adapter.

Alternatives Considered

Managed API with an egress‑only contract.
Hybrid approach: Managed API for non‑PII requests, self‑hosted for PII.

Rationale

Residency eliminates any cross‑border data movement. Latency can be measured directly on the internal network, giving confidence that the target is met. The hybrid option introduced routing complexity without clear cost benefit.

Acceptance Criteria

End‑to‑end latency meets the internal target defined by the product team.
No raw data leaves the internal network, as verified by network flow logs that show no outbound traffic to provider IP ranges.

ADR 002 – Skill Availability vs Speed‑to‑Market

Context

An edtech startup needs to launch an adaptive‑quiz feature within a tight schedule. The team has strong product designers but only one DevOps engineer with limited GPU experience.

Decision

Adopt a managed inference API (e.g., Azure OpenAI or AWS Bedrock) for the initial launch.

Alternatives Considered

Build a self‑hosted stack using Kubernetes + Triton.
Contract an external GPU‑ops consultancy to set up the infrastructure.

Rationale

The managed service removes the need for GPU driver maintenance, scaling logic, and model‑server configuration, fitting the short timeline. The consultancy option added procurement overhead and did not guarantee long‑term knowledge transfer.

Acceptance Criteria

Feature released within the planned timeline.
Integration tests pass with an acceptable failure rate as defined by the product team.
Post‑launch cost analysis shows usage within the budgeted forecast.

Evaluation & Maintenance Checklist

Latency verification – Run a calibrated load test; confirm that observed latency satisfies the target defined for the project.
Throughput capacity – Ensure the system sustains the expected token volume without excessive queuing.
Data‑flow audit – Verify that no raw input leaves the trusted zone (self‑hosted) or that egress contracts are in place (managed).
Patch & driver updates – For self‑hosted, schedule regular GPU driver refresh; for managed, monitor provider release notes.
Model version governance – Tag each deployed checkpoint; keep a changelog linking business releases to model updates.
Observability – Instrument request latency, error rates, and GPU utilization; set alerts on threshold breaches.
Cost reconciliation – Reconcile monthly token usage against provider invoices; for self‑hosted, reconcile GPU‑hour billing and power consumption.
Compliance re‑validation – At each major regulatory review, confirm that the chosen operating model still satisfies jurisdictional requirements.

References

Microsoft – Retrieval‑augmented generation concepts. https://learn.microsoft.com/en-us/azure/foundry/concepts/retrieval-augmented-generation
LoRA: Low‑Rank Adaptation of Large Language Models. https://arxiv.org/abs/2106.09685

Enterprise AI gateway

Salmon Joy — Wed, 14 Oct 2026 04:54:39 +0000

Enterprise AI Gateway – Technical Guide

A control‑plane that authenticates, routes, and governs every LLM request across an organization.

📋 Diagnosis Table

Symptom	Likely Root Cause	When a Gateway Becomes Worthwhile
Sprawl of API keys	Each team provisions its own provider credentials	Multiple independent consumer services and/or a mix of cloud providers
Unpredictable monthly LLM spend	No centralized quota enforcement	Organization needs centralized budgeting and cost‑center accountability
Inconsistent policy enforcement (e.g., PII leakage)	Policy logic duplicated in many services	Regulatory or legal policies must be auditable enterprise‑wide
Latent or failed model calls during peak loads	No fail‑over or routing logic	Strong service‑level expectations for availability and response time
Difficulty onboarding a new foundation model	Hard‑coded provider URLs in code	A central “allow‑list” of approved models is required

🗺️ Decision Diagram

flowchart TD
    A["Multiple LLM Consumers"] --> B["Compliance / Cost Constraints"]
    B --> C["Need Central Auth & Quotas"]
    C --> D["Enterprise AI Gateway"]
    D --> E["Implement Shared Control Plane"]
    A --> F["Few Consumers"]
    F --> G["Ad‑hoc Direct Provider Calls"]

If the left‑hand path is taken, a gateway is strongly recommended. The right‑hand path may remain viable for very small deployments.

🏗️ Architecture & Responsibility Matrix

Layer	Responsibility	Typical Implementation Details
Authentication	Verify caller identity (API‑key, JWT, mTLS).	Use corporate IdP (Azure AD, Okta). Map identities → tenants → quota buckets.
Quota & Throttling	Enforce per‑team, per‑product, and per‑model limits.	Token‑bucket or leaky‑bucket counters stored in a fast cache (Redis, Azure Cache).
Routing	Select provider & model based on policy, latency, cost, or feature set.	Rule engine (e.g., JSON‑based policy) that evaluates `preferredProvider`, `fallbackProvider`, `region`.
Policy Enforcement	Apply legal/ethical guardrails (PII redaction, hate‑speech blocking, regulated exposure caps).	Plug‑in filters that run before request reaches the model; can short‑circuit with a “blocked” response.
Logging & Auditing	Capture request metadata, decisions, and outcomes for compliance dashboards.	Structured logs (JSON) streamed to SIEM/Log Analytics; include request ID, caller, model, policy decisions.
Provider Abstraction	Hide vendor‑specific SDKs behind a uniform interface.	Internal `ModelClient` with methods `invoke(prompt, params)` that internally selects Azure OpenAI, AWS Bedrock, etc.
Failover	Detect degraded latency or error rates and transparently switch to a secondary provider.	Health‑check ping; routing rule `if latency deteriorates → use fallback`.
Model Allow‑list	Central catalog of approved foundation models (name, version, vendor, licensing).	Single source of truth (e.g., Azure Table, DynamoDB) referenced during routing.

🐍 Python Decision Helper (Standard Library Only)

Teaching‑only – The snippet demonstrates how a service could decide whether to invoke the gateway or fall back to a direct provider, based solely on a configuration file.

# enterprise_ai_gateway_helper.py
"""
Decision helper for routing LLM requests.

- Reads a JSON config that contains:
    * `allow_list`: approved model identifiers
    * `quota_limits`: per‑tenant request caps
    * `fallback_provider`: secondary cloud vendor
- Returns a dict with routing metadata; raises AssertionError on mis‑configuration.
"""

import json
import os
from pathlib import Path
from typing import Dict, Any

CONFIG_PATH = Path(os.getenv("GATEWAY_CONFIG", "./gateway_config.json"))

def load_config() -> Dict[str, Any]:
    """Load and validate the gateway configuration."""
    with CONFIG_PATH.open() as fp:
        cfg = json.load(fp)

    # Basic sanity checks – deterministic assertions, not runtime heuristics.
    assert isinstance(cfg.get("allow_list"), list), "allow_list must be a list"
    assert cfg.get("allow_list"), "allow_list cannot be empty"
    assert isinstance(cfg.get("quota_limits", {}), dict), "quota_limits must be a dict"
    assert isinstance(cfg.get("fallback_provider"), str), "fallback_provider required"
    return cfg

def decide_routing(tenant_id: str, requested_model: str, current_usage: int) -> Dict[str, str]:
    """
    Determine the provider and model to use.

    Parameters
    ----------
    tenant_id: str – Identifier of the calling team/product.
    requested_model: str – Model name requested by the caller.
    current_usage: int – Number of requests already made this billing period.

    Returns
    -------
    dict with keys `provider`, `model`, `via_gateway`.
    """
    cfg = load_config()

    # Enforce allow‑list
    assert requested_model in cfg["allow_list"], f"Model {requested_model} not approved"

    # Enforce quota (simple per‑tenant cap)
    quota = cfg["quota_limits"].get(tenant_id, 0)
    assert quota > 0, f"No quota defined for tenant {tenant_id}"
    assert current_usage < quota, f"Tenant {tenant_id} exceeded quota ({current_usage}/{quota})"

    # Routing decision – primary provider derived from model name convention
    primary_provider = "azure" if requested_model.lower().startswith("gpt") else "aws"

    # In a real deployment the health check would be dynamic; here we assume primary is healthy.
    provider = primary_provider

    # Example of deterministic fail‑over logic (placeholder)
    # if health_check(provider) == "unhealthy":
    #     provider = cfg["fallback_provider"]

    return {
        "provider": provider,
        "model": requested_model,
        "via_gateway": "true",  # All decisions flow through the gateway
    }

# Example usage (for illustration only)
if __name__ == "__main__":
    # Simulate a request from team 'risk‑engine' asking for 'gpt-4o-mini' with 12 prior calls.
    routing = decide_routing("risk-engine", "gpt-4o-mini", 12)
    print(routing)

📄 Illustrative Architecture Decision Records

ADR 1 – Adopt Central Model Allow‑List

Context

Multiple engineering squads independently select foundation models. Procurement data shows several distinct model versions in production, leading to fragmented licensing and security reviews.

Decision

Introduce a central allow‑list stored in a managed key‑value service. The gateway validates every request against this list before routing.

Consequences

Positive	Negative
Single source of truth for compliance and licensing.	Requires coordination for model deprecation or addition.
Faster onboarding of new services (they inherit the list).	Slight increase in request latency due to the lookup.

Acceptance Criteria

All production LLM calls should be validated by the gateway before reaching a provider.
Requests for models not present in the allow‑list are rejected with an appropriate error response.
Auditors can retrieve a read‑only JSON snapshot of the allow‑list via a dedicated API.

Uncertainties

Frequency of model version churn may pressure the update process.
Impact on latency for high‑throughput micro‑services needs monitoring.

ADR 2 – Implement Provider‑Failover Policy

Context

During a load test, Azure OpenAI latency spiked markedly, breaching internal availability expectations. No automatic reroute existed, leading to request timeouts.

Decision

Encode a fail‑over rule in the gateway: if the primary provider’s recent latency exceeds an acceptable threshold, automatically route new requests to the secondary provider defined in fallback_provider.

Consequences

Positive	Negative
Improves resilience without code changes in downstream services.	Requires reliable health‑monitoring infrastructure.
Enables cost optimisation by preferring lower‑priced providers when performance is acceptable.	Possible model‑output variance between providers; downstream must tolerate it.

Acceptance Criteria

Health monitoring must emit latency metrics to a central store at regular intervals.
Fail‑over activation must be logged with timestamp, measured latency, and chosen provider.
After latency improves and remains within acceptable bounds for a sustained period, routing should revert to the primary provider.

Uncertainties

How downstream services will handle subtle differences in tokenisation or response format between providers.
Potential licensing implications of dynamically switching providers mid‑request.

✅ Evaluation & Maintenance Checklist

[ ] Authentication Review – Verify token validation against the corporate IdP on a quarterly cadence.
[ ] Quota Accuracy – Reconcile gateway counters with cloud billing reports each month.
[ ] Policy Coverage – Run an automated test suite that injects known PII, regulated terms, and profanity; ensure the gateway blocks or redacts as expected.
[ ] Latency & Fail‑over Tests – Simulate provider latency degradation; confirm automatic reroute and proper logging.
[ ] Allow‑list Audits – Generate a weekly immutable snapshot; compare against procurement records for consistency.
[ ] Log Retention & Access Control – Ensure logs are encrypted at rest and retained for the mandated compliance period.
[ ] Disaster Recovery Drill – Power‑cycle the gateway service and verify that downstream services handle the temporary unavailability according to the defined fallback strategy.
[ ] Cost Monitoring – Correlate gateway‑reported usage with cloud billing dashboards; investigate any significant variance.

📚 References

Microsoft RAG concepts – Retrieval‑augmented generation overview for Azure AI services. https://learn.microsoft.com/en-us/azure/foundry/concepts/retrieval-augmented-generation
LoRA paper – Low‑Rank Adaptation of Large Language Models. https://arxiv.org/abs/2106.09685

This guide is intended for architects and platform engineers evaluating the trade‑offs of an Enterprise AI Gateway. Adoption decisions should be driven by measurable compliance, cost, and resilience signals rather than assumed performance gains.

Reasoning depth vs latency vs cost

Salmon Joy — Tue, 13 Oct 2026 04:54:38 +0000

Reasoning Depth vs Latency vs Cost – An Escalation‑Based Design Guide

Balancing model capability, response time, and expense by routing work through a defined escalation path.

1️⃣ Concise Diagnosis Table

Symptom 🩺	Likely Primary Cause	Immediate Check
Sub‑second latency but answers are vague or incomplete	Model selected lacks sufficient reasoning depth	Verify the request’s reasoning steps (chain‑of‑thought, policy nuance)
Latency spikes above service‑level objective while cost stays low	Small model forced to iterate many retries or falls back to heuristics	Review token usage and number of inference calls
Cost per 1 K tokens is high but latency is acceptable	Large model used for a trivially‑easy task	Confirm the routing rules that map “easy” workload to a lightweight model
Inconsistent policy compliance across similar requests	Missing or stale retrieval context; deterministic business logic executed elsewhere	Ensure retrieval component pulls the latest authoritative source; check rule engine logs
Frequent escalation to the biggest model	Reasoning budget definition too permissive or risk‑threshold too low	Re‑evaluate the thresholds that trigger escalation

2️⃣ Decision Flow (Mermaid)

flowchart TD
    A["Start"] --> B["Is request easy?"]
    B -->|Yes| C["Route to Small Model"]
    B -->|No| D["Is latency budget sufficient?"]
    D -->|Yes| E["Route to Medium Model"]
    D -->|No| F["Escalate to Large Model"]
    C --> G["Execute"]
    E --> G
    F --> G

Key definitions

Easy request – single‑lookup, no multi‑step reasoning, deterministic rule check.
Latency budget – maximum wall‑clock time allocated for the request tier (set in the Service‑Level Objective table).
Escalation – handing the request to a higher‑capacity model when either depth or risk exceeds the current tier’s limits.

3️⃣ Architecture & Responsibility Matrix

Component	Core Responsibility	Interaction Points
Request Router	Classifies incoming payloads (easy vs. complex) and selects the initial model tier based on the decision flow.	Consumes request metadata, consults Reasoning Budget Engine.
Reasoning Budget Engine	Computes a budget ‑ tokens × cost × max latency – for the current tier and exposes a “budget‑exceeded” flag.	Reads pricing table, latency SLA, and request complexity estimate.
Cost / Latency Monitor	Records actual token consumption, wall‑clock latency, and monetary cost per request; feeds back to the router for adaptive throttling.	Emits metrics to observability stack; triggers alerts when SLA drift occurs.
Model Pool	Holds at least three model variants: • Small – fast, cheap, limited depth (e.g., gpt‑3.5‑turbo). • Medium – balanced latency and reasoning (e.g., gpt‑4‑base). • Large – highest depth, higher cost (e.g., gpt‑4‑32k).	Exposes a uniform inference API; each model reports its token‑rate and per‑token price.
Retrieval Layer	Supplies external, up‑to‑date documents (RAG) only for contexts that require factual grounding.	Uses Microsoft RAG concepts; never substitutes deterministic rule evaluation.
Tooling / APIs	Executes authoritative calls (e.g., live currency rates, regulatory APIs) when a request requires live values.	Invoked by the model via function‑call style prompts; results are never cached for live‑value requests.
Deterministic Rule Engine	Enforces business‑critical policies that must be 100 % correct (e.g., compliance thresholds).	Runs outside the LLM; the LLM may suggest a rule but the engine validates it.
Fine‑tuning / LoRA Service (optional)	Trains low‑rank adapters on stable, high‑quality labeled data when prompting and retrieval are proven insufficient.	Uses LoRA paper guidance; adapters are frozen after evaluation and merged for inference only when regression tests pass.

4️⃣ Python Decision Helper (Teaching‑Only)

# -*- coding: utf-8 -*-
"""
decision_helper.py – map a request to the appropriate model tier.

Teaching‑only: this module illustrates the budget calculation and routing
logic; production systems should substitute the placeholder constants
with configuration values sourced from a centralized policy store.
"""

from datetime import datetime, timezone
from typing import Literal, Dict

# ---- Configuration placeholders (populate from authoritative sources) ----
TOKEN_COST: Dict[Literal["small", "medium", "large"], float] = {
    "small":   0.0,   # $ per token – replace with real provider price
    "medium":  0.0,
    "large":   0.0,
}
MAX_LATENCY_MS: Dict[Literal["small", "medium", "large"], int] = {
    "small":   0,    # milliseconds – replace with SLA value
    "medium":  0,
    "large":   0,
}
MAX_TOKENS: Dict[Literal["small", "medium", "large"], int] = {
    "small":   0,    # context window size – replace with model capability
    "medium":  0,
    "large":   0,
}
# -------------------------------------------------------------------------

def _budget_exceeded(
    tier: Literal["small", "medium", "large"],
    estimated_tokens: int,
    elapsed_ms: int,
) -> bool:
    """
    Return True when either cost, latency, or token‑count exceeds the limits
    defined for the given tier.  All limits are injected via the placeholder
    dictionaries above.
    """
    # Cost check – expressed in dollars for the estimated token count
    cost = estimated_tokens * TOKEN_COST[tier]

    # Latency check – compare observed wall‑clock time with tier SLA
    latency_ok = elapsed_ms <= MAX_LATENCY_MS[tier]

    # Token‑count sanity – avoid exceeding model context window
    token_ok = estimated_tokens <= MAX_TOKENS[tier]

    # All three conditions must hold; otherwise we need to escalate
    return not (latency_ok and token_ok and cost <= 0.0)   # 0.0 is a placeholder ceiling


def decide_model(
    request_id: str,
    estimated_tokens: int,
    is_easy: bool,
    start_time: datetime,
) -> Literal["small", "medium", "large"]:
    """
    Determine the model tier for a request.

    Assertions act as sanity checks for the teaching example.
    """
    assert estimated_tokens > 0, "Token estimate must be positive"
    assert isinstance(is_easy, bool), "is_easy must be boolean"
    assert start_time.tzinfo is not None, "start_time must be timezone‑aware"

    # 1️⃣ Easy requests start on the smallest tier
    tier: Literal["small", "medium", "large"] = "small" if is_easy else "medium"

    # 2️⃣ Measure elapsed time so far (simulated here as zero)
    elapsed_ms = int((datetime.now(timezone.utc) - start_time).total_seconds() * 1000)

    # 3️⃣ Escalate if budget is exceeded
    if _budget_exceeded(tier, estimated_tokens, elapsed_ms):
        tier = "large"

    return tier

5️⃣ Architectural Decision Records (ADRs)

ADR 001 – Use Escalation‑Based Model Selection

Context

Our platform processes a mixture of high‑throughput lookups and low‑throughput policy analyses. Early prototypes used a single large model for simplicity, resulting in unacceptable latency for the bulk of traffic and inflated compute spend.

Decision

Adopt a three‑tier escalation policy (Small → Medium → Large) governed by a Reasoning Budget Engine. The router initially classifies each request as easy or complex based on a lightweight static analysis (token estimate, presence of chain‑of‑thought cues). If the budget is exceeded after the first inference, the system automatically re‑routes to the next tier.

Alternatives Considered

Single‑model approach – rejected due to SLA violations and cost overruns.
Dynamic model scaling (on‑the‑fly parameter adjustment) – deemed too risky for compliance‑critical paths.

Acceptance Criteria

95 % of requests complete within the latency SLA of their assigned tier.
Total compute cost per day reduces by ≥ 30 % compared with the single‑model baseline.
No increase in policy‑violation incidents after introducing escalation.

ADR 002 – Keep Live‑Value Retrieval Outside LLM Inference

Context

A subset of fraud‑detection requests require the latest exchange‑rate. Early designs cached the rate in the prompt to avoid extra calls, leading to occasional compliance errors when the cache was stale.

Decision

Introduce a Tooling / APIs component that the LLM invokes via function‑call style prompts. The component fetches the value from the authoritative service at request time; the result is never cached for live‑value calls.

Alternatives Considered

Pre‑embedding rates in the prompt – rejected because cache invalidation complexity exceeds benefit.
Relying on the LLM’s internal knowledge – rejected; the model’s knowledge cutoff is older than required.

Acceptance Criteria

Every live‑value request triggers exactly one external API call.
Auditable logs show the timestamp of the fetched value matched the request timestamp ± 5 seconds.
No compliance test failures attributable to stale data after rollout.

6️⃣ Evaluation & Maintenance Checklist

[ ] Unit tests for decide_model covering easy/complex, budget‑exceed, and time‑edge cases.
[ ] Integration tests that simulate end‑to‑end routing with mock latency and cost profiles.
[ ] Regression suite for deterministic rule engine output (must remain unchanged after any LoRA update).
[ ] Metric verification – confirm that Cost / Latency Monitor reports per‑tier averages matching SLA tables.
[ ] RAG validation – sample a set of retrieval‑augmented prompts and verify factual grounding against the source documents; do not rely on RAG for policy enforcement.
[ ] LoRA audit – after any adapter training, run full‑coverage functional tests; retain the frozen base model unchanged.
[ ] Policy review cadence – quarterly check that the Reasoning Budget Engine thresholds still align with business risk tolerances.
[ ] Documentation sync – ensure the decision flow diagram and ADRs reflect the current routing logic.

7️⃣ Editorial Accuracy Guidance

A symptom listed in the diagnosis table is a clue, not proof of a root cause. Investigate label quality, missing context, preprocessing pipelines, and intrinsic model capability before attributing errors to pre‑training.
Retrieval‑Augmented Generation (RAG) improves factual grounding but does not guarantee correctness, completeness, or compliance. Deterministic rule checks must remain separate and authoritative.
Low‑Rank Adaptation (LoRA) freezes the base model weights and learns low‑rank updates; this does not guarantee that all prior behaviours are preserved. Regression testing outside the target task is required.
Live‑value APIs must always be invoked at request time; cached or stale values are unacceptable for any compliance‑sensitive operation.
All acceptance criteria in ADRs contain measurable thresholds and acknowledge uncertainty; they are not claims of absolute performance.

8️⃣ References

Microsoft RAG concepts – Retrieval‑Augmented Generation for grounding LLM output. https://learn.microsoft.com/en-us/azure/foundry/concepts/retrieval-augmented-generation
LoRA: Low‑Rank Adaptation of Large Language Models. https://arxiv.org/abs/2106.09685

By defining explicit escalation paths, quantifying a reasoning budget, and separating deterministic logic from probabilistic inference, teams can keep latency low, cost proportional, and capability aligned with the true difficulty of each request.

Prompting vs RAG vs fine-tuning

Salmon Joy — Mon, 12 Oct 2026 12:26:46 +0000

Prompting vs RAG vs Fine-Tuning: Diagnose the Failure First

A wrong answer is an observation, not a root cause. Inspect the input, source evidence, retrieval results, tool responses, and expected output before choosing an intervention. The same symptom can come from missing context, ambiguous labels, a weak base model, or a bug outside the model. The examples and architectural decisions below are hypothetical teaching examples.

Match the intervention to evidence

Observed issue	Evidence to inspect	Candidate intervention
Instructions are unclear or the answer has the wrong style	Prompt wording, examples, instruction conflicts, and task tests	Improve prompting and evaluate the change
An answer needs private or changing documents	Source version, access filters, retrieval coverage, and citation support	Retrieve relevant documents and provide them as context through RAG
The answer needs a current numeric value	Authoritative service, timestamp, units, and response schema	Call a tool or API at request time
A rule must be enforced exactly	Approved rule specification, inputs, boundary cases, and authorization	Implement and test deterministic application code
A stable task pattern remains weak after baseline improvements	Label quality, representative examples, model capability, and held-out results	Evaluate fine-tuning against the improved baseline

Prompting supplies instructions and examples at inference time. RAG retrieves relevant content and adds it to the model input. Fine-tuning changes learned behavior using training examples. They are composable: a tuned model can still use prompts, retrieval, and tools. None removes the need to evaluate the complete application.

Do not infer that a repetitive error came from pre-training merely because it occurs often. Check the labels, missing context, preprocessing, and task definition first. A model that cannot handle the underlying task may require a different base model rather than more prompt text or tuning.

Decision flow

This diagram summarizes candidate interventions after diagnosis. A symptom can lead to more than one candidate. Validate each change with task-level evaluation before adopting it.

flowchart TD
    A["Inspect failure evidence"]
    B["Clarify instructions and examples"]
    C["Retrieve relevant current documents"]
    D["Query authoritative live values"]
    E["Enforce exact rules in code"]
    F["Evaluate tuning for stable patterns"]
    G["Compare results and operating costs"]
    A --> B
    A --> C
    A --> D
    A --> E
    A --> F
    B --> G
    C --> G
    D --> G
    E --> G
    F --> G

Architecture and responsibilities

Component	Responsibility	Evaluation focus
Prompt templates	Version instructions, examples, and context assembly	Task success and sensitivity to phrasing
Document ingestion and retrieval	Maintain source versions, access filters, indexes, and relevant excerpts	Coverage, freshness, ranking, and citation support
Tool execution	Fetch authoritative values and perform authorized actions	Schema, units, timestamps, errors, and authorization
Deterministic application logic	Enforce explicit business rules and output constraints	Boundary cases and reproducible results
Training and model registry	Track datasets, base versions, tuning artifacts, and evaluation evidence	Target-task improvement and regressions elsewhere
Orchestration and monitoring	Assemble context, route requests, and record useful diagnostics	End-to-end quality, cost, latency, and failure recovery

For a hypothetical policy assistant, use retrieval to supply the applicable document version and citations. Use an API for a live rate, and code for an exact threshold calculation. The model may explain the result, but application logic must enforce any required rule. Retrieval does not guarantee a correct answer or regulatory compliance; source coverage, interpretation, and review still matter.

LoRA freezes base weights and learns low-rank updates to selected weight matrices. Frozen base weights do not imply unchanged behavior when the updates are applied. Evaluate target-task quality and regressions on other tasks. Where the model and deployment support merging, LoRA updates can be merged into the base weights; runtime adapter selection is a separate deployment choice. Measure latency and cost for the actual serving configuration.

Python decision helper: teaching only

This helper maps an already diagnosed category to a candidate intervention. It does not diagnose a model, authorize an action, or decide compliance. A real investigation may produce several categories, so the function accepts multiple findings. It uses only the Python standard library.

from enum import Enum


class Finding(Enum):
    UNCLEAR_INSTRUCTIONS = "unclear_instructions"
    DOCUMENT_GAP = "document_gap"
    LIVE_VALUE = "live_value"
    EXACT_RULE = "exact_rule"
    STABLE_TASK_GAP = "stable_task_gap_after_baselines"


INTERVENTIONS = {
    Finding.UNCLEAR_INSTRUCTIONS: "evaluate_prompting",
    Finding.DOCUMENT_GAP: "evaluate_rag",
    Finding.LIVE_VALUE: "query_authoritative_tool",
    Finding.EXACT_RULE: "implement_tested_code",
    Finding.STABLE_TASK_GAP: "evaluate_fine_tuning",
}


def candidate_interventions(findings: list[Finding]) -> list[str]:
    """Return unique candidates in the order the findings were supplied."""
    if not isinstance(findings, list):
        raise TypeError("findings must be a list of Finding members")
    candidates = []
    for finding in findings:
        if not isinstance(finding, Finding):
            raise TypeError("each finding must be a Finding member")
        candidate = INTERVENTIONS[finding]
        if candidate not in candidates:
            candidates.append(candidate)
    return candidates


if __name__ == "__main__":
    assert candidate_interventions([]) == []
    assert candidate_interventions([
        Finding.DOCUMENT_GAP, Finding.EXACT_RULE, Finding.DOCUMENT_GAP
    ]) == ["evaluate_rag", "implement_tested_code"]
    assert candidate_interventions([Finding.STABLE_TASK_GAP]) == [
        "evaluate_fine_tuning"
    ]

The explicit type checks remain active when Python assertions are disabled. The assertions demonstrate expected behavior; they are not input validation. Candidate names intentionally describe the next engineering step rather than promise a successful fix.

Illustrative architectural decision records

ADR 001: Retrieve changing policy documents

Context: A hypothetical assistant must explain policy documents that change independently of model releases.

Decision: Maintain a searchable source collection with versions and effective dates. Refresh it according to the application's freshness requirement, apply access filters, retrieve relevant passages, and retain citation metadata. Evaluate missing-document and conflicting-version cases. Escalate when supporting evidence is insufficient.

Consequences: Document updates can be delivered without retraining the model. Ingestion and retrieval add operational work and latency. A fresh index alone cannot establish that an answer is correct, complete, or compliant. Exact enforceable rules remain in tested code.

ADR 002: Evaluate tuning for stable classification behavior

Context: A hypothetical classifier still makes a recurring error after task definition, labels, prompts, and supplied context have been checked.

Decision: Compare a tuning candidate with the improved baseline using representative, disjoint evaluation data. Track the base model, training data, and tuning configuration. Include regression cases outside the target category and keep a rollback path. Consider LoRA if the chosen model and training stack support it.

Consequences: Tuning may improve stable task behavior, but improvement must be demonstrated. Training data, evaluation, and model lifecycle management add cost. Small trainable updates do not guarantee preserved general performance. Tuning is not a dependable mechanism for refreshing changing facts or enforcing exact rules.

Evaluation and maintenance checklist

Version prompts, model configurations, data sources, and expected outputs so comparisons are reproducible.
Measure end-to-end task success with representative examples; inspect individual failure cases as well as averages.
For RAG, evaluate source coverage, retrieval relevance, document versions, access control, and whether citations support each factual claim.
For tools, validate schemas, units, timestamps, and error states. Define when a cached result is acceptable; do not silently use stale data for a request requiring a live value.
For deterministic rules, test boundary values and invalid inputs against the approved specification.
For tuning, keep training and evaluation data separate and compare against the strongest practical baseline. Check regressions and drift before promotion.
Measure latency and cost per successful task for the actual application. Set thresholds from its requirements rather than invented universal benchmarks.
Keep diagnostic records limited to the information needed for investigation, with appropriate retention and data handling.

Start with the evidence, choose a candidate change, and measure the result. Keep retrieval, tool execution, rule enforcement, and learned task behavior accountable to their own tests.

Primary references

Long context vs RAG

Salmon Joy — Fri, 09 Oct 2026 05:10:36 +0000

Long Context Did Not Kill RAG: The Real Architecture Decision

When I first plugged a LLM into our fintech risk engine, I thought “just dump the whole CSV, it’s only a few thousand rows”. The model froze, the API balked, and our devs started yelling about token limits. Turns out, feeding a 5 MB PDF of credit‑card statements into the prompt is like trying to read War and Peace in one breath – you get the gist but miss the details, and you pay a fortune in tokens.

So I went back to the drawing board and asked: when does feeding the whole document actually work, and when do we need a Retrieval‑Augmented Generation (RAG) layer?

Static, short, self‑contained sources – e.g., a one‑page terms‑of‑service, a quiz question bank, or a syllabus snippet – can safely be shoved into the prompt. Accuracy stays high because the model sees everything at once, there’s virtually no retrieval‑noise, and provenance is simple – the prompt itself is the source. The token cost is predictable and usually cheap.
As soon as the corpus grows beyond a few kilobytes or changes daily, the “long context” trick dies. In my edutech chatbot we tried to feed a whole course catalog (200+ pages). The model started hallucinating module numbers and latency shot up. RAG saved us: we indexed each lesson, pulled only the three most relevant chunks, and the answer became spot‑on. Retrieval reduces noise because you only surface what the similarity score says is relevant, but you also introduce a new failure mode – the retriever might miss the right paragraph.

Let’s break it down

Aspect	Full‑Doc	RAG
Accuracy	100% recall of the text but can drown in irrelevant bits.	Higher precision when the retriever is good; a poor retriever can drop critical facts.
Noise	Noisy by design.	Filters noise but adds risk of “retrieval hallucination”.
Token cost	1 token ≈ 4 characters. A 10 KB doc ≈ 2,500 tokens, eating most of the 4k limit for GPT‑4.	Typically 2‑3 chunks of 200‑300 tokens each → saves >80% of tokens.
Access control	Must embed the whole sensitive file in every API call – not ideal for PCI‑compliant fintech data.	Store docs in a secure vector store, fetch only needed vectors, keep raw file off‑chain.
Freshness	Updating a static prompt means re‑sending the whole new document each time.	Swap a single vector or re‑index a changed chapter in seconds.
Provenance	Can point to line numbers directly.	Provides a document ID and chunk offset, which can be logged for audit trails – actually better for compliance.

Case study 1 – Fintech

We built a KYC assistant that needed the latest AML regulations. Regulations update weekly, so we indexed each PDF. The assistant now answers “What’s the new threshold for cash transactions?” by pulling the exact clause, not by guessing from a stale prompt.

Case study 2 – Edutech

Our tutoring bot used RAG to fetch only the relevant lesson from a 300‑page physics textbook. Students got answers in 0.8 s instead of the 4‑second lag we saw when we tried to push the whole PDF.

Bottom line

Long context didn’t kill RAG; it just reminded us that each tool has a sweet spot.

Use full‑doc when the source is tiny, immutable, and low‑risk.
Flip to RAG when you need scalability, freshness, or tighter control.

What’s the biggest “long‑context disaster” you’ve survived? Drop a story below – I love a good war tale!

If you are someone who loves to know the technical work and architecture design I have shared more details based on my experience on this here: https://github.com/SalmonJoy/My_guide_for_building_AI_systems/blob/main/Long_context_vs_RAG.md

Function calling vs free-form generation

Salmon Joy — Thu, 08 Oct 2026 05:10:53 +0000

LLMs Should Decide What to Ask For, Not Secretly Execute Your Business Logic

Last month I was knee‑deep in a fintech app that let users snap a receipt and instantly see a spending summary. The LLM was supposed to read the image, pull out numbers, and dump them into our ledger. Instead it started “doing its own business logic”: rounding oddly, applying a grandma‑style discount, even flagging a transaction as fraud on a hunch. It felt like handing a chef a pan and watching them secretly add mystery spices.

The issue was clear: I was using free‑form generation as a Swiss‑army knife and letting the model decide how to do the work. The result? Non‑deterministic output, audit nightmares, and a pile of “why did the AI do that?” tickets.

Enter function calling. Think of it as a polite waiter: you tell the model, “If you need a number, call the extract_amount(image) function and I’ll give you a float.” The model can’t magically compute inside its brain; it must ask your API for the exact piece you typed. Your service handles the heavy lifting, has auth checks, logs every request, retries on failure, and returns a deterministic result. This is exactly what OpenAI’s Function Calling and the “Tools” feature do – expose narrow, typed endpoints and keep the business logic on the server side.

A quick analogy: free‑form generation is like giving a kid a 3‑D printer and saying “make me a toy.” They might print a dinosaur, a spaceship, or a banana. Function calling is like handing them a LEGO set with a clear instruction sheet – they can only build what the instructions allow, and you can count each piece as it’s added.

In our fintech case we defined three tools:

extract_amount(image_bytes) → float
record_transaction(user_id, amount, category) → bool
check_fraud(user_id, amount) → bool

Now the LLM says, “I need the amount from this receipt – call extract_amount,” and we get a clean number. If the API hiccups we retry, everything is logged, and auditors can see exactly what was asked and what was returned. No hidden discounts.

In edutech we built a tutoring bot that could answer math questions, but we didn’t want it to hallucinate formulas. We exposed a solve_linear(a, b) → float function. The model now asks, “What’s x in 2x+3=7? Call solve_linear with a=2, b=-4.” The answer is always correct, traceable, and teachers can audit the logs.

Key take‑aways

Separate interpretation (LLM decides what to ask) from execution (your code does the how).
Keep tools narrow and parameters typed – no free‑form JSON blobs.
Wire up auth, audit trails, retries, and error handling on the tool side.
Keep business logic deterministic in your code, not hidden in the model’s imagination.

So the next time you’re tempted to let the LLM write the whole recipe, remember: let it be the clever waiter, not the secret chef. What narrow tool would you love to add to your stack? Drop a comment and let’s swap ideas!

Structured outputs as application contracts

Salmon Joy — Wed, 07 Oct 2026 05:10:32 +0000

Your LLM Output Should Have a Contract, Not a Vibe

So there I was, an AI engineer at a fintech startup, trying to turn a GPT‑4 chat into a credit‑score suggestion engine. The user typed “I just got a promotion, how much should I loan?” and the model spat back a paragraph that looked like poetry: “Congrats! Maybe $5‑10k could work, but keep your debt‑to‑income ratio low…” I tried to yank the numbers out with a regex like /\$(\d+)-(\d+)k/. Spoiler: it broke the moment someone said “$5‑10 k” with a thin space or used “5‑10k” without the dollar sign. One mis‑typed hyphen and my downstream service crashed, mis‑classifying a safe applicant as risky. It felt like trying to catch a greased pig with a colander.

That’s when I discovered the power of structured outputs as application contracts. Instead of hoping the model “gives me what I need”, I told it exactly what shape the response must have – a JSON object that matches a JSON Schema and a typed TypeScript interface. The prompt became:

{
  "loan_range": { "min": int, "max": int },
  "confidence": "high" | "medium" | "low"
}

Now the model can’t accidentally sprinkle a poem in there; it either obeys or says “I’m sorry, I can’t comply”. That refusal is a feature: my code checks if (response.refusal) … and falls back to a safe default.

Why is this better than regex? Regex is a brittle detective that looks for patterns in free text. Anything outside the pattern – extra whitespace, emojis, a different phrasing – sends it into a dead end. A schema validator, on the other hand, parses a proper data structure and instantly tells you which fields are missing or of the wrong type. It’s like checking a passport instead of eyeballing a face.

In my edutech side‑project, we needed a list of quiz questions with “question”, “options”, and “answer”. By versioning the schema ("$schema": "http://json-schema.org/draft-07/schema#", "title": "QuizV2"), we could roll out a new field “explanation” without breaking older clients. Downstream safety checks – e.g., “no option string longer than 200 chars” – are baked into the schema, so the LLM can’t hand us a monster answer that blows up the UI.

A quick case study: after switching to structured outputs and enabling refusals, our fintech loan‑approval pipeline’s error rate dropped from 12% to under 1%. The only time we got a refusal was when the user asked for advice on illegal activity, and we gracefully returned a polite error message.

Bottom line: treat the LLM like a junior dev who follows a contract, not a poet who vibes with your mood. Define a JSON Schema, generate typed interfaces, validate, version, and let refusals be your safety net.

Got a funny regex story or a schema win? Drop a comment – I’d love to hear how you’re taming your LLMs!

Context engineering vs prompt engineering

Salmon Joy — Tue, 06 Oct 2026 13:17:36 +0000

Prompt Engineering Is Too Small a Mental Model, Think Context Architecture

I was knee‑deep in a fintech app that tried to answer user questions about credit‑score changes. My first version was a classic “prompt engineering” hack:

“You are a friendly finance bot, explain the score drop in plain English.”

It worked for the first few queries, then the answers got weird – sometimes the bot repeated the same disclaimer, other times it started spilling the policy docs verbatim. I was losing token budget fast and the model kept “remembering” the old disclaimer even after I removed it. I realized I was treating the whole prompt as a monolith, when in reality I was juggling five different pieces of context that should have their own lanes.

Enter context architecture

Think of an LLM conversation like a pizza kitchen.

System instructions – the chef’s recipe book; they stay on the board all night.
User state – the current order ticket (what the customer just asked, plus any previous choices).
Retrieved knowledge – the pantry shelf you pull ingredients from (a database lookup of the user’s credit history).
Tool results – the side‑dish the sous‑chef prepares (e.g., a risk‑score API call).
Policy – the health‑inspection sticker you slap on the box to make sure you never serve raw chicken.

The magic happens in precedence. The model first looks at system instructions, then layers user state, then fetched knowledge, then tool results, and finally policy overrides anything that would break compliance. If you drop a policy clause at the bottom of a huge prompt, it might never be seen because the model runs out of tokens before it gets there. That’s why you always put the “stable‑prefix” (system + policy) at the front – it’s cached, cheap, and guarantees the model never forgets the rules.

Token budget = pizza dough budget

You only have so many tokens per request; if you waste them on stale context (like a disclaimer you already sent), you run out of room for the juicy answer.

Contamination risk is when old user state leaks into a new conversation – imagine the kitchen forgetting to clear the ticket and adding yesterday’s pepperoni to today’s vegan pizza. To avoid that, we clear the user state each session or use a fresh context window.

Stable‑prefix caching

(OpenAI Prompt Caching) lets you store the system + policy block once and reuse it across millions of calls. In my edutech project, we cached a 120‑token prefix that described the “tutor bot” persona and compliance rules. The cache cut latency by 30 % and saved us $2 K a month on token spend.

Case study: Fintech fraud‑alert bot

Piece	Content
System	“You are a compliance‑aware fraud analyst.”
Policy	“Never reveal raw transaction IDs.”
User	“My card was declined twice.”
Retrieved	“Last 5 transactions: $23, $0, $0.”
Tool	“Risk engine returns confidence 0.92.”

The final prompt stayed under 350 tokens, the answer was spot‑on, and no policy leakage occurred.

So the next time you’re tempted to dump a giant prompt into the model, pause and ask yourself:

What belongs in system instructions?
What’s the user state?
What do I need to fetch?
What tool results am I adding?
What policy must sit at the very front?

Build that context architecture, and you’ll spend less time chasing bugs and more time building cool features.

What weird context mishap have you run into? Drop a comment and let’s swap stories!

If you are someone who loves to know the technical work and architecture design I have shared more details based on my experience on this here:

https://github.com/SalmonJoy/My_guide_for_building_AI_systems/blob/main/Context_engineering_vs_prompt_engineering.md

DEV Community: Salmon Joy

Multi-hop retrieval

Multi‑hop Retrieval for Enterprise Knowledge‑Driven Q&A

📋 Concise Diagnosis

⏩ Decision Flow

📐 Architecture & Responsibility Matrix

🧑‍💻 Teaching‑Only Python Helper (Standard Library Only)

📚 Illustrative Architectural Decision Records

ADR 1 – Choice of Retrieval Backend

ADR 2 – Handling Live Numerical Values

✅ Evaluation & Maintenance Checklist

📖 References

Query rewriting and HyDE

Query Rewriting & HyDE in Retrieval‑Augmented Generation

Diagnosis Table

Decision Flow

Architecture & Responsibilities

Decision Helper (Python – standard library only)

ADR 1 – When to Apply Query Rewriting

ADR 2 – When to Use HyDE

Evaluation & Maintenance Checklist

References

Two-stage retrieval and reranking

Two‑Stage Retrieval & Reranking: A Technical Guide

📋 Concise Diagnosis Table

🧭 Decision Flow

🏗️ Architecture & Responsibility Matrix

🐍 Teaching‑Only Python Helper

📜 Architectural Decision Records (ADRs)

ADR 1 – Choosing a Dense Vector Index

ADR 2 – Reranker Scoring Threshold

✅ Evaluation & Maintenance Checklist

📚 References

Editorial Note

Hybrid search: lexical + vector

Hybrid Search in Enterprise Retrieval: Lexical + Vector Fusion

Quick Diagnosis

Decision Flow (Mermaid)

Architecture & Responsibility Matrix

Core Distinctions for Hybrid Retrieval

Python Decision Helper (standard library only)

Architecture Decision Record 1 – Weighting Strategy Selection

Architecture Decision Record 2 – Entity Boosting Policy

Evaluation & Maintenance Checklist

References

Managed model API vs self-hosted inference

Managed Model API vs Self‑Hosted Inference

Quick‑Diagnosis Table

Decision Flow (Mermaid)

Architecture Responsibility Matrix

Decision Helper (Python, std‑lib only)

Architectural Decision Records (ADRs)

ADR 001 – Data Residency vs Latency

ADR 002 – Skill Availability vs Speed‑to‑Market

Evaluation & Maintenance Checklist

References

Enterprise AI gateway

Enterprise AI Gateway – Technical Guide

📋 Diagnosis Table

🗺️ Decision Diagram

🏗️ Architecture & Responsibility Matrix

🐍 Python Decision Helper (Standard Library Only)

📄 Illustrative Architecture Decision Records

ADR 1 – Adopt Central Model Allow‑List

ADR 2 – Implement Provider‑Failover Policy

✅ Evaluation & Maintenance Checklist

📚 References

Reasoning depth vs latency vs cost

Reasoning Depth vs Latency vs Cost – An Escalation‑Based Design Guide

1️⃣ Concise Diagnosis Table

2️⃣ Decision Flow (Mermaid)

3️⃣ Architecture & Responsibility Matrix

4️⃣ Python Decision Helper (Teaching‑Only)

5️⃣ Architectural Decision Records (ADRs)

ADR 001 – Use Escalation‑Based Model Selection

ADR 002 – Keep Live‑Value Retrieval Outside LLM Inference

6️⃣ Evaluation & Maintenance Checklist

7️⃣ Editorial Accuracy Guidance

8️⃣ References

Prompting vs RAG vs fine-tuning

ADR 1 – Choice of Retrieval Backend

ADR 2 – Handling Live Numerical Values

ADR 1 – When to Apply Query Rewriting

ADR 2 – When to Use HyDE

ADR 1 – Choosing a Dense Vector Index

ADR 2 – Reranker Scoring Threshold

Hybrid Search in Enterprise Retrieval: Lexical + Vector Fusion

Managed Model API vs Self‑Hosted Inference

ADR 001 – Data Residency vs Latency

ADR 002 – Skill Availability vs Speed‑to‑Market

Reasoning Depth vs Latency vs Cost – An Escalation‑Based Design Guide

ADR 001 – Use Escalation‑Based Model Selection

ADR 002 – Keep Live‑Value Retrieval Outside LLM Inference