DEV Community: Lei Ye

Why Your Production RAG System Slowly Gets Worse

Lei Ye — Mon, 29 Jun 2026 04:17:36 +0000

Background

Production RAG systems rarely fail through a single catastrophic event. More commonly, reliability erodes through a sequence of operational changes: documentation evolves, retrieval behavior shifts, prompts are revised, dependencies change, and evaluation datasets become stale.

Traditional engineering practices classify failures by system components—retrievers, prompts, vector databases, or language models. While useful for implementation, this perspective provides limited guidance for operating production AI systems over time.

This article proposes a reliability framework based on three complementary dimensions:

Failure Dynamics — how reliability changes over time
Reliability Control Surface — where engineers can observe and intervene
Detectability — how easily the failure is discovered before users are affected

To illustrate the framework, a controlled experiment simulates seven weeks of gradual documentation evolution in a production-style RAG system. The experiment demonstrates one representative failure class—Gradual Knowledge Drift—and shows why this class of failure frequently escapes traditional operational monitoring.

1. Introduction — AI Systems Rarely Fail the Way Traditional Software Does

Modern software systems fail in ways that operations teams understand well. A bad deployment increases error rates. A database outage causes requests to fail. A networking issue adds latency. Infrastructure becomes unavailable. These failures are disruptive, but they are also highly visible. Dashboards turn red, alerts fire, and engineers know where to start investigating.

Retrieval-Augmented Generation (RAG) systems introduce a different class of failure. Usually , a production RAG application can appear perfectly healthy from an operational perspective. Requests complete successfully, APIs return HTTP 200 responses, latency remains within service-level objectives, and every component in the architecture is online. Traditional monitoring tools report a healthy system. Yet users begin to lose confidence in the answers.

Fundamentally, we are trying to solve the AI reliability problem instead of the traditional software reliability problem.

" width="800" height="533">

From the graph, the key differences is that traditional software failures are around discrete events and gives immediate feedback; while RAG systems degrades gradually and usually invisible to infrastructure-level monitoring. Fundamentally, traditional software’s reliability is typically judged by correctness and availability: either the service works or it doesn't. RAG systems add another dimension—knowledge quality. A system can achieve excellent uptime while steadily becoming less reliable.

This reframes reliability from a problem of system correctness to a problem of sustained knowledge quality.

2. Why Existing Classifications Are Insufficient

What do we know about RAG system failures. Perhaps newly published documentation isn't being retrieved. Maybe document metadata has drifted, reducing retrieval accuracy. An embedding model has changed, but only part of the corpus has been re-indexed…

Current discussions usually classify failures by components, some of the examples are :

Component	Typical failures
Embedding model	Poor semantic representations, embedding drift after model changes, domain mismatch, multilingual mismatch
Vector database	Low recall, indexing errors, stale or missing vectors, incorrect filtering, ANN search inaccuracies
Chunking	Chunks too large/small, broken context boundaries, duplicated information, loss of semantic coherence
Retriever	Irrelevant documents retrieved, low recall, poor ranking, metadata filtering mistakes
Reranker	Relevant documents demoted, irrelevant documents promoted, unstable ranking
Prompt	Hallucinations, ignored context, prompt injection, poor instruction following, format inconsistencies
LLM / Generator	Hallucination, incorrect synthesis, unsupported claims, reasoning errors, overconfidence
Knowledge base	Outdated documents, incomplete corpus, inconsistent information, stale data
Ingestion pipeline	Failed indexing, partial ingestion, parsing/OCR errors, metadata extraction failures

Figure 2 - AI Failure Examples

These do explain where failures originate. However, they hardly explain:

how failures evolve
when engineers discover them
which operational strategy is appropriate

Production RAG system operations require a reliability model, not only an architecture model.

3. A Reliability Framework for Production AI Systems

Imagine an engineer receiving the following incident report:

"The RAG system is hallucinating more than usual."

Although the statement describes a symptom, it immediately raises several unanswered questions.

Has the system failed suddenly after a deployment, or has answer quality been declining for weeks? Is the root cause likely to be in the knowledge base, the retrieval pipeline, or the generation stage? Should engineers inspect operational dashboards, rerun evaluation suites, or begin a deeper investigation?

The difficulty is not a lack of observability—it is a lack of structure for reasoning about production AI failures.

From examining recurring production incidents, I found that most failures can be described along three complementary dimensions:

Failure Dynamics describe how reliability changes over time.
Reliability Control Surfaces identify where corrective action is most effective.
Detectability characterizes how easily the failure is discovered before affecting users.

Rather than treating every incident as unique, these dimensions provide a common language for understanding, classifying, and responding to production AI failures.

Dimension 1 — Failure Dynamics

When a RAG incident occurs, the first question engineers should ask is not what failed, but how reliability changed over time.

Traditional software systems are typically designed around discrete failures. A deployment introduces a regression, a dependency fails, or a resource becomes exhausted. Reliability changes are usually tied to identifiable events, allowing engineers to reason about incidents as immediate failures.

Production RAG systems behave differently. Reliability often changes continuously rather than discretely. Documentation evolves, retrieval behavior shifts, prompts are revised, and evaluation datasets become stale. Individually, these changes appear harmless; collectively, they reshape the behavior of the system. As a result, understanding a production AI incident begins with a different question:

How did reliability evolve over time?

This leads to the first dimension of the framework: Failure Dynamics.

Immediate
Immediate failures appear immediately after a discrete system change or unexpected input. They are typically associated with deployments, prompt revisions, tool misconfiguration, or invalid context injection. Engineers usually observe an immediate drop in correctness or task completion

Gradual
Gradual failures emerge through a sequence of individually harmless changes. Documentation evolves, retrieval behavior shifts, evaluation datasets become stale, or models are upgraded incrementally. No single change is sufficient to trigger an incident, but their cumulative effect steadily erodes reliability.

Threshold
Threshold failures remain latent until accumulated changes push the system beyond a critical operating boundary. Reliability appears stable until a tipping point is reached, after which performance degrades abruptly.

Oscillating
Oscillating failures exhibit inconsistent reliability under similar operating conditions. Performance alternates between successful and unsuccessful outcomes because the underlying system behavior depends on input distribution, retrieval ordering, model stochasticity, or changing operational conditions.

Cascading
Cascading failures originate from a local defect that propagates through downstream workflow stages. A retrieval error may influence planning, which affects tool selection, memory updates, and ultimately produces a significantly larger end-user failure than the original defect alone.

Dimension 2 — Reliability Control Surface

Once the failure dynamics have been identified, the next engineering question is:

Where should engineers intervene?

Failure Dynamics describe how reliability changes. Reliability Control Surfaces describe where reliability can be observed, influenced, and improved.

In traditional software systems, the answer is often localized. Engineers scale infrastructure to address resource contention, upgrade dependencies to resolve compatibility issues, or adjust service-level trade-offs between latency, availability, and consistency. The intervention point is usually well-defined because the system itself is deterministic.

Production RAG systems are different. A single user-visible failure may emerge from interactions across multiple stages of the pipeline. Corrective actions therefore require engineers to identify the control surface where reliability can be most effectively improved.

We define five primary Reliability Control Surfaces.

Knowledge
The knowledge surface governs the quality of the information available to the system. Engineers intervene here by improving the corpus itself: removing stale documents, eliminating duplicates, correcting inconsistencies, or refining document organization. If the system retrieves incorrect knowledge, no downstream component can reliably recover the correct answer.

Retrieval
The retrieval surface determines which knowledge reaches the model. Engineers adjust retrieval algorithms, chunking strategies, embedding models, metadata filters, rerankers, and search parameters to improve the relevance and completeness of retrieved context.

Generation
The generation surface governs how retrieved context is transformed into an answer. Prompt design, model selection, decoding strategies, and structured output constraints all influence whether the model produces accurate, complete, and faithful responses.

Evaluation
The evaluation surface determines how reliability is measured and enforced. Rather than improving answers directly, evaluation establishes quality gates through automated benchmarks, regression tests, and production monitoring. It answers the question: Has reliability changed enough to require intervention?

Operations
The operations surface coordinates how the entire system behaves in production. Version management, deployment policies, rollout strategies, monitoring, traffic routing, and incident response all influence the long-term reliability of the application, even when individual components remain unchanged.

Dimension 3 — Detectability

The previous dimension answered where engineers should intervene. Detectability answers a different operational question:

How likely is this failure to be discovered before users experience it?

Not all failures are equally visible. Some immediately trigger monitoring systems, while others remain hidden behind apparently successful requests and fluent model responses. From an operational perspective, the cost of a failure depends not only on its severity but also on how long it remains undetected.

Traditional software systems have benefited from decades of investment in observability. Infrastructure failures, resource exhaustion, deployment regressions, and service interruptions typically produce measurable signals that monitoring systems can detect automatically.

Production AI systems introduce a different class of reliability problems. A request may complete successfully, latency may remain stable, and no infrastructure alarms may fire, yet answer quality can still deteriorate. In these cases, correctness—not availability—becomes the primary operational concern.

We therefore classify production AI failures according to their detectability.

D0 — Immediately observable
Failures are directly visible through conventional operational signals or obvious incorrect behavior. Engineers are typically alerted immediately through monitoring systems or user-facing errors.

D1 — Operationally observable
Failures become apparent through changes in production telemetry, deployment behavior, or runtime characteristics. Although the application continues functioning, operational metrics indicate that reliability has changed.

D2 — Evaluation observable
Failures cannot be detected reliably through infrastructure monitoring alone. Instead, they require scheduled or continuous evaluation using representative workloads to identify declining correctness, retrieval quality, or answer fidelity before users notice the regression.

D3 — Investigation observable
Failures remain operationally invisible until a specific customer incident triggers investigation. Root cause identification requires manual analysis, reproduction, and engineering judgment, making this class the most expensive and operationally disruptive.

The Complete AI Reliability Framework

The three dimensions introduced in this section are intended to be used together rather than independently.

Failure Dynamics describe how reliability changes over time. Reliability Control Surfaces identify where engineers should investigate and intervene. Detectability determines how failures become visible during operation. Together, these dimensions transform an isolated production incident into a structured reliability problem with an appropriate operational response.

Figure 3 summarizes the complete AI Reliability Framework. Rather than treating AI failures as disconnected symptoms such as hallucination, retrieval drift, or poor answer quality, the framework provides a systematic reasoning process—from identifying the failure dynamics to selecting the most appropriate operational response.

4. Controlled Experiment

To illustrate how the proposed framework can be applied in practice, we conducted a controlled experiment on a representative production RAG failure. Rather than attempting to validate every combination of failure dynamics, control surfaces, and detectability levels, this article focuses on a single, realistic scenario that demonstrates how the framework guides diagnosis and operational response.

The selected failure represents a Gradual failure dynamic, occurring on the Knowledge control surface with D2 (Evaluation Observable) detectability. This class of failure was chosen because it is both common in production RAG systems and operationally expensive. Reliability degrades incrementally as the knowledge corpus evolves, yet the decline often remains invisible until systematic evaluation reveals a measurable regression.

The experimental system consists of a Retrieval-Augmented Generation (RAG) application serving API documentation. Over a seven-week period, the underlying documentation corpus is progressively modified to simulate realistic knowledge evolution while the retrieval pipeline, model configuration, and evaluation dataset remain unchanged. This isolates knowledge evolution as the primary independent variable.

Each week, the system is evaluated against the same benchmark using four reliability metrics:

Source Hit@k — whether the correct supporting documents are successfully retrieved.
Answer Pass Rate — proportion of responses meeting predefined correctness criteria.
Faithfulness — degree to which generated answers are supported by retrieved evidence.
Unsupported Answer Rate — frequency of responses containing unsupported or fabricated information.

Figure 4 illustrates the overall experimental design.

5. Result

The controlled experiment produced three observations that characterize Gradual Knowledge Drift in production RAG systems.

Observation 1 — User-visible behavior changed before the system appeared to fail

Figure 5 follows a single production question across seven weeks.

Although the retrieval pipeline, evaluation dataset, and application remained unchanged, the answer gradually shifted from a correct response to repeated uncertainty before later converging on an incomplete explanation.

Importantly, the application never failed operationally. Every request completed successfully, yet the quality of the answer changed substantially as the knowledge corpus evolved.

week	question	answer
week_0	How should clients authenticate API requests?	Clients must include an `Authorization: Bearer <api_key>` header in all API requests.
week_1	How should clients authenticate API requests?	Unknown from the available documentation.
week_2	How should clients authenticate API requests?	Unknown from the available documentation.
week_3	How should clients authenticate API requests?	Unknown from the available documentation.
week_4	How should clients authenticate API requests?	Unknown from the available documentation.
week_5	How should clients authenticate API requests?	Clients authenticate API requests using a token, as indicated by the per-token rate limits and the `401` status code returned when authentication fails.
week_6	How should clients authenticate API requests?	Clients authenticate API requests using a token, as indicated by the per-token rate limits and the `401` status code returned when authentication fails.

Figure 5 - Q&A Result

Figure 5 demonstrates that user-visible reliability can deteriorate even when the system appears operationally healthy.

Observation 2 — Reliability degraded without operational failures

To determine whether this qualitative change reflected a measurable reliability decline, we evaluated the system weekly using four metrics.

Figure 6 records the evaluation results throughout the experiment.

Several observations stand out.

Source Hit@4 remained consistently high for most weeks.
Faithfulness changed only slightly.
No infrastructure failures or retrieval outages occurred.
However, Answer Pass Rate declined significantly, reaching its lowest point during Week 3.

The degradation therefore cannot be explained by infrastructure instability or catastrophic retrieval failure. Instead, the experiment demonstrates a gradual reduction in answer quality while conventional operational indicators remained largely stable.

Week	Source Hit@4	Answer Pass Rate	Faithfulness Pass Rate	Unsupported Answer Rate	Signal
week_0	1	0.85	1	0.05	Early drift visible
week_1	0.95	0.8	0.95	0.1	Early drift visible
week_2	0.95	0.85	0.95	0.05	Early drift visible
week_3	0.95	0.6	0.95	0.1	Material decay
week_4	0.95	0.8	0.95	0.05	Early drift visible
week_5	0.95	0.8	1	0	Early drift visible
week_6	0.85	0.75	0.95	0.05	Early drift visible

Figure 6 - Evaluation Record

Observation 3 — Progressive degradation only becomes obvious longitudinally

Figure 7 plots the evaluation metrics over time.

Viewed week-by-week, each individual regression appears relatively minor. None would typically justify an operational incident on its own.

Viewed longitudinally, however, the pattern becomes unmistakable. Small fluctuations accumulate into a sustained decline in answer quality despite stable infrastructure and largely unchanged retrieval metrics.

This illustrates an important characteristic of Gradual Knowledge Drift: the operational signal emerges from the trend rather than any individual observation.

Interpretation

The experiment supports the hypothesis that some production RAG failures evolve gradually rather than immediately.

Unlike a deployment regression, where a single event produces an obvious operational change, gradual knowledge drift emerges through a sequence of individually reasonable modifications to the knowledge corpus.

Each weekly change appears harmless in isolation. Collectively, however, they alter the behavior of the retrieval-generation system enough to produce measurable reliability degradation.

Consequently, conventional monitoring—which is designed to detect discrete failures—provides little warning before users begin experiencing degraded answers.

The appropriate operational response is therefore fundamentally different. Immediate failures require incident response and rollback. Gradual failures require continuous evaluation, longitudinal monitoring, and periodic reliability assessment to identify trends before they become customer-visible.

6. Operational Principals

Principle 1: Reliability should be monitored longitudinally rather than episodically.

The experiment demonstrates that no individual weekly change was sufficient to trigger an operational incident. Instead, reliability declined through a sequence of small, individually reasonable modifications to the knowledge corpus. This suggests that production AI reliability is fundamentally temporal. The reliability state of the system cannot be inferred from a single evaluation or deployment event; it emerges from trends observed over time.

Consequently, evaluation should not be treated as a periodic validation activity performed before release. It should become a continuous operational process capable of identifying long-term changes in system behavior before they become customer-visible.

Principle 2: Operational strategies should depend on failure dynamics rather than implementation components.

The observed degradation originated from changes to the knowledge corpus, yet the operational response was determined by the behavior of the failure rather than its technical origin. An immediate deployment regression and a gradual knowledge drift may involve the same retrieval pipeline, but they require fundamentally different operational strategies. One favors rollback and incident response; the other favors continuous evaluation and trend monitoring.

Principle 3: Different detectability classes require different operational controls.

Not every reliability failure can be discovered through conventional monitoring. The experiment illustrates a D2 (Evaluation Observable) failure. Operational metrics remained largely stable while answer quality gradually declined. Traditional infrastructure monitoring therefore provided little indication that reliability had changed.

Operational controls should therefore be matched to the detectability class of the failure. Immediately observable failures benefit from alerting and monitoring, whereas evaluation-observable failures require continuous benchmark evaluation capable of detecting subtle quality regressions before users encounter them.

Principle 4: Knowledge evolution is an operational concern, not merely a documentation concern.

Throughout the experiment, the application code remained unchanged. Reliability changed because the knowledge available to the system changed. This observation highlights an important distinction between traditional software systems and production AI systems. Documentation is no longer passive reference material consumed only by engineers; it has become executable operational state consumed directly by the application.

7. Limitations

This article intentionally evaluates a single representative failure profile rather than the complete AI Reliability Framework.

The objective of this experiment is not to validate every possible combination of Failure Dynamics, Reliability Control Surfaces, and Detectability levels. Instead, it demonstrates how the proposed framework can be applied to reason about a realistic production incident under controlled experimental conditions.

The selected scenario—Gradual failure dynamics on the Knowledge control surface with D2 (Evaluation Observable) detectability—was chosen because it represents a common production failure that is difficult to identify using conventional operational monitoring.

Future work will extend the experimental series to additional failure profiles, including:

Threshold failures
Oscillating failures
Cascading failures

using the same controlled experimental methodology. Evaluating multiple failure classes will allow the proposed framework to be assessed across a broader range of production AI reliability scenarios.

8. Conclusion

Production AI systems cannot be understood solely through their architectural components. Models, retrievers, prompts, and vector databases explain how an AI system is built, but they do not explain how reliability changes once the system enters production.

This article proposed an operational reliability framework that complements architectural thinking by introducing three additional dimensions: Failure Dynamics, Reliability Control Surfaces, and Detectability. Together, these dimensions provide a structured way to reason from a production incident toward an appropriate engineering response.

The controlled experiment on Gradual Knowledge Drift demonstrates one representative failure class within this framework. More importantly, it illustrates a broader operational reality: many production AI failures emerge not through catastrophic regressions, but through the accumulation of individually reasonable changes that gradually alter system behavior.

Traditional software engineering matured by developing shared languages for correctness, availability, and performance. As AI systems become long-lived production systems, reliability engineering will likewise require new operational concepts that describe how AI behavior evolves, how failures should be classified, and how reliability should be managed over time.

The framework presented here is one step toward that operational vocabulary.

Appendix A — Experiment Details

Corpus composition

auth.md - Explains bearer-token authentication and how to interpret 401 and 403 responses
errors.md - Summarizes HTTP error handling, including 429 rate limits, 5xx retries, and auth-related errors.
pagination.md - Describes cursor-based pagination using limit, next_cursor, and cursor.
rate_limits.md - Defines rate-limit responses, relevant headers, and worker coordination after 429s.
retry_behavior.md - Specifies retry rules for 429, 5xx, idempotent requests, jitter, backoff, and Retry-After.
troubleshooting.m - Lists support-debugging details, timeout guidance, and webhook retry behavior.

Weekly mutations

week 1 - Added overlapping troubleshooting content
week 2 - Added stale migration guide with outdated retry behavior
week 3 - Re-indexed retry behavior doc with worse chunking
week 4 - Added legacy SDK docs with similar wording
week 5 - Switched to looser prompt version
week 6 - Added noisy FAQ content competing with correct source

Support questions

When should clients retry after a 429 error?

What should an API client do when it receives 429 Too Many Requests?

Should clients use Retry-After or retry immediately after a 429?

How should workers behave after a rate limit retry?

Is a fixed 60 seconds the current retry policy for 429 responses?

Which header tells clients how long to wait after a rate limit?

Which operations should be retried automatically?

How should clients handle 5xx server errors?

How should clients authenticate API requests?

What does a 401 response mean?

What does a 403 response mean?

How do clients request the next page of results?

When should clients stop paginating?

What should a client do after a timeout?

What information should be included when contacting support?

Are webhook retries the same as client retry behavior?

What should clients do to request volume after receiving 429s?

Which headers describe the rate limit budget?

Should clients automatically retry requests that create side effects?

Which response value helps support investigate failures?

Repository link

rag-decay-experiment

Appendix B — Framework Summary

A summary the three-dimension reliability framework:

Failure Dynamics	Typical Control Surface	Typical Detectability
Immediate	Generation	D0
Gradual	Knowledge	D2
Threshold	Retrieval	D1–D2
Oscillating	Operations	D3
Cascading	Cross-surface	D3

Suggested Citation:

Lei Ye. Why Your Production RAG System Slowly Gets Worse. 2026. https://lei-ye.dev/blog/reliability-framework-for-ai-engineers/

The Hidden Problem With Prompts in Production AI

Lei Ye — Wed, 11 Mar 2026 23:36:45 +0000

Originally published at: Prompt as Code — Build Prompt Registry with Versioning

When teams first build AI features, prompts usually start simple.

A string in a function.
A template inside a route.
Maybe a small helper function.

Something like:

prompt = f"Summarize the following system event:\n\n{event_text}"

Then with system evolves, prompts start changing.

A word here.
A constraint there.
Someone adds a new instruction for better formatting.

Before long, the system behaves differently and nobody can explain why.

That’s when prompt chaos begins.

1. The Problem: Prompt Chaos

Unlike normal code, prompts are often invisible infrastructure.

They live inside strings scattered across services. They change quietly during experimentation.

Over time this creates several problems:

Responses change unexpectedly
Evaluation metrics become unreliable
Debugging becomes difficult
Prompt history disappears

If an output changes today, you may not know whether the cause was:

A prompt change ?
A model change ?
A parameter change ?

Without prompt identity, the system becomes difficult to reason about.

2. Why Prompts Need Versioning

Prompts influence system behavior as much as code does.

In fact, prompts are closer to configuration that drives behavior.

That means prompts deserve the same discipline as code:

Version control
Reproducibility
Traceability

Instead of treating prompts as strings, we can treat them as versioned assets.

This approach allows us to answer important questions:

Which prompt generated this output?
Which version was deployed last week?
Which prompt version performs best during evaluation?

This is the idea behind Prompt as Code.

3. What a Prompt Registry Is

A Prompt Registry is a small service responsible for managing prompt templates.

Instead of constructing prompts directly in application logic, the application resolves them from a registry.

A prompt registry provides:

Prompt templates
Version management
Deterministic rendering
Prompt hashing

This transforms prompts from ad-hoc strings into structured runtime assets.

Example prompt template:

PromptTemplate(
    name="system_summary",
    version="v2",
    template="""
You are a production AI assistant focused on reliability.
Summarize the following system event:
{event_text}
"""
)

Now prompts have identity.

4. Architecture

The prompt registry sits between the API layer and the model gateway.

Client Request
      ↓
Prompt Registry
      ↓
Rendered Prompt
      ↓
Model Gateway
      ↓
Provider Adapter
      ↓
Cost Metering
      ↓
Evaluation

This architecture ensures:

Prompts are resolved before inference
Prompt versions are logged
Evaluation remains reproducible

It also cleanly separates prompt management from model execution.

5. Implementation

In the Maester toolkit, the prompt registry lives inside:

packages/
  prompt_registry/
      models.py
      registry.py
      service.py
      hashing.py

A prompt template is defined as a structured object.

@dataclass
class PromptTemplate:
    name: str
    version: str
    template: str

Templates are stored in a registry:

registry.register(
    PromptTemplate(
        name="system_summary",
        version="v1",
        template="Summarize the following system event:\n\n{event_text}",
    )
)

When the API receives a request, the prompt service resolves and renders the prompt.

rendered = prompt_service.render(
    name="system_summary",
    variables={
        "event_text": "User downloaded a large dataset."
    }
)

The rendered prompt includes:

prompt name
prompt version
prompt content
prompt hash

The hash guarantees that the exact prompt used during inference can be traced later.

6. Prompt Versioning Examples

Once prompts become versioned assets, evaluation becomes much more reliable.

Each request now records:

model
provider
prompt name
prompt version
prompt hash

This allows teams to compare prompt performance across versions.

Example Prompt v1

Request:

{
    "prompt_name": "system_summary",
    "prompt_version": "v1",
    "variables": {
      "event_text": "Admin revoked API key for user account 742."
    },
    "model": "gpt-4.1-mini",
    "max_tokens": 120
  }

Response:

{"provider":"openai",
"model":"gpt-4.1-mini",
"trace_id":"61050fd4e94849d791e566ead8c8f1c6",
"prompt_name":"system_summary",
"prompt_version":"v1","prompt_hash":"5ba4cce2a985f8234698a63fe2260428b029dfd7d61e53a5793cc963b8737036",
"content":"[OpenAI:gpt-4.1-mini] Generated response for prompt: You are a production AI assistant.\nSummarize the following system event clearly:\n\nAdmin revoked API key for user account",
"cost":{
    "model":"gpt-4.1-mini",
    "input_tokens":40,
    "output_tokens":48,
    "total_tokens":88,
    "input_cost_usd":"0.000016",
    "output_cost_usd":"0.000077",
    "total_cost_usd":"0.000093",
    "unit":"USD"
},
"evaluation":{
    "status":"pass",
    "reliability_score":1.0,
    "metrics":[
        {"name":"non_empty","score":1.0,"passed":true,"reason":null},
        {"name":"max_length","score":1.0,"passed":true,"reason":null}
    ]
}}

Example Prompt v2

Request (default to latest):

{
    "prompt_name": "system_summary",
    "variables": {
      "event_text": "System latency increased above 300ms for the inference service."
    }
  }

Response:

{"provider":"openai",
"model":"gpt-4.1-mini",
"trace_id":"7ec85dc989dc4da8a0ac9bb73f2317a7",
"prompt_name":"system_summary",
"prompt_version":"v2",
"prompt_hash":"06c08f6125a189abf90b44c9a63a5bc0f5307f06319363a922a476b38776b8c6",
"content":"[OpenAI:gpt-4.1-mini] Generated response for prompt: You are a production AI assistant focused on reliability.\nSummarize the following system event.\nBe concise, mention operational impact, and keep the tone factual.\n\nSystem latency increased above 300ms for the inference service.",
"cost":{
    "model":"gpt-4.1-mini",
    "input_tokens":66,
    "output_tokens":46,
    "total_tokens":112,
    "input_cost_usd":"0.000026",
    "output_cost_usd":"0.000074",
    "total_cost_usd":"0.000100",
    "unit":"USD"
},
"evaluation":{
    "status":"pass",
    "reliability_score":1.0,
    "metrics":[
        {"name":"non_empty","score":1.0,"passed":true,"reason":null},
        {"name":"max_length","score":1.0,"passed":true,"reason":null}
    ]
}}

Now prompt optimization becomes measurable rather than guesswork.

7. Lessons Learned

Building a prompt registry revealed a few important lessons.

1. Prompts evolve quickly

Even small systems accumulate many prompt variations.

2. Reproducibility matters early

Without prompt versioning, evaluation results become meaningless.

3. Prompt identity simplifies debugging

When responses change, engineers can immediately identify the cause.

4. Prompts should live outside business logic

Separating prompts from application code improves maintainability.

8. The Code

The implementation described in this article is part of an open-source project called Maester.

Maester is a lightweight toolkit focused on AI API reliability, including:

Model gateway routing
Cost metering
Observability
Evaluation pipelines
Prompt registry

Repository: maester

The goal is to explore how production AI systems can remain observable, reproducible, and resilient as they grow.

Note: This article was originally published on my engineering blog where I’m documenting the design of Maester, a production AI SaaS infrastructure system built in public. Original post:Prompt as Code — Build Prompt Registry with Versioning

Building Maester — Enable Multi-provider LLM APIs

Lei Ye — Tue, 10 Mar 2026 21:28:17 +0000

Originally published at Building Maester — Enable Multi-provider LLM APIs.

We Locked Ourselves Into GCP

Most infrastructure mistakes don’t start as mistakes. They start as reasonable decisions. This one started with a discount.

It Worked Beautifully

In the beginning, the decision felt obvious. We had a large GCP startup credit, so our entire stack ran there.

Compute.
Storage.
Data pipelines.
Model training.
...
Everything.

And honestly, it worked beautifully! Monitoring was already integrated.
Identity management was built in. IAM policies were easy to manage.
Even LDAP integration was already available.

One of my teammates said something that sounded perfectly reasonable:

“Don’t reinvent the wheel.”

And he was right. Why build infrastructure when the cloud already solved it?

We were a small team. Most of our compute was tied to token usage, so costs looked predictable. Everything felt lightweight.

So we did what most startups do. We committed.

Where Did the Cost Come From?

"Don't spend like a billionaire with the company's money !"

What? We were all so much confused with the bill complaints at a Monday morning standup meeting months later. The bill arrived and nobody could clearly explain it. It was the cloud bill. And it started eating into margins.

Where did the cost come from?
Storage?
Network egress?
Pipelines?
Inference traffic?

Someone suggested hiring a cloud optimization engineer. Another suggested redesigning the entire data pipeline.

But we were still a startup. Every time we opened the roadmap we saw something else staring at us:

Customer requests.
Feature releases.
Revenue milestones.

Infrastructure work always lost that fight. So the bills kept climbing. We weren't bankrupt. But we were trapped.

We Split the Stack

Eventually we did something radical. We split the stack. The architecture finally looked like this:

Azure → Identity / Compliance
AWS → Applications / Storage
GCP → Data Pipelines / Training

And the cost?

Still expensive. But predictable. Even without our original startup discount, the system became easier to control.

Vendor lock-in is invisible when things work. It becomes obvious only when you try to leave.

We Are Not Going to Lock into OpenAI

So when we started building the AI APIs, I began seeing the same pattern again.

It was just:

response = openai.responses.create(...)

And honestly, that works.

But I kept remembering the GCP moment. The moment when switching vendors became impossible. We were about to repeat the same mistake.

Except this time the vendor was not a cloud. It was a model provider. So I made a decision. We are not going to lock into OpenAI.

Approach 1 — Let the client choose the model

The simplest idea was letting the client select the model.

POST /generate
{
  "model": "gpt-4.1-mini"
}

This allowed switching between providers.

OpenAI
Anthropic
Others later

Technically it worked. But users quickly complained.

“I don't want to choose the model. I just want the best answer.”

The user is always right. They just wanted results.

Approach 2 — Introduce a Model Gateway

So we moved the decision out of the client. Instead of clients choosing providers, we introduced a Model Gateway.

Application
     ↓
Model Gateway
     ↓
Provider Router
     ↓
Provider Adapter
(OpenAI / Anthropic / others)

This gateway would manage:

provider routing
fallback logic
cost tracking
observability
evaluation

The application now simply asks for a response. And the infrastructure decides how to produce it.

The Real Code

The implementation lives inside a small reference project I’ve been building called Maester.

The goal of the project is not to build a full AI platform, but to demonstrate a reliable AI API architecture.

The gateway sits inside the system like this:

apps/
   api/
      routes/
         reliable_completion.py

packages/
   model_gateway/
      base.py
      provider_openai.py
      provider_anthropic.py
      router.py
      client.py

The Provider Contract

The first step was defining a provider interface. This follows the Adapter Pattern, allowing different model vendors to conform to a shared interface.

class ModelProvider(Protocol):

    def supports(self, model: str) -> bool:
        ...

    def generate(self, request: GenerationRequest) -> GenerationResponse:
        ...

Each provider adapter simply implements this contract.

For example:

OpenAIProvider
AnthropicProvider

Both produce the same normalized response.

GenerationResponse
 ├─ provider
 ├─ model
 ├─ content
 └─ usage

This means the rest of the system never deals with vendor-specific formats.

The Router

Next comes the router. The router decides which provider handles a request.

class ModelRouter:

    def route(self, model: str) -> ModelProvider:
        for provider in self.providers:
            if provider.supports(model):
                return provider

        return self.fallback_provider

In production systems this layer can later evolve into:

cost-aware routing
latency-aware routing
capability routing
traffic shaping

But the interface stays the same.

The Gateway Client

Finally the application talks to the gateway through a simple client.

class ModelGateway:

    def generate(self, model: str, prompt: str, max_tokens: int):
        request = GenerationRequest(
            model=model,
            prompt=prompt,
            max_tokens=max_tokens,
        )

        provider = self.router.route(model)

        return provider.generate(request)

The API layer doesn't know which provider was selected. It just receives a normalized response.

The API Layer

The FastAPI route becomes extremely simple.

model_response = model_gateway.generate(
    model=requested_model,
    prompt=payload.prompt,
    max_tokens=payload.max_tokens,
)

After generation, the system runs the reliability pipeline:

Cost metering
Evaluation
Structured logging

Example log:

model_routed
requested_model: gpt-4.1-mini
selected_provider: openai
fallback_used: false

This gives operators visibility without leaking provider logic into application code.

Why This Architecture Matters

This design combines several classic software engineering principles:

Dependency Inversion Application code depends on abstractions, not providers.
Adapter Pattern Each vendor SDK is wrapped behind a provider adapter.
Strategy Pattern Routing policies are interchangeable strategies.
Separation of Concerns API layer handles orchestration.Gateway handles provider logic.

What This Enables Later

Once this boundary exists, the system becomes far easier to evolve.

For example:

multi-provider fallback
provider benchmarking
cost-aware routing
latency optimization
evaluation-based routing

All of those changes can happen inside the gateway. The application API never changes. That is the real value of the design.

The Lesson

Vendor lock-in rarely feels dangerous at the beginning. Everything works. Costs look reasonable. The roadmap is full of features.

Then one day something changes. Prices rise. Performance shifts. A better provider appears.And suddenly the architecture makes switching painful.

The lesson I learned from our cloud migration was simple: Always design one layer where you can change your mind later.

For our AI systems, that layer became the Model Gateway.

The application talks to the gateway.
The gateway talks to providers.
And the providers can change.

Because eventually they always do.

Note: This article was originally published on my egineering blog where I document the design of Maester, an AI SaaS infrastructure system built in public.
Original post: Building Maester — Enable Multi-provider LLM APIs.

What Breaks After Your AI Demo Works

Lei Ye — Sun, 08 Mar 2026 05:32:45 +0000

Originally published at What Breaks After Your AI Demo Works.

A Short Story of How My AI Demo Worked and Failed

A few weeks ago I built a small AI API. Nothing fancy. Just a simple endpoint.

response = llm(prompt)

It worked.

Requests came in. The model responded.Everything looked good.
Until the second week.

The First Question

A teammate asked:

“Which request generated this output?”
I checked the logs. There was nothing useful there.

NO request ID.
NO trace.
NO connection between the prompt and the output.

The system worked — but it wasn’t traceable.

The Second Question

Very quickly another question appeared.

“Why did our AI bill jump yesterday?”

I had no answer.

We were calling models through an API wrapper, but we weren’t recording:

Token usage
Model pricing
Request-level cost

We had built an AI system that spent money invisibly.

The Third Question

Then something more subtle happened.

A user reported that an output looked wrong. The model had responded successfully, but the answer was clearly not useful. Which raised another question:

“How do we know if a model response is acceptable?”

We didn’t.

The API only knew whether the model responded, not whether the result made sense.

The Realization

The model wasn't the problem. The system around the model was. AI APIs are fundamentally different from traditional APIs. They introduce three operational challenges:

Challenge	What it means
Observability	Can we trace what happened?
Economics	How much did this request cost?
Output reliability	Was the response acceptable?

Without solving these, AI systems quickly become hard to operate. So I built a small reference project to explore this problem. I called it Maester.

The Minimal Reliability Architecture

A reliable AI API request should pass through a few structured steps.

Client Request
      ↓
API Middleware
(request_id + trace_id)
      ↓
Route Handler
      ↓
Model Gateway
      ↓
Cost Metering
      ↓
Evaluation
      ↓
Structured Logs
      ↓
Response

Each step adds operational clarity.

1. Observability: Making AI Requests Traceable

The first primitive is observability. Every request should be traceable.
In Maester, middleware attaches:

request_id
trace_id

to the request context.

Example:

request_id = new_id()
trace_id = start_trace()

These identifiers propagate through the entire request lifecycle. Then operations are wrapped in spans:

with span("model_generate", model=model_name) as sp:
    response = gateway.generate(prompt)

The span records:

Operation name
Duration
Attributes

Example log output:

{
  "event": "span_end",
  "span": "model_generate",
  "duration_ms": 412,
  "model": "gpt-4o-mini"
}

This gives immediate insight into where time is spent.

2. Cost Metering: AI Systems Spend Money Per Request

Unlike traditional APIs, AI requests have direct monetary cost. Token usage translates into real spend. So every request should produce a cost record.

Example:

cost_record = meter.record(
    model=model_name,
    input_tokens=usage.input_tokens,
    output_tokens=usage.output_tokens,
)

The meter uses a pricing catalog:

MODEL_PRICING = {
    "gpt-4o-mini": {
        "input_per_1k": 0.00015,
        "output_per_1k": 0.00060,
    }
}

The request returns:

input_tokens
output_tokens
total_cost

Example response fragment:

{
  "input_tokens": 1200,
  "output_tokens": 350,
  "total_cost_usd": 0.00042
}

Now the API answers a critical question:

“What did this request cost?”

3. Evaluation: Successful Calls Aren’t Always Correct

Even if a model responds successfully, the output may still be unusable.That is where evaluation comes in.

In Maester, responses pass through a simple evaluator:

result = evaluator.evaluate(prompt, response)

Current checks include:

Non-empty response
Required term presence
Maximum length

Example evaluation result:

{
  "passed": true,
  "checks": {
    "non_empty": true,
    "required_terms": true,
    "max_length": true
  }
}

This pattern becomes more important as systems grow. Evaluation can evolve into:

Structured output validation
Hallucination detection
Policy enforcement
Safety filters

Why Not Just Use OpenTelemetry

I thoght to just adopt OpenTelemetry at the very beginning of this project, but decided to use home-made instead. Because OpenTelemetry solves a different problem. It provides:

Distributed tracing
Metrics exporters
Telemetry pipelines

But Maester focuses on application-level reliability primitives. Think of it as the layer that answers:

What happened in this AI request?
What model was called?
What did it cost?
Did the result pass validation?

These signals can later be exported to full observability stacks.

The Worker Path

AI systems rarely run only inside HTTP requests. Background jobs often run:

Batch inference
Evaluation pipelines
Data enrichment tasks

Maester includes a worker example to demonstrate that the same reliability primitives apply there. Worker execution uses the same tools:

Tracing
Cost metering
Evaluation
Structured logs

Reliability should not depend on the entrypoint.

What This Architecture Achieves

With only a few modules, the system now answers:

Question	Component
What request generated this output?	tracing
How long did the model call take?	spans
How many tokens were used?	cost meter
What did it cost?	pricing model
Was the output valid?	evaluator

These signals turn a black-box AI API into a traceable system.

Final Thought

Most reliability discussions around AI focus on models. But reliability often comes from system design, not model quality.

A simple architecture that records:

1. What happened
2. What it costs
3. Whether the result was acceptable

can dramatically improve how AI systems are operated.

The earlier these ideas are introduced into a system, the easier that system will be to maintain.

Note: This article was originally published on my engineering blog where I document the design of Maester, an AI SaaS infrastructure system built in public.
Original post: What Breaks After Your AI Demo Works.

[Boost]

Lei Ye — Thu, 05 Mar 2026 20:12:17 +0000

Introducing Maester

Lei Ye ・ Mar 5

#ai #saas #architecture #machinelearning

Introducing Maester

Lei Ye — Thu, 05 Mar 2026 20:03:16 +0000

The Knowledge Engine of Your Company

Most companies today want the same thing from AI: Turn their internal knowledge into something queryable, explainable, and operational.

In practice this means:

Documents scattered across tools
Institutional knowledge trapped in teams
Data that exists but cannot be used

And the typical solution becomes:

“Let’s build an AI assistant.”

But building an AI demo and building AI infrastructure that survives production are very different things.

Maester is our attempt to build the latter.

What Maester Is

Maester is a reference implementation of a B2B SaaS AI knowledge engine. It demonstrates how a company can transform internal data into a production-grade knowledge system.

At its core, Maester allows organizations to:

ingest internal documents
structure and embed them
retrieve relevant knowledge
generate responses with citations
trace every operation across the system

But more importantly, Maester is designed as infrastructure, not just an AI feature. That means we are focusing on:

reliability
traceability
operational cost control
asynchronous pipelines
multi-tenant architecture

We are building this project in public, both as a working system and as a learning artifact. Every design choice will be documented. Every architecture decision will be explained.

This blog serves as a system design journal

The Infrastructure Problem Most AI SaaS Products Ignore

When teams first add AI to a product, the initial version often works. A prototype connects an LLM, retrieves some documents, and produces answers. But once the system meets real users, things break quickly. We repeatedly see the same failure modes:

1. Timeouts

LLM calls are slow and unpredictable. Without proper timeouts and retries, requests cascade into system failures.

2. Uncontrolled costs

Every query triggers embedding calls, retrieval operations, and model inference. Without cost tracking and guardrails, usage grows faster than expected.

3. Queues and ingestion pipelines

Document ingestion is not instantaneous. Parsing, chunking, and embedding require asynchronous pipelines that many systems lack.

4. Traceability gaps

When something goes wrong, teams often cannot answer simple questions:

What document generated this answer?
Which embedding version was used?
Which model responded?

Without observability, AI becomes a black box in production.

But What “Production-ready AI Infrastructure” Actually Means

For us, production readiness is not about model quality. It is about system design. A production AI SaaS system must provide:

1. Asynchronous ingestion pipelines

Documents must move through structured stages:
parse → chunk → embed → index.
Each stage should be observable and retryable.

2. Reliable model access

All LLM access must go through a gateway that manages:

timeouts
retries
provider fallback

3. Usage and cost accounting

Every request must produce a usage record. Production systems must answer:

Which tenant generated this cost?
Which model generated this response?

4. Traceability

Requests must carry a correlation ID through:

API layer
worker queues
model calls

This is how production systems become debuggable.

How Maester is Structured

Instead of treating AI as a feature, we treat it as infrastructure. Maester separates the system into clear operational layers.

Architectural Layers

API Layer

Handles request entry, tenant routing, and request validation. This layer also generates request IDs used for tracing.

Knowledge Engine Core

This is where Maester’s core logic lives. Responsibilities include:

document retrieval
query orchestration
interaction with the model gateway
enforcing cost budgets

Async Worker System

All heavy processing moves to asynchronous workers:

document parsing
chunking
embedding
vector indexing

This prevents ingestion tasks from blocking user requests.

Model Gateway

Instead of calling models directly, all inference flows through a gateway. This gateway manages:

provider abstraction
retry logic
token usage tracking
future fallback support

Observability Layer

We treat observability as a first-class concern. Every request is traceable across:

API requests
worker jobs
model calls

This allows production debugging without guesswork.

Closing

Maester is not just an AI application.

It is an exploration of how AI systems should be engineered. In the coming posts, we will document:

architecture decisions
reliability patterns
cost control strategies
production ML infrastructure design

Our goal is simple: To build a knowledge engine that companies can trust in production. And to make every engineering decision transparent and explainable.

The system starts SMALL.

But the architecture is designed to SCALE.

Originally published on my engineering blog: https://lei-ye.dev/blog/introducing-maester