DEV Community: Lei Ye

The Hidden Problem With Prompts in Production AI

Lei Ye — Wed, 11 Mar 2026 23:36:45 +0000

Originally published at: Prompt as Code — Build Prompt Registry with Versioning

When teams first build AI features, prompts usually start simple.

A string in a function.
A template inside a route.
Maybe a small helper function.

Something like:

prompt = f"Summarize the following system event:\n\n{event_text}"

Then with system evolves, prompts start changing.

A word here.
A constraint there.
Someone adds a new instruction for better formatting.

Before long, the system behaves differently and nobody can explain why.

That’s when prompt chaos begins.

1. The Problem: Prompt Chaos

Unlike normal code, prompts are often invisible infrastructure.

They live inside strings scattered across services. They change quietly during experimentation.

Over time this creates several problems:

Responses change unexpectedly
Evaluation metrics become unreliable
Debugging becomes difficult
Prompt history disappears

If an output changes today, you may not know whether the cause was:

A prompt change ?
A model change ?
A parameter change ?

Without prompt identity, the system becomes difficult to reason about.

2. Why Prompts Need Versioning

Prompts influence system behavior as much as code does.

In fact, prompts are closer to configuration that drives behavior.

That means prompts deserve the same discipline as code:

Version control
Reproducibility
Traceability

Instead of treating prompts as strings, we can treat them as versioned assets.

This approach allows us to answer important questions:

Which prompt generated this output?
Which version was deployed last week?
Which prompt version performs best during evaluation?

This is the idea behind Prompt as Code.

3. What a Prompt Registry Is

A Prompt Registry is a small service responsible for managing prompt templates.

Instead of constructing prompts directly in application logic, the application resolves them from a registry.

A prompt registry provides:

Prompt templates
Version management
Deterministic rendering
Prompt hashing

This transforms prompts from ad-hoc strings into structured runtime assets.

Example prompt template:

PromptTemplate(
    name="system_summary",
    version="v2",
    template="""
You are a production AI assistant focused on reliability.
Summarize the following system event:
{event_text}
"""
)

Now prompts have identity.

4. Architecture

The prompt registry sits between the API layer and the model gateway.

Client Request
      ↓
Prompt Registry
      ↓
Rendered Prompt
      ↓
Model Gateway
      ↓
Provider Adapter
      ↓
Cost Metering
      ↓
Evaluation

This architecture ensures:

Prompts are resolved before inference
Prompt versions are logged
Evaluation remains reproducible

It also cleanly separates prompt management from model execution.

5. Implementation

In the Maester toolkit, the prompt registry lives inside:

packages/
  prompt_registry/
      models.py
      registry.py
      service.py
      hashing.py

A prompt template is defined as a structured object.

@dataclass
class PromptTemplate:
    name: str
    version: str
    template: str

Templates are stored in a registry:

registry.register(
    PromptTemplate(
        name="system_summary",
        version="v1",
        template="Summarize the following system event:\n\n{event_text}",
    )
)

When the API receives a request, the prompt service resolves and renders the prompt.

rendered = prompt_service.render(
    name="system_summary",
    variables={
        "event_text": "User downloaded a large dataset."
    }
)

The rendered prompt includes:

prompt name
prompt version
prompt content
prompt hash

The hash guarantees that the exact prompt used during inference can be traced later.

6. Prompt Versioning Examples

Once prompts become versioned assets, evaluation becomes much more reliable.

Each request now records:

model
provider
prompt name
prompt version
prompt hash

This allows teams to compare prompt performance across versions.

Example Prompt v1

Request:

{
    "prompt_name": "system_summary",
    "prompt_version": "v1",
    "variables": {
      "event_text": "Admin revoked API key for user account 742."
    },
    "model": "gpt-4.1-mini",
    "max_tokens": 120
  }

Response:

{"provider":"openai",
"model":"gpt-4.1-mini",
"trace_id":"61050fd4e94849d791e566ead8c8f1c6",
"prompt_name":"system_summary",
"prompt_version":"v1","prompt_hash":"5ba4cce2a985f8234698a63fe2260428b029dfd7d61e53a5793cc963b8737036",
"content":"[OpenAI:gpt-4.1-mini] Generated response for prompt: You are a production AI assistant.\nSummarize the following system event clearly:\n\nAdmin revoked API key for user account",
"cost":{
    "model":"gpt-4.1-mini",
    "input_tokens":40,
    "output_tokens":48,
    "total_tokens":88,
    "input_cost_usd":"0.000016",
    "output_cost_usd":"0.000077",
    "total_cost_usd":"0.000093",
    "unit":"USD"
},
"evaluation":{
    "status":"pass",
    "reliability_score":1.0,
    "metrics":[
        {"name":"non_empty","score":1.0,"passed":true,"reason":null},
        {"name":"max_length","score":1.0,"passed":true,"reason":null}
    ]
}}

Example Prompt v2

Request (default to latest):

{
    "prompt_name": "system_summary",
    "variables": {
      "event_text": "System latency increased above 300ms for the inference service."
    }
  }

Response:

{"provider":"openai",
"model":"gpt-4.1-mini",
"trace_id":"7ec85dc989dc4da8a0ac9bb73f2317a7",
"prompt_name":"system_summary",
"prompt_version":"v2",
"prompt_hash":"06c08f6125a189abf90b44c9a63a5bc0f5307f06319363a922a476b38776b8c6",
"content":"[OpenAI:gpt-4.1-mini] Generated response for prompt: You are a production AI assistant focused on reliability.\nSummarize the following system event.\nBe concise, mention operational impact, and keep the tone factual.\n\nSystem latency increased above 300ms for the inference service.",
"cost":{
    "model":"gpt-4.1-mini",
    "input_tokens":66,
    "output_tokens":46,
    "total_tokens":112,
    "input_cost_usd":"0.000026",
    "output_cost_usd":"0.000074",
    "total_cost_usd":"0.000100",
    "unit":"USD"
},
"evaluation":{
    "status":"pass",
    "reliability_score":1.0,
    "metrics":[
        {"name":"non_empty","score":1.0,"passed":true,"reason":null},
        {"name":"max_length","score":1.0,"passed":true,"reason":null}
    ]
}}

Now prompt optimization becomes measurable rather than guesswork.

7. Lessons Learned

Building a prompt registry revealed a few important lessons.

1. Prompts evolve quickly

Even small systems accumulate many prompt variations.

2. Reproducibility matters early

Without prompt versioning, evaluation results become meaningless.

3. Prompt identity simplifies debugging

When responses change, engineers can immediately identify the cause.

4. Prompts should live outside business logic

Separating prompts from application code improves maintainability.

8. The Code

The implementation described in this article is part of an open-source project called Maester.

Maester is a lightweight toolkit focused on AI API reliability, including:

Model gateway routing
Cost metering
Observability
Evaluation pipelines
Prompt registry

Repository: maester

The goal is to explore how production AI systems can remain observable, reproducible, and resilient as they grow.

Note: This article was originally published on my engineering blog where I’m documenting the design of Maester, a production AI SaaS infrastructure system built in public. Original post:Prompt as Code — Build Prompt Registry with Versioning

Building Maester — Enable Multi-provider LLM APIs

Lei Ye — Tue, 10 Mar 2026 21:28:17 +0000

Originally published at Building Maester — Enable Multi-provider LLM APIs.

We Locked Ourselves Into GCP

Most infrastructure mistakes don’t start as mistakes. They start as reasonable decisions. This one started with a discount.

It Worked Beautifully

In the beginning, the decision felt obvious. We had a large GCP startup credit, so our entire stack ran there.

Compute.
Storage.
Data pipelines.
Model training.
...
Everything.

And honestly, it worked beautifully! Monitoring was already integrated.
Identity management was built in. IAM policies were easy to manage.
Even LDAP integration was already available.

One of my teammates said something that sounded perfectly reasonable:

“Don’t reinvent the wheel.”

And he was right. Why build infrastructure when the cloud already solved it?

We were a small team. Most of our compute was tied to token usage, so costs looked predictable. Everything felt lightweight.

So we did what most startups do. We committed.

Where Did the Cost Come From?

"Don't spend like a billionaire with the company's money !"

What? We were all so much confused with the bill complaints at a Monday morning standup meeting months later. The bill arrived and nobody could clearly explain it. It was the cloud bill. And it started eating into margins.

Where did the cost come from?
Storage?
Network egress?
Pipelines?
Inference traffic?

Someone suggested hiring a cloud optimization engineer. Another suggested redesigning the entire data pipeline.

But we were still a startup. Every time we opened the roadmap we saw something else staring at us:

Customer requests.
Feature releases.
Revenue milestones.

Infrastructure work always lost that fight. So the bills kept climbing. We weren't bankrupt. But we were trapped.

We Split the Stack

Eventually we did something radical. We split the stack. The architecture finally looked like this:

Azure → Identity / Compliance
AWS → Applications / Storage
GCP → Data Pipelines / Training

And the cost?

Still expensive. But predictable. Even without our original startup discount, the system became easier to control.

Vendor lock-in is invisible when things work. It becomes obvious only when you try to leave.

We Are Not Going to Lock into OpenAI

So when we started building the AI APIs, I began seeing the same pattern again.

It was just:

response = openai.responses.create(...)

And honestly, that works.

But I kept remembering the GCP moment. The moment when switching vendors became impossible. We were about to repeat the same mistake.

Except this time the vendor was not a cloud. It was a model provider. So I made a decision. We are not going to lock into OpenAI.

Approach 1 — Let the client choose the model

The simplest idea was letting the client select the model.

POST /generate
{
  "model": "gpt-4.1-mini"
}

This allowed switching between providers.

OpenAI
Anthropic
Others later

Technically it worked. But users quickly complained.

“I don't want to choose the model. I just want the best answer.”

The user is always right. They just wanted results.

Approach 2 — Introduce a Model Gateway

So we moved the decision out of the client. Instead of clients choosing providers, we introduced a Model Gateway.

Application
     ↓
Model Gateway
     ↓
Provider Router
     ↓
Provider Adapter
(OpenAI / Anthropic / others)

This gateway would manage:

provider routing
fallback logic
cost tracking
observability
evaluation

The application now simply asks for a response. And the infrastructure decides how to produce it.

The Real Code

The implementation lives inside a small reference project I’ve been building called Maester.

The goal of the project is not to build a full AI platform, but to demonstrate a reliable AI API architecture.

The gateway sits inside the system like this:

apps/
   api/
      routes/
         reliable_completion.py

packages/
   model_gateway/
      base.py
      provider_openai.py
      provider_anthropic.py
      router.py
      client.py

The Provider Contract

The first step was defining a provider interface. This follows the Adapter Pattern, allowing different model vendors to conform to a shared interface.

class ModelProvider(Protocol):

    def supports(self, model: str) -> bool:
        ...

    def generate(self, request: GenerationRequest) -> GenerationResponse:
        ...

Each provider adapter simply implements this contract.

For example:

OpenAIProvider
AnthropicProvider

Both produce the same normalized response.

GenerationResponse
 ├─ provider
 ├─ model
 ├─ content
 └─ usage

This means the rest of the system never deals with vendor-specific formats.

The Router

Next comes the router. The router decides which provider handles a request.

class ModelRouter:

    def route(self, model: str) -> ModelProvider:
        for provider in self.providers:
            if provider.supports(model):
                return provider

        return self.fallback_provider

In production systems this layer can later evolve into:

cost-aware routing
latency-aware routing
capability routing
traffic shaping

But the interface stays the same.

The Gateway Client

Finally the application talks to the gateway through a simple client.

class ModelGateway:

    def generate(self, model: str, prompt: str, max_tokens: int):
        request = GenerationRequest(
            model=model,
            prompt=prompt,
            max_tokens=max_tokens,
        )

        provider = self.router.route(model)

        return provider.generate(request)

The API layer doesn't know which provider was selected. It just receives a normalized response.

The API Layer

The FastAPI route becomes extremely simple.

model_response = model_gateway.generate(
    model=requested_model,
    prompt=payload.prompt,
    max_tokens=payload.max_tokens,
)

After generation, the system runs the reliability pipeline:

Cost metering
Evaluation
Structured logging

Example log:

model_routed
requested_model: gpt-4.1-mini
selected_provider: openai
fallback_used: false

This gives operators visibility without leaking provider logic into application code.

Why This Architecture Matters

This design combines several classic software engineering principles:

Dependency Inversion Application code depends on abstractions, not providers.
Adapter Pattern Each vendor SDK is wrapped behind a provider adapter.
Strategy Pattern Routing policies are interchangeable strategies.
Separation of Concerns API layer handles orchestration.Gateway handles provider logic.

What This Enables Later

Once this boundary exists, the system becomes far easier to evolve.

For example:

multi-provider fallback
provider benchmarking
cost-aware routing
latency optimization
evaluation-based routing

All of those changes can happen inside the gateway. The application API never changes. That is the real value of the design.

The Lesson

Vendor lock-in rarely feels dangerous at the beginning. Everything works. Costs look reasonable. The roadmap is full of features.

Then one day something changes. Prices rise. Performance shifts. A better provider appears.And suddenly the architecture makes switching painful.

The lesson I learned from our cloud migration was simple: Always design one layer where you can change your mind later.

For our AI systems, that layer became the Model Gateway.

The application talks to the gateway.
The gateway talks to providers.
And the providers can change.

Because eventually they always do.

Note: This article was originally published on my egineering blog where I document the design of Maester, an AI SaaS infrastructure system built in public.
Original post: Building Maester — Enable Multi-provider LLM APIs.

What Breaks After Your AI Demo Works

Lei Ye — Sun, 08 Mar 2026 05:32:45 +0000

Originally published at What Breaks After Your AI Demo Works.

A Short Story of How My AI Demo Worked and Failed

A few weeks ago I built a small AI API. Nothing fancy. Just a simple endpoint.

response = llm(prompt)

It worked.

Requests came in. The model responded.Everything looked good.
Until the second week.

The First Question

A teammate asked:

“Which request generated this output?”
I checked the logs. There was nothing useful there.

NO request ID.
NO trace.
NO connection between the prompt and the output.

The system worked — but it wasn’t traceable.

The Second Question

Very quickly another question appeared.

“Why did our AI bill jump yesterday?”

I had no answer.

We were calling models through an API wrapper, but we weren’t recording:

Token usage
Model pricing
Request-level cost

We had built an AI system that spent money invisibly.

The Third Question

Then something more subtle happened.

A user reported that an output looked wrong. The model had responded successfully, but the answer was clearly not useful. Which raised another question:

“How do we know if a model response is acceptable?”

We didn’t.

The API only knew whether the model responded, not whether the result made sense.

The Realization

The model wasn't the problem. The system around the model was. AI APIs are fundamentally different from traditional APIs. They introduce three operational challenges:

Challenge	What it means
Observability	Can we trace what happened?
Economics	How much did this request cost?
Output reliability	Was the response acceptable?

Without solving these, AI systems quickly become hard to operate. So I built a small reference project to explore this problem. I called it Maester.

The Minimal Reliability Architecture

A reliable AI API request should pass through a few structured steps.

Client Request
      ↓
API Middleware
(request_id + trace_id)
      ↓
Route Handler
      ↓
Model Gateway
      ↓
Cost Metering
      ↓
Evaluation
      ↓
Structured Logs
      ↓
Response

Each step adds operational clarity.

1. Observability: Making AI Requests Traceable

The first primitive is observability. Every request should be traceable.
In Maester, middleware attaches:

request_id
trace_id

to the request context.

Example:

request_id = new_id()
trace_id = start_trace()

These identifiers propagate through the entire request lifecycle. Then operations are wrapped in spans:

with span("model_generate", model=model_name) as sp:
    response = gateway.generate(prompt)

The span records:

Operation name
Duration
Attributes

Example log output:

{
  "event": "span_end",
  "span": "model_generate",
  "duration_ms": 412,
  "model": "gpt-4o-mini"
}

This gives immediate insight into where time is spent.

2. Cost Metering: AI Systems Spend Money Per Request

Unlike traditional APIs, AI requests have direct monetary cost. Token usage translates into real spend. So every request should produce a cost record.

Example:

cost_record = meter.record(
    model=model_name,
    input_tokens=usage.input_tokens,
    output_tokens=usage.output_tokens,
)

The meter uses a pricing catalog:

MODEL_PRICING = {
    "gpt-4o-mini": {
        "input_per_1k": 0.00015,
        "output_per_1k": 0.00060,
    }
}

The request returns:

input_tokens
output_tokens
total_cost

Example response fragment:

{
  "input_tokens": 1200,
  "output_tokens": 350,
  "total_cost_usd": 0.00042
}

Now the API answers a critical question:

“What did this request cost?”

3. Evaluation: Successful Calls Aren’t Always Correct

Even if a model responds successfully, the output may still be unusable.That is where evaluation comes in.

In Maester, responses pass through a simple evaluator:

result = evaluator.evaluate(prompt, response)

Current checks include:

Non-empty response
Required term presence
Maximum length

Example evaluation result:

{
  "passed": true,
  "checks": {
    "non_empty": true,
    "required_terms": true,
    "max_length": true
  }
}

This pattern becomes more important as systems grow. Evaluation can evolve into:

Structured output validation
Hallucination detection
Policy enforcement
Safety filters

Why Not Just Use OpenTelemetry

I thoght to just adopt OpenTelemetry at the very beginning of this project, but decided to use home-made instead. Because OpenTelemetry solves a different problem. It provides:

Distributed tracing
Metrics exporters
Telemetry pipelines

But Maester focuses on application-level reliability primitives. Think of it as the layer that answers:

What happened in this AI request?
What model was called?
What did it cost?
Did the result pass validation?

These signals can later be exported to full observability stacks.

The Worker Path

AI systems rarely run only inside HTTP requests. Background jobs often run:

Batch inference
Evaluation pipelines
Data enrichment tasks

Maester includes a worker example to demonstrate that the same reliability primitives apply there. Worker execution uses the same tools:

Tracing
Cost metering
Evaluation
Structured logs

Reliability should not depend on the entrypoint.

What This Architecture Achieves

With only a few modules, the system now answers:

Question	Component
What request generated this output?	tracing
How long did the model call take?	spans
How many tokens were used?	cost meter
What did it cost?	pricing model
Was the output valid?	evaluator

These signals turn a black-box AI API into a traceable system.

Final Thought

Most reliability discussions around AI focus on models. But reliability often comes from system design, not model quality.

A simple architecture that records:

1. What happened
2. What it costs
3. Whether the result was acceptable

can dramatically improve how AI systems are operated.

The earlier these ideas are introduced into a system, the easier that system will be to maintain.

Note: This article was originally published on my engineering blog where I document the design of Maester, an AI SaaS infrastructure system built in public.
Original post: What Breaks After Your AI Demo Works.

[Boost]

Lei Ye — Thu, 05 Mar 2026 20:12:17 +0000

Introducing Maester

Lei Ye ・ Mar 5

#ai #saas #architecture #machinelearning

Introducing Maester

Lei Ye — Thu, 05 Mar 2026 20:03:16 +0000

The Knowledge Engine of Your Company

Most companies today want the same thing from AI: Turn their internal knowledge into something queryable, explainable, and operational.

In practice this means:

Documents scattered across tools
Institutional knowledge trapped in teams
Data that exists but cannot be used

And the typical solution becomes:

“Let’s build an AI assistant.”

But building an AI demo and building AI infrastructure that survives production are very different things.

Maester is our attempt to build the latter.

What Maester Is

Maester is a reference implementation of a B2B SaaS AI knowledge engine. It demonstrates how a company can transform internal data into a production-grade knowledge system.

At its core, Maester allows organizations to:

ingest internal documents
structure and embed them
retrieve relevant knowledge
generate responses with citations
trace every operation across the system

But more importantly, Maester is designed as infrastructure, not just an AI feature. That means we are focusing on:

reliability
traceability
operational cost control
asynchronous pipelines
multi-tenant architecture

We are building this project in public, both as a working system and as a learning artifact. Every design choice will be documented. Every architecture decision will be explained.

This blog serves as a system design journal

The Infrastructure Problem Most AI SaaS Products Ignore

When teams first add AI to a product, the initial version often works. A prototype connects an LLM, retrieves some documents, and produces answers. But once the system meets real users, things break quickly. We repeatedly see the same failure modes:

1. Timeouts

LLM calls are slow and unpredictable. Without proper timeouts and retries, requests cascade into system failures.

2. Uncontrolled costs

Every query triggers embedding calls, retrieval operations, and model inference. Without cost tracking and guardrails, usage grows faster than expected.

3. Queues and ingestion pipelines

Document ingestion is not instantaneous. Parsing, chunking, and embedding require asynchronous pipelines that many systems lack.

4. Traceability gaps

When something goes wrong, teams often cannot answer simple questions:

What document generated this answer?
Which embedding version was used?
Which model responded?

Without observability, AI becomes a black box in production.

But What “Production-ready AI Infrastructure” Actually Means

For us, production readiness is not about model quality. It is about system design. A production AI SaaS system must provide:

1. Asynchronous ingestion pipelines

Documents must move through structured stages:
parse → chunk → embed → index.
Each stage should be observable and retryable.

2. Reliable model access

All LLM access must go through a gateway that manages:

timeouts
retries
provider fallback

3. Usage and cost accounting

Every request must produce a usage record. Production systems must answer:

Which tenant generated this cost?
Which model generated this response?

4. Traceability

Requests must carry a correlation ID through:

API layer
worker queues
model calls

This is how production systems become debuggable.

How Maester is Structured

Instead of treating AI as a feature, we treat it as infrastructure. Maester separates the system into clear operational layers.

Architectural Layers

API Layer

Handles request entry, tenant routing, and request validation. This layer also generates request IDs used for tracing.

Knowledge Engine Core

This is where Maester’s core logic lives. Responsibilities include:

document retrieval
query orchestration
interaction with the model gateway
enforcing cost budgets

Async Worker System

All heavy processing moves to asynchronous workers:

document parsing
chunking
embedding
vector indexing

This prevents ingestion tasks from blocking user requests.

Model Gateway

Instead of calling models directly, all inference flows through a gateway. This gateway manages:

provider abstraction
retry logic
token usage tracking
future fallback support

Observability Layer

We treat observability as a first-class concern. Every request is traceable across:

API requests
worker jobs
model calls

This allows production debugging without guesswork.

Closing

Maester is not just an AI application.

It is an exploration of how AI systems should be engineered. In the coming posts, we will document:

architecture decisions
reliability patterns
cost control strategies
production ML infrastructure design

Our goal is simple: To build a knowledge engine that companies can trust in production. And to make every engineering decision transparent and explainable.

The system starts SMALL.

But the architecture is designed to SCALE.

Originally published on my engineering blog: https://lei-ye.dev/blog/introducing-maester