DEV Community: Armin Burger

Beyond the ReAct Loop: Why Your AI Agent Breaks in Production

Armin Burger — Fri, 09 Oct 2026 09:55:31 +0000

The ReAct (Reasoning and Acting) pattern is deceptively simple. In a tutorial, you define an agent, give it a few tools, and watch it solve complex queries. It looks magical. But moving that same loop into production reveals a harsh reality: the complexity isn’t in the reasoning logic itself, but in the infrastructure surrounding it. Most "agent failures" attributed to poor LLM reasoning are actually symptoms of inadequate tool design, missing cost controls, or security oversights.

Iteration Limits Are Cost Controls, Not Just Safety Nets

A common configuration mistake is treating max_iterations solely as a safeguard against infinite loops. While it prevents runaway processes, its primary function in production is cost management. Every iteration consumes tokens for both the input context and the generated output. Without strict limits derived from your specific cost model, a single user query can drain resources rapidly.

For example, setting max_iterations=10 in a LangChain AgentExecutor isn't just about stopping a stuck bot; it's about capping the financial exposure per request. You must also handle parsing errors gracefully. Instead of raising exceptions when the model outputs malformed JSON, feed the error back as an observation. This allows the agent to self-correct without terminating the session prematurely.

from langchain.agents import AgentExecutor

executor = AgentExecutor(
    agent=agent, 
    tools=tools, 
    max_iterations=10, 
    handle_parsing_errors=True
)

Tool Design Determines Quality More Than Prompts

There is a misconception that better prompts lead to better agents. In practice, agent quality depends heavily on well-scoped, clearly described tools. Ambiguous descriptions or poorly truncated results cause what appears to be model failure but is actually an interface problem.

Consider a web search tool. If it returns 50 raw HTML snippets, you flood the context window, increasing costs and confusing the model. You must enforce truncation at the tool boundary. A robust implementation might limit results explicitly:

from langchain.tools import Tool
import duckduckgo_search

def search_web(query: str):
    with ddgs() as d:
        return d.text(query, max_results=5) # Strict limit

web_search = Tool(
    name="WebSearch",
    func=search_web,
    description="Useful for searching current events. Returns top 5 results."
)

The description matters as much as the code. Vague instructions lead to hallucinated tool usage. Precise scoping ensures the model knows exactly when and how to use the tool.

Security: Treat Tool Results as Untrusted Input

This is often overlooked. In a standard chatbot, user input is untrusted. In an agent system, tool outputs are also untrusted. If your agent fetches data from a public API or scrapes a website, that content could contain prompt injection attacks. Malicious text embedded in a search result could instruct the agent to execute dangerous commands or leak data. Because agents have side effects (executing code, sending emails), these injections are far more dangerous than in static chats.

Streaming Requires Intermediate Events

Finally, user experience suffers if you only stream the final answer. An agent takes time to think, select tools, and process results. To maintain engagement and provide transparency, you must stream intermediate step events—such as "thinking," "tool selection," and "execution status"—rather than just waiting for the final token. This requires a custom streaming protocol that exposes the internal state of the ReAct loop.

Production agents aren't built by tweaking prompts. They are engineered through rigorous cost modeling, secure tool boundaries, and transparent execution flows.

Repo: github.com/armbur19-collab/chimerai-app

ChimerAI Blog

Stop Outsourcing Your Agent’s Brain: Why DIY Often Beats Frameworks in Production

Armin Burger — Tue, 06 Oct 2026 16:46:50 +0000

The debate around AI agent architecture often frames the choice between using a hosted platform, adopting an open-source framework, or building from scratch as a question of engineering maturity. This is a misconception. The decision is strictly a matter of fit, determined by two critical factors: the complexity of your control flow and who owns the failure modes when things inevitably break.

When to Use a Platform

Hosted platforms are excellent for specific scenarios, primarily when the agent is a secondary feature within a larger product rather than the core value proposition itself. If you are adding a chatbot to a SaaS dashboard where the primary business logic resides elsewhere, outsourcing the agent infrastructure makes sense. However, this convenience comes with a significant trade-off: moving agent operations to a platform shifts core behavior outside your codebase. This limits your ability to locally reproduce failures and patch issues independently. You become dependent on third-party release schedules for fixes that affect your user experience.

When to Use a Framework

Frameworks occupy the middle ground. They are optimal when your agent’s logic aligns with standard multi-step reasoning patterns and common tool interfaces. These tools handle the tedious plumbing, such as the tool-calling protocol, message history formatting, retry logic for parse errors, and streaming interfaces. For many developers, this abstraction saves time. But beware: working around a framework’s abstraction can sometimes be harder than implementing the functionality without it. If your requirements deviate from the "happy path" supported by the framework, you may find yourself fighting the library rather than leveraging it.

The Case for Building It Yourself (DIY)

Building an agent loop from scratch is justified when specific requirements make framework abstractions obstructive. A prime example is exact cost accounting or unusual control flow logic. A DIY implementation typically involves calling provider APIs directly within a simple while loop with a clear stop condition. While this requires more initial effort, it grants complete visibility into every step of the execution.

If the agent is your product, its loop, error handling, and cost model constitute core competencies that should not be outsourced. Relying on external libraries means you cannot fully control how prompt injection vulnerabilities are handled or how tool results are sanitized before being returned to the model context.

The Non-Negotiable: Cost and Observability

Regardless of the strategy chosen—platform, framework, or DIY—two technical realities remain mandatory. First, unbounded agent execution equates to unbounded financial liability. Every agent loop must have an iteration limit derived from a strict cost model to prevent runaway spending. Second, debugging capability is more critical than feature richness. Observability requires tracing every tool call, argument, and result. If a solution does not provide full trace visibility, it is fundamentally flawed for production use because you cannot diagnose why an agent failed.

Conclusion

Choose your architecture based on ownership of failure modes. If you need independent patching capabilities and precise cost controls, build it yourself. If you are shipping a minor feature quickly, use a platform. Only choose a framework if your logic perfectly matches its abstractions. Don’t let convenience compromise your ability to debug your own product.

Repo: github.com/armbur19-collab/chimerai-app

ChimerAI Blog

Your prompts are code. Manage them like data — but know what you lose when you do

Armin Burger — Wed, 30 Sep 2026 11:43:16 +0000

The three things that separate an AI demo from an AI product aren't models. They're: being able
to change a prompt without a deploy, knowing what each request cost you, and not leaking a
user's credit card into logs. None of these are exciting. All three are in the ChimerAI stack,
so here's what they do and — more usefully — the tradeoffs they hide.

1. Prompt templates in the database

Instead of prompt strings scattered across .py files, there's a PromptTemplate model with
Mustache-style variables:

model PromptTemplate {
  id        String   @id @default(cuid())
  name      String   @unique
  category  String   // "system" | "user" | "rag" | "agent" | "chat" | "custom"
  content   String   @db.Text
  variables String[] // extracted from {{placeholders}} at save time
  language  String   @default("en")
  version   Int      @default(1)
  isDefault Boolean  @default(false)
  isActive  Boolean  @default(true)
  tags      String[]
}

The variables array is derived from the content, so the app can validate that a caller
supplied every {{context}} / {{query}} before rendering — a missing variable becomes a
400 at the API boundary, not a prompt that silently ships a literal {{context}} to the model.

POST /api/prompts
{ "name": "RAG System Prompt",
  "category": "rag",
  "content": "Use {{context}} to answer {{query}}",
  "isDefault": true }

Exactly one default per category, multi-language (EN/DE/FR/ES/IT), and the whole thing is
behind an manage_prompts permission so non-admins can't edit production prompts.

The tradeoff nobody tells you about: the moment a prompt lives in the DB instead of git, you
lose your diff, your code review, and your "what shipped on Tuesday" answer. Versioning here is
an Int counter, not a history — it tells you this is v7, not what v5 said. If prompts
matter to you (and for a RAG system they do), either export template changes to your audit log
or keep the canonical copy in git and treat the DB as a runtime override. I lean toward the
latter and haven't fully committed to either, which is honest.

2. Per-request cost tracking

Every chat call runs through trackApiUsage, which records the endpoint, model, token counts,
success/failure, and status code. The provider's own token counts are used where available; the
streaming path accumulates them from chunks and reports once at the end:

if provider_id and user_id and (total_prompt_tokens or total_completion_tokens):
    await provider_client.report_usage(
        provider_id=provider_id, user_id=user_id, model=request.model,
        prompt_tokens=total_prompt_tokens, completion_tokens=total_completion_tokens,
        endpoint="/api/chat/stream",
    )

This is what lets the credit check in the request path exist at all, and what turns "our AI
bill doubled" from a mystery into a query. The model is chosen at runtime and can be swapped
per provider without a deploy, so cost is tracked against the model that actually answered, not
a config constant.

Caveat: the pre-request budget check uses a character-based token estimate
(chars / 4 × 1.2), not the real tokenizer. It's a soft gate to stop runaway spend, not a
billing system. The recorded usage is the accurate number; the estimate is a heuristic that
will occasionally reject a request slightly early. Don't invoice off the estimate.

3. Guardrails: moderation, PII, injection

chimerai add guardrails gives you four endpoints. The PII detector is regex-based — which is
both its strength (fast, no network, deterministic) and its ceiling (it catches patterns, not
meaning):

"email":       re.compile(r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b'),
"phone":       re.compile(r'\b\d{3}[-.]?\d{3}[-.]?\d{4}\b'),
"ssn":         re.compile(r'\b\d{3}-\d{2}-\d{4}\b'),
"credit_card": re.compile(r'\b\d{4}[- ]?\d{4}[- ]?\d{4}[- ]?\d{4}\b'),

Endpoints:

Method	Path	What
POST	`/api/guardrails/pii/detect`	find email/phone/SSN/CC/IP
POST	`/api/guardrails/pii/redact`	mask the above
POST	`/api/guardrails/toxicity`	keyword score 0–1 + risk level
POST	`/api/guardrails/injection`	prompt-injection patterns

Toxicity and injection detection are keyword/pattern-based with a confidence score and a risk
classification. That's enough to catch the obvious stuff and to add a cheap first line of
defense before a request hits your paid model.

Where it stops being sufficient: US phone/SSN/CC formats only (a German project will want
different patterns — worth noting since this kit is partly German-authored); "toxicity" by
keyword misses rephrased hostility entirely; regex injection detection is a tripwire, not a
classifier. If compliance demands real content moderation, put a dedicated model in front and
keep these as a fast pre-filter. detect vs redact as separate calls is deliberate — you
often want to log what was found even when you forward a masked version.

How they fit together

The RAG pipeline in the same stack reads its system prompt from category rag (feature 1),
runs the user query through guardrails before embedding (feature 3), and reports token usage
after the LLM call (feature 2). Three boring features, one grounded-and-metered request.

None of this is the part that demos well. It's the part that lets you sleep once users find
your app.

Repo: github.com/armbur19-collab/chimerai-app

ChimerAI Blog

A RAG system in one command — what chimerai add rag actually wires up

Armin Burger — Tue, 29 Sep 2026 12:24:15 +0000

RAG is one of those things where the tutorial is 30 lines and the production system is a
vector-database procurement process. This post is about the middle ground: a working retrieval
pipeline you can drop into a project in one command, what it actually installs, and the exact
point where you'd want to replace it.

The command

cd my-nextjs-app
npx chimerai add rag

rag depends on the chat module, so if ai-chat isn't installed the CLI adds it first. What
you get is a self-contained Python AI service (FastAPI + LiteLLM) under services/ai/ with the
RAG module switched on:

services/ai/
├── config.py                    # pydantic settings, env loading
├── provider_client.py           # LiteLLM — multi-provider routing
├── main.py                      # FastAPI entry (generated from manifest)
├── services/
│   ├── rag_service.py           # ingest → retrieve → answer pipeline
│   ├── vector_store.py          # FAISS index + persistence
│   └── embedding_service.py     # text embeddings via LiteLLM
├── routes/rag_routes.py         # /api/rag/upload, /query, /stats
└── data/                        # FAISS index on disk

The Next.js side gets proxy routes that forward to AI_SERVICE_URL (default
http://localhost:8002), so your frontend calls /api/rag/... and never talks to Python
directly. chimerai dev starts both.

The pipeline, in the order it runs

Ingest — you hand it raw text, it chunks and embeds. Chunking is
RecursiveCharacterTextSplitter with defaults you can see rather than guess:

self.text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200,
    length_function=len,
    separators=["\n\n", "\n", ". ", " ", ""],
)

chunk_size=1000 is in characters, not tokens (length_function=len) — a detail that
matters if you're used to token-based splitters; 1000 chars is roughly 250 tokens of English.
The 200-char overlap is there so a fact split across a boundary still shows up whole in at
least one chunk. Each chunk keeps a pointer back to its source:

for j, chunk in enumerate(chunks):
    all_chunks.append(chunk)
    all_metadatas.append({
        **meta,
        "_chunk_index": j,
        "_chunk_total": len(chunks),
        "_source_doc_index": i,
    })

Those _chunk_* fields are what let you show "source: page 3" in the UI instead of a bare
similarity score.

Store — a faiss.IndexFlatL2 on 1536-dim vectors (OpenAI text-embedding-ada-002). Flat
L2 means exact brute-force search: no approximation, so recall is perfect and the index just
scans everything. That's fine to tens of thousands of vectors and gets slow past that — the
code comment says as much:

# Use IndexFlatL2 for exact search (can be changed to IndexIVFFlat for large datasets)
self.index = faiss.IndexFlatL2(self.dimension)

Retrieve + answer — the RAG chat is a plain "stuff context into the system prompt" flow:

relevant_docs = await vector_store.similarity_search(query, k=k)

context_parts = []
for i, doc in enumerate(relevant_docs, 1):
    context_parts.append(f"[Document {i}]\n{doc['text']}")
context = "\n\n".join(context_parts) if context_parts else "No relevant documents found."

system_message = f"""You are a helpful assistant. Use the following context to answer \
the user's question. If the context doesn't contain relevant information, say so and \
provide a general answer.

Context:
{context}"""

The [Document i] labels exist so the model can cite, and the "if the context doesn't contain
the answer, say so" line is the single most important prompt sentence in the whole file —
without it the model answers from parametric memory and your eval numbers are fiction.

Using it

Add documents:

POST http://localhost:8002/api/rag/documents
{ "documents": ["FastAPI is a modern web framework for Python."],
  "metadatas": [{ "source": "docs", "page": 1 }] }

Ask a grounded question:

POST http://localhost:8002/api/rag/chat
{ "query": "Tell me about FastAPI", "model": "gpt-3.5-turbo", "k": 3 }

The response is a normal OpenAI-shaped completion plus a rag_metadata block that lists the
retrieved chunks and their scores — so you can render citations and debug "why did it answer
that" without a separate search call:

"rag_metadata": {
  "retrieved_documents": 3,
  "documents": [{ "text": "FastAPI is a modern web framework...",
                  "score": 0.123, "metadata": { "source": "docs" } }]
}

There's also /api/rag/search (just retrieval, no LLM), /api/rag/stats, and
/api/rag/clear.

The FAISS import is guarded — and that's the point

On Python 3.13 / some Windows setups, faiss-cpu or numpy won't install cleanly. Rather than
crash the whole service at import time, the module degrades:

try:
    import faiss
    import numpy as np
    FAISS_AVAILABLE = True
except Exception as e:
    FAISS_AVAILABLE = False
    print(f"Warning: FAISS/Numpy not available: {e}")

Every RAG entry point calls _check_availability() first and raises a clear
"FAISS vector store is not available" instead of an ImportError three frames deep. Chat,
guardrails, and tools keep working; only retrieval is off. Small design choice, but it's the
difference between "my RAG doesn't work" and "my whole app won't boot."

Persistence

The index is saved to data/faiss_index and metadata to a matching .pkl — loaded on startup,
written after ingest. It's a single-process, on-disk store with no locking. Which is exactly the
right scope for a starter and exactly the wrong scope for multi-instance:

When you outgrow it

Honest boundaries, so you know what you're buying:

Flat index → fine to ~tens of thousands of chunks, then switch to IndexIVFFlat/HNSW or a real vector DB (pgvector, Qdrant, Weaviate).
One global index → no per-tenant / per-user namespaces. Multi-tenant retrieval needs metadata filtering you'd add yourself, or a DB that does it.
Local disk, single writer → no horizontal scaling; two instances = two divergent indexes.
k is a fixed int → no hybrid (BM25 + dense) search, no re-ranking, no MMR diversification. Those are the usual next levers when retrieval quality plateaus.

None of that is a criticism of a scaffold — it's the list of what a "RAG in one command" is
not, so you replace the right thing at the right time instead of rewriting from scratch.

Try it

chimerai create my-rag-app --sqlite --yes
cd my-rag-app
chimerai add rag
chimerai dev

--sqlite skips Docker for the app DB; the AI service still runs as its own Python process.

Repo: github.com/armbur19-collab/chimerai-kickstart

ChimerAI Blog

Streaming an AI chat from Python FastAPI through a Next.js proxy: the four annoying details

Armin Burger — Mon, 28 Sep 2026 14:03:53 +0000

Streaming chat looks trivial in tutorials: for chunk in response: print(chunk.text). It stops
looking trivial the moment you add the things a real product needs — a logged-in user, a
provider that can be swapped at runtime, a cost record per request, and an error that happens
after you already sent 200 OK.

Here's the path a message takes in the ChimerAI stack, with the parts that cost me time.

The shape

ChatWindow (React)
  → POST /api/v1/chat/stream        Next.js API route, TypeScript
      auth + credits + persist user message
  → POST /api/chat/stream           FastAPI, Python + LiteLLM
      async generator, yields SSE chunks
  ← data: {...}\n\n  …  data: [DONE]\n\n

Two layers, because the LLM ecosystem is Python and the UI is TypeScript. The Next.js route is
not a pass-through — it's where auth, persistence, and metering happen, so the Python service
stays a stateless completion endpoint you could put behind any other frontend.

1. The frontend is a component, not a page

import { ChatWindow } from '@chimerai/chat-ui';

export default function ChatPage() {
  return (
    <ChatWindow
      apiEndpoint="/api/v1/chat/stream"
      placeholder="Ask me anything..."
      showModelSelector={true}
      models={['gpt-4', 'claude-3', 'ollama/llama3']}
    />
  );
}

The component handles markdown rendering, code blocks with a copy button, auto-scroll, and the
streaming reader. If you're embedding into someone else's site rather than shipping your own
app, there's a separate chat-widget — a self-contained Web Component with Shadow DOM, mounted
per API key:

<script src="https://your-app.com/widget/chat.js"></script>
<div id="chat"></div>
<script>ChimerAI.mount('#chat', { apiKey: 'sk_live_...', theme: 'auto' });</script>

2. Check what you can't verify: estimate tokens before the call

You don't know the completion length before the model generates it, so a credit check can't be
exact. The route estimates from the prompt and treats it as a precondition, then reconciles
afterwards:

function estimateTokens(messages: any[]): number {
  if (!messages || !Array.isArray(messages)) return 500;
  const totalChars = messages.reduce((sum, msg) => sum + (msg.content?.length || 0), 0);
  return Math.ceil((totalChars / 4) * 1.2);
}

const estimatedTokens = estimateTokens(payload.messages);
const creditResult = await requireCredits(request, estimatedTokens);
if (!creditResult.authorized) {
  return new Response(JSON.stringify(createErrorResponse(creditResult)), { status: 402 });
}

chars / 4 is the usual rough tokenizer rule; the 1.2 is headroom. It's a heuristic and it
fails in both directions — a short prompt with a long completion overshoots the balance, a
user with exactly enough credits can get cut off mid-stream. That's acceptable for a soft
limit; it is not acceptable as billing. Actual usage is recorded from the provider's own
token counts at the end of the stream, and that's the number that goes into ApiUsage.

Note the ordering: permission check → credit check → then create the conversation row and
persist the user message. Doing the DB writes first means a rejected request still costs you a
conversation with an orphan message in it.

3. Errors after `200 OK` must travel inside the stream

This is the one that bites everyone. StreamingResponse sends headers immediately, so a Python
try/except around the generator body is useless — by the time the provider call fails, you've
already committed to a 200:

@router.post("/chat/stream")
async def create_chat_completion_stream(request: ChatCompletionRequest):
    # No try/except here — StreamingResponse returns 200 immediately,
    # the generator runs lazily. Errors are handled inside the generator
    # as SSE error events.
    return StreamingResponse(
        chat_service.create_streaming_completion(request, provider_id=..., user_id=...),
        media_type="text/event-stream",
    )

So the error handling lives inside the generator and speaks the protocol:

        except Exception as e:
            logger.error("streaming_completion_error", error=str(e))
            # Yield error as SSE event so the client receives it
            # (raising here would be swallowed — 200 OK was already sent)
            error_payload = json.dumps({"error": {"message": str(e), "type": "server_error"}})
            yield f"data: {error_payload}\n\n"
            yield "data: [DONE]\n\n"

The client therefore has to handle an error key in the middle of an otherwise healthy stream,
and still terminate on [DONE]. If your reader assumes "chunks only contain content", a
provider outage renders as a silently truncated answer instead of an error message — which is
the worst possible failure mode, because it looks like the model just stopped talking.

4. Normalize chunks into your own schema

The service doesn't forward provider chunks verbatim; it rebuilds each one:

            async for chunk in response:
                if not chunk.choices:
                    continue
                choice = chunk.choices[0]

                if hasattr(chunk, "usage") and chunk.usage:
                    total_prompt_tokens = getattr(chunk.usage, "prompt_tokens", 0) or 0
                    total_completion_tokens = getattr(chunk.usage, "completion_tokens", 0) or 0

                chunk_response = ChatCompletionChunk(
                    id=chunk.id or f"chatcmpl-{uuid.uuid4().hex[:8]}",
                    object="chat.completion.chunk",
                    created=int(time.time()),
                    model=chunk.model,
                    choices=[ChatCompletionChunkChoice(
                        index=choice.index,
                        delta=DeltaMessage(
                            role=getattr(choice.delta, "role", None),
                            content=getattr(choice.delta, "content", None),
                        ),
                        finish_reason=choice.finish_reason,
                    )],
                )
                yield f"data: {chunk_response.model_dump_json()}\n\n"

Two reasons this indirection is worth the boilerplate:

if not chunk.choices: continue — providers emit usage-only or role-only frames. Anthropic and Ollama don't frame deltas the way OpenAI does; LiteLLM gets you most of the way, and these edge cases are the rest.
id=chunk.id or f"chatcmpl-{uuid4().hex[:8]}" — some providers omit the id on chunks. The client uses it to correlate, so a missing id becomes a dropped message.

Usage is accumulated from whichever chunks carry it and reported once, after the stream closes:

            if provider_id and user_id and (total_prompt_tokens or total_completion_tokens):
                await provider_client.report_usage(
                    provider_id=provider_id, user_id=user_id, model=request.model,
                    prompt_tokens=total_prompt_tokens,
                    completion_tokens=total_completion_tokens,
                    endpoint="/api/chat/stream",
                )

Swapping providers without touching the chat code

The model isn't hardcoded in the route. The request may omit it, in which case it falls back to
the active provider's config.defaultModel:

let model = payload.model; // Optional — resolved after provider loading
// Model validation is deferred until after provider loading,
// so we can fall back to provider.config.defaultModel

That's why the permission check has a deferred branch (requireModelPermission(request, '__deferred__')) — you can't validate a model you haven't resolved yet. Providers are DB rows
with AES-256-encrypted keys, so adding Claude or a local Ollama instance is an admin-UI action,
not a deploy:

await fetch('/api/providers', {
  method: 'POST',
  body: JSON.stringify({ name: 'Local Llama', provider: 'ollama',
                         baseUrl: 'http://localhost:11434', isActive: true }),
});

What I'd do differently

The token estimator is the weakest link — it's a guess in the request path. Cleaner would be a
queue/worker split: accept the request, enqueue it, stream from the worker, and enforce budget
per user per window instead of per request. I haven't done that because at current traffic the
heuristic's failure mode (a request rejected slightly early) is cheaper than the complexity.

Also: 1.2 is a magic number I chose. If you have real prompt/completion distributions,
replace it with a p95 multiplier from your own logs.

Repo: github.com/armbur19-collab/chimerai-kickstart

ChimerAI Blog

I built a CLI that scaffolds the boring parts of an AI SaaS — here's what it actually generates

Armin Burger — Thu, 24 Sep 2026 09:56:07 +0000

Every AI SaaS project starts with the same two days of nothing-interesting: auth, a users
table, an encrypted place to store provider API keys, a Prisma schema. Then you finally get to
the part you actually wanted to build.

I've been working on a CLI (@chimerai/cli) that scaffolds exactly that prefix. This post is
about what it actually writes to disk — not what a landing page would claim — because that's
the only thing that matters when you decide whether to adopt a generator.

The one command

npx @chimerai/cli create my-ai-app

Interactive feature selector. Defaults are auth + RBAC + admin dashboard + analytics; AI
features are opt-in. Flags worth knowing:

chimerai create my-ai-app --yes              # no prompts, defaults
chimerai create my-ai-app --sqlite           # no Docker needed, DATABASE_URL=file:./dev.db
chimerai create my-ai-app --yes --install    # + npm install

--sqlite is the one I'd start with if you just want to look at the code. Without it you get
a docker-compose.yml for PostgreSQL 16 + Redis 7.

What's in the output

my-ai-app/
├── app/
│   ├── layout.tsx
│   ├── page.tsx
│   ├── api/auth/          # NextAuth routes (if auth selected)
│   └── admin/             # admin pages (if selected)
├── components/ui/         # shadcn/ui components
├── lib/
│   ├── prisma.ts          # PrismaClient singleton
│   ├── auth.ts            # auth config
│   └── encryption.ts      # API key encryption
├── prisma/schema.prisma   # only models for selected features
├── .env
├── docker-compose.yml
└── package.json

The generated Prisma schema is additive per feature, which is the part I check first in any
scaffolding tool — if it gives me six models when I asked for two, I stop trusting it:

Selected	Models added
Auth	`User`, `Account`, `Session`, `VerificationToken`
RBAC	`Role`, `Permission`
Providers	`ModelProvider`
Prompts	`PromptTemplate`
Analytics	`ApiUsage`

Feature dependencies are resolved by the CLI rather than left to you: admin-dashboard
requires RBAC, model-providers requires auth (the encryption key lives in the auth config),
chat-ui requires model-providers, rag requires model-providers.

Stack, stated plainly

Next.js 14/15 App Router, API routes serve as the backend — no separate Node service
TypeScript everywhere except the AI service, which is Python (FastAPI + LiteLLM)
Prisma on PostgreSQL, NextAuth for credentials + OAuth
Tailwind + shadcn/ui

The split is deliberate: the LLM orchestration layer is Python because that's where the
ecosystem lives (LangChain, FAISS, spaCy), and everything type-safe and user-facing stays
TypeScript. The Next.js side proxies to AI_SERVICE_URL (default http://localhost:8002).

The bit I'd point at as "not boilerplate"

Provider management with encrypted keys, which is the thing everyone re-implements badly:

const response = await fetch('/api/providers', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({
    name: 'OpenAI GPT-4',
    provider: 'openai',      // | 'anthropic' | 'ollama' | 'custom'
    apiKey: 'sk-...',
    isActive: true,
  }),
});

Keys are stored AES-256 encrypted at rest using ENCRYPTION_KEY from .env, never in the
database in plaintext. provider: 'custom' takes any OpenAI-compatible baseURL, which is how
you wire up a self-hosted vLLM or an OpenRouter-style gateway without the core knowing about it.
You can also add providers through the admin UI and test the connection before using it — small
thing, saves an annoying debugging loop.

RBAC uses resource:action permission strings, checked in API routes:

import { requirePermission } from '@/lib/auth/require-permission';

export async function GET(req: NextRequest) {
  const permissionError = await requirePermission('posts:read');
  if (permissionError) return permissionError;
  // ...
}

admin:* is a wildcard. Roles carry permission lists; there's a seeded admin
(admin@example.com / admin123 — remove it, obviously).

Honest limitations

Packages aren't on npm yet. @chimerai/model-providers, @chimerai/admin-ui etc. exist in the monorepo. chimerai create therefore generates standalone projects with the selected features inlined as code templates — you get readable code you own, not a dependency you can't inspect. For the full monorepo with shared packages you'd clone the repo instead.
chimerai add requires Next.js 15+ (generated route handlers use the async params API). The CLI detects the version and warns below 15. Standalone create output runs on port 3001 by default to avoid colliding with another dev server.
The feature set is opinionated toward the SaaS shape (auth, providers, prompts, chat, RAG, admin). If your app doesn't need that shape, a plain create-next-app plus a library like LangChain is a perfectly reasonable starting point — pick the generator that matches the problem.

Adding to an existing project instead

You don't have to start from a generated project:

cd my-nextjs-app
npx chimerai add auth
npx chimerai add model-providers
npx chimerai add chat-ui
pnpm install
npx prisma db push && npx prisma db seed
pnpm dev

The CLI finds the project root by walking up for a .chimerai marker, then falls back to a
registry in ~/.chimerai/projects.json, then --dir. In a monorepo, point --dir at the
Next.js app (apps/frontend), not the workspace root — the root has no app/ and no prisma/.

chimerai doctor runs health checks on env vars, DB connectivity, and installed components,
which is mostly there because I forgot to set ENCRYPTION_KEY often enough to be embarrassing.

Next in this series

Streaming chat: SSE from FastAPI through a Next.js proxy route
RAG with chimerai add rag — FAISS index, chunking, and the /api/rag/query round trip
Prompt templates in the DB instead of in git, and what breaks when you do that

Repo: github.com/armbur19-collab/chimerai-kickstart · site: chimerai.dev

ChimerAI Blog

Stop Counting Requests: The Case for Token-Based Quotas in LLM SaaS

Armin Burger — Tue, 22 Sep 2026 11:42:23 +0000

If you are building a multi-tenant AI application, your current rate-limiting strategy is likely broken. Traditional "requests per minute" (RPM) metrics, inherited from standard REST APIs, fail catastrophically when applied to Large Language Models. This isn't just an imprecision; it is a fundamental architectural error that systematically throttles efficient short queries while allowing high-cost massive contexts to bypass limits undetected.

Why RPM Metrics Fail for LLMs

The core issue is cost variance. In traditional APIs, the computational cost of a request is relatively uniform. In LLMs, cost is driven by token count, not frequency. A single request with a 100k-token context window can cost hundreds of times more than a simple 50-token query. If you limit users by request count, you inadvertently penalize lightweight users while giving heavy users free rein to drain resources until the provider’s account-level quota is exhausted.

This leads to the "noisy neighbor" effect. Most LLM providers enforce rate limits at the organization or account level, not per user. Without internal controls, one tenant’s batch processing spike can throttle API access for every other customer on your platform, causing widespread service degradation.

Tokens ≠ Costs: The Complexity Layer

Even shifting to token counts introduces inaccuracies because tokens do not map linearly to dollars. Three factors complicate this relationship:

Input vs. Output Pricing: Output tokens are often 3 to 5 times more expensive than input tokens.
Model Variance: Tenants using premium models can incur 3 to 4 times the costs of those using cheaper models, even with identical usage patterns.
Prompt Caching: Repeated context blocks may be charged at a fraction of the regular price upon cache hits, meaning two requests with the same token count can have vastly different costs.

Therefore, financial control requires managing token credits rather than raw token counts. You must normalize costs across different models and pricing tiers into a common currency for your internal ledger.

The Reservation Pattern: Solving the Temporal Problem

A critical challenge in LLM billing is temporal uncertainty. You only know the exact cost after generation completes. If you wait for completion to charge, a user could trigger a long-running stream that exceeds their budget, resulting in negative balances.

The solution is a reservation pattern, similar to credit card pre-authorizations. Before calling the provider, your system should:

Estimate the maximum possible cost using max_tokens as a conservative upper bound.
Reserve these funds in an internal credit ledger (creditLedger.reserve).
Execute the API call.
Settle the exact cost based on the provider’s actual usage report, releasing any excess reservation.

If the call fails, the reservation is released immediately. This ensures that no tenant can spend more than they have authorized, even during streaming processes.

Implementation Strategy: Two-Tier Limits

To maintain stability, implement a two-tier limiting structure:

Internal Per-Tenant Sub-Limits: Use a token bucket algorithm tied to the user’s credit balance to prevent individual overspending.
Global Provider Quota Checks: Monitor the aggregate usage against the provider’s account limits to prevent the noisy neighbor effect.

Crucially, always base final cost tracking on the actual usage data returned in the provider’s API response. Internal estimates are useful for reservations, but discrepancies between your bookkeeping and provider invoices signal bugs in your cost logic, not mere estimation errors. Real-time monitoring of usage fidelity is essential for accurate billing.

By shifting from request counting to token-based credit management with reservation patterns, you align your architecture with the economic reality of LLMs, ensuring both financial accuracy and service reliability for all tenants.

Repo: github.com/armbur19-collab/chimerai-app

ChimerAI Blog

Beyond `tenant_id`: Why Classical Multi-Tenancy Fails for RAG Systems

Armin Burger — Fri, 18 Sep 2026 12:09:46 +0000

Most engineering teams assume that because they have implemented row-level security (RLS) and a tenant_id column in their relational database, their application is securely multi-tenant. This assumption holds true for traditional CRUD applications. However, when you integrate Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) pipelines, classical isolation patterns become dangerously insufficient.

The core issue is that AI systems introduce new attack surfaces and failure modes that do not exist in standard SaaS architectures. Here are four critical frontiers where traditional multi-tenancy breaks down, along with the architectural adjustments required to secure them.

1. Vector Store Isolation: The Silent Leak Risk

Approximate Nearest Neighbor (ANN) indexes, which power most vector databases, lack default access controls comparable to SQL. If you rely solely on post-filtering results by tenant_id, you risk exposing data during the search phase or leaking metadata through similarity scores.

Solution: You must enforce isolation at the index level. Use namespace models or explicit pre-filtering strategies within the vector store itself. Treating the tenant boundary as an absolute filter component—rather than just a metadata tag—is essential to prevent cross-tenant data leaks.

2. Semantic Caching: Keys Must Include Tenant ID

Caching is often the most overlooked frontier for security breaches in AI apps. Semantic caches retrieve answers based on question similarity rather than exact matches. If your cache key does not include the tenant_id as a hard component, a query from Tenant A might match a cached response generated for Tenant B.

This leads to silent failures: users receive plausible-sounding but incorrect answers derived from another company’s data. There are no error codes or crashes; the system simply returns wrong information.

Solution: Ensure the tenant ID is a mandatory part of every semantic cache key. Maintain per-tenant cache spaces instead of using a single global cache where tenant identity is treated merely as a similarity signal.

3. Token Cost Management: Beyond Request Rate Limiting

Traditional rate limiting counts requests per minute. In RAG products, this is inadequate because token costs vary significantly by provider, model, and direction (input vs. output). A "noisy neighbor" can consume disproportionate resources with a few complex prompts, impacting latency and cost for all other tenants.

Solution: Shift from request counting to token usage tracking. Implement reservation patterns to manage resource allocation effectively, ensuring that one tenant’s heavy usage does not degrade service quality for others.

4. Context Assembly and Guardrails

Hardcoding global middleware for guardrails fails when different tenants have varying compliance needs and latency tolerances. Furthermore, concurrency issues in prompt assembly can lead to catastrophic mixing of sensitive data if module-global state is used.

Solution:

Dynamic Guardrails: Configure PII filters and prompt-injection defenses per tenant.
Strict Scoping: Avoid object sharing between parallel requests. Place base system instructions before the cache boundary, and keep all tenant-specific context (names, configs, RAG data) strictly after it.
Granular Observability: Global average metrics mask individual tenant failures. Measure evaluation metrics like RAGAS faithfulness at the tenant granularity to detect subtle quality degradation early.

Conclusion

Security architecture for AI requires moving beyond database-level isolation. It must encompass vector indices, prompt construction logic, and API-level caching mechanisms. Thinking through each dimension individually during design beats bundling them under a generic "multi-tenancy" label, because failure modes in AI systems are silent and subtle, not obvious crashes.

Repo: github.com/armbur19-collab/chimerai-app

ChimerAI Blog

Stop Hardcoding Your AI Guardrails: A Multi-Tenant Configuration Strategy

Armin Burger — Tue, 15 Sep 2026 15:36:15 +0000

Most early-stage AI products make a critical architectural mistake: they implement security guardrails as global, hardcoded middleware. While this works for a single use case, it becomes a liability the moment you onboard a second enterprise customer with different regulatory needs.

The core problem is that "one-size-fits-all" security fails in multi-tenant environments. A healthcare tenant requires strict PII detection and high latency tolerance, while a developer tools tenant prioritizes low-latency responses and minimal false positives. When these conflicting requirements meet a static codebase, you face risky rewrites instead of simple config changes.

The Pipeline Model

Instead of treating guardrails as binary on/off feature flags, model them as an ordered, parameterized pipeline. This allows each tenant to define their own sequence of checks for both input and output streams.

A robust configuration structure looks like this:

{
  "input": [
    {
      "type": "pii_detection",
      "sensitivity": "high",
      "action": "redact"
    },
    {
      "type": "toxicity_check",
      "sensitivity": "medium",
      "action": "flag"
    }
  ],
  "output": [
    {
      "type": "harmful_content",
      "sensitivity": "low",
      "action": "block"
    }
  ]
}

By loading this JSON from a database per tenant, you enable adjustments without code deployments. Actions like block, flag, or redact provide granular control over how violations are handled.

Handling Streaming Output

Input guardrails are relatively straightforward because you can analyze the full prompt before processing. Output guardrails present a unique architectural conflict: real-time streaming UX versus holistic text analysis.

Waiting for the entire response to finish defeats the purpose of streaming, but checking every token individually misses context-dependent toxicity. The solution is a middle-ground approach: process output in sentences or chunks. If a chunk triggers a rule, abort the stream immediately. This balances user experience with security compliance.

Audit Trails and Compliance

Guardrails are not just blockers; they are evidence generators. Without granular audit logs containing confidence scores and specific triggered rules, you cannot prove compliance to auditors—you can only claim it exists.

Implementing tenant-specific "golden sets" for regression testing ensures that configuration changes do not inadvertently break security policies. Furthermore, sensitivity thresholds should be treated as business risk decisions configurable by the tenant, not fixed by the platform.

The Two-Tier Configuration Model

To balance usability with legal safety, adopt a two-tier model:

Coarse Presets: Allow self-service tenants to choose from predefined security profiles (e.g., "Standard," "Strict").
Granular Overrides: Enable Enterprise customers to tweak specific parameters via approval workflows.

Hardcoding guardrails globally is not just bad practice; it is a guaranteed path to losing enterprise contracts. By shifting to a configurable, per-tenant pipeline, you transform security from a rigid constraint into a flexible product feature.

DEV Community: Armin Burger

Beyond the ReAct Loop: Why Your AI Agent Breaks in Production

Iteration Limits Are Cost Controls, Not Just Safety Nets

Tool Design Determines Quality More Than Prompts

Security: Treat Tool Results as Untrusted Input

Streaming Requires Intermediate Events

Stop Outsourcing Your Agent’s Brain: Why DIY Often Beats Frameworks in Production

When to Use a Platform

When to Use a Framework

The Case for Building It Yourself (DIY)

The Non-Negotiable: Cost and Observability

Conclusion

Your prompts are code. Manage them like data — but know what you lose when you do

1. Prompt templates in the database

2. Per-request cost tracking

3. Guardrails: moderation, PII, injection

How they fit together

A RAG system in one command — what chimerai add rag actually wires up

The command

The pipeline, in the order it runs

Using it

The FAISS import is guarded — and that's the point

Persistence

When you outgrow it

Try it

Streaming an AI chat from Python FastAPI through a Next.js proxy: the four annoying details

The shape

1. The frontend is a component, not a page

2. Check what you can't verify: estimate tokens before the call

3. Errors after 200 OK must travel inside the stream

4. Normalize chunks into your own schema

Swapping providers without touching the chat code

What I'd do differently

I built a CLI that scaffolds the boring parts of an AI SaaS — here's what it actually generates

The one command

What's in the output

Stack, stated plainly

The bit I'd point at as "not boilerplate"

Honest limitations

Adding to an existing project instead

Next in this series

Stop Counting Requests: The Case for Token-Based Quotas in LLM SaaS

Why RPM Metrics Fail for LLMs

Tokens ≠ Costs: The Complexity Layer

The Reservation Pattern: Solving the Temporal Problem

Implementation Strategy: Two-Tier Limits

Beyond `tenant_id`: Why Classical Multi-Tenancy Fails for RAG Systems

1. Vector Store Isolation: The Silent Leak Risk

2. Semantic Caching: Keys Must Include Tenant ID

3. Token Cost Management: Beyond Request Rate Limiting

4. Context Assembly and Guardrails

Conclusion

Stop Hardcoding Your AI Guardrails: A Multi-Tenant Configuration Strategy

The Pipeline Model

Handling Streaming Output

Audit Trails and Compliance

The Two-Tier Configuration Model

3. Errors after `200 OK` must travel inside the stream