Not a researcher. Not a professional dev. Civil engineering background. Started building an AI bot because I wanted to understand what's actually happening inside these systems — not theoretically, just practically.
Wanted an assistant that could live with a conversation instead of treating every message as an isolated API call.
It started as an experiment. Then the experiment grew.
What it turned into
At some point I had:
- routing between reasoning profiles
- multiple tools
- web search
- memory (flat + semantic + graph)
- caching
- context compression
- feedback loops
- self-learning experiments
- quality monitoring
- batch processing
- telemetry
- health checks
- and enough logs to make me question my own sanity
I wasn't trying to build a benchmark. I was trying to make the thing actually work for me.
That distinction changed everything.
An agent is not an LLM call
You think it's:
User → LLM → Answer
Reality:
User
↓
Telegram
↓
input handling
↓
session state
↓
intent detection
↓
routing
↓
profile selection
↓
context construction
↓
memory
↓
cache
↓
tools
↓
LLM
↓
post-processing
↓
quality checks
↓
Telegram
Every layer is another chance to break something.
The model was often not the problem. The machinery around it was.
The cache thing
Was looking at prompt caching and noticed something weird.
Same model — different behavior depending on the provider. Changed the provider — problem disappeared. Connected the same provider from another IDE — cache worked fine. Back in my bot — sometimes it didn't.
Then I realized: OpenRouter can silently switch providers under the same model name. Cache hit rate changes with it.
Stopped thinking "does the model support caching". Started thinking "what exactly is being sent, and what does the provider consider identical".
Much more useful question.
Why it was different in the IDE
The IDE was sending:
model → cache works
My bot was sending:
routing
+ dynamic context
+ session metadata
+ memory
+ profile-specific stuff
→ provider
→ model
→ cache may or may not match
The model hadn't changed. The provider hadn't changed. The request structure had.
Caching is extremely sensitive to prefix stability. Once I understood that and stabilized the relevant parts — got around 66% cache hit rate on average, up to ~80% in favorable conditions (same topic, stable structure).
That's the part benchmark screenshots don't show. Cache hit rate is not a model trait. It's a property of your workload.
"90% token savings"
When I see this claim now I don't think it's fake. I think: show me the workload.
90% is possible if your prefix looks like:
system instructions (stable)
+ tools (stable)
+ project context (stable)
+ conversation history (stable)
+ small new message
Real agents often look like:
system instructions
+ changing tools
+ changing memory
+ changing routing metadata
+ changing summaries
+ changing retrieved docs
+ changing timestamps
+ new message
Prefix isn't stable. Theoretical saving and practical saving become very different numbers.
Routing is harder than it looks
Obvious idea: route every request to the cheapest model that can handle it.
Works fine until you have a real conversation.
User says: "What's the weather?" — easy.
Then: "Compare it with yesterday." — now context matters.
Then: "Actually forget the weather. I was thinking about that thing we discussed yesterday." — now memory matters.
Then: "No, not that. The other one." — router needs to understand the whole conversation, not just the current sentence.
The routing decision isn't question → model. It's:
conversation state
+ user intent
+ previous actions
+ available tools
+ risk
+ latency
+ cache state
→ routing decision
And here's the problem: a routing decision changes the request. A changed request affects caching. A changed profile affects context. A changed context affects the answer. A different answer affects feedback.
A tiny routing optimization has consequences five layers away.
The latency numbers
Real VPS measurements at some point:
| Metric | Value |
|---|---|
| LLM p50 | ~2.2s |
| LLM p95 | ~17.3s |
| LLM max | ~51s |
| Agent median | ~6–11s |
| Agent p95 | ~20–57s |
| Earlier pipeline tails | up to ~116s |
Interesting number isn't LLM latency. It's the gap between LLM latency and agent latency.
Model answers in 2 seconds. Agent takes 10–57.
Because the agent isn't just the model. It's everything before and after it.
Telemetry became the most important thing
Ended up recording: LLM usage, prompt tokens, cached tokens, latency, route decisions, quality events, memory operations, feedback, errors, conversation traces.
Looked excessive at first. Then it became obvious why.
When something broke I could ask "what actually happened" instead of "I think the model was confused".
Very different debugging strategies.
A lesson about noisy telemetry
One audit showed thousands of route_risk records. Sounds catastrophic.
But most were repeated observations like:
quality_loop:search_skipped
quality_loop:price_hallucination
quality_loop:reply_echo
They were useful signals. But not thousands of independent disasters.
Telemetry volume is not incident volume.
A system can produce huge numbers of observations about a small number of underlying failure patterns. You have to cluster them, or the monitoring system itself becomes noisy.
Self-learning is a trap if your data is bad
Experimented with feedback loops. Bot could receive 👍 👎 and connect that to routing quality, skill reputation, scenario history.
Attractive idea: agent learns from mistakes.
Dangerous question: what exactly is it learning from?
If telemetry is noisy — it learns noise. If a synthetic probe looks like real user traffic — it learns from the wrong population. If a failed tool call gets logged as a routing failure — the wrong lesson gets created.
So the learning loop needs another loop around it:
experience → evidence → validation → lesson → application → new evidence
Otherwise you're automating superstition.
Context compression
Added protection for recent messages — compress old history, preserve the newest turns.
OLD OLD OLD OLD NEW NEW
→
[summary] [summary] NEW NEW
Reason: a summary is an interpretation of the conversation, not the conversation.
Sometimes the missing detail is one sentence. And that sentence changes the meaning of everything.
Batch processing
Parallel execution looks great on paper. 12 tasks × 2s each = 2s instead of 24s.
But natural language tasks aren't always independent.
- Find three products.
- Compare them.
- Tell me which is best.
Task 2 depends on 1. Task 3 depends on 2.
Ended up checking for cross-references, pronouns, comparative language, explicit dependencies before deciding to parallelize.
The optimization wasn't hard. Knowing when it's safe was.
Memory introduced its own problems
Added knowledge graph + semantic memory + flat persistence + vector search + entity relationships.
Sounds sophisticated.
But memory introduces a basic question: should this actually be remembered?
Then: should it be retrieved now?
Then: is this memory still relevant?
Then: is this memory more important than what the user just said?
A memory system doesn't automatically make an agent remember better. Sometimes it makes it remember too much.
The most important lesson
After adding enough machinery you eventually discover that machinery itself becomes the problem.
An agent can have router + memory + tools + cache + planner + evaluator + self-learning + scenario engine + quality loop and still perform worse than:
short prompt + one good model
for a simple task.
Sometimes the right optimization is subtraction.
Remove unnecessary context. Remove unnecessary routing. Remove unnecessary abstraction. Keep the useful part.
What I'd do differently
Start with:
one model
one provider
one short system prompt
one conversation store
one cache strategy
minimal tools
excellent telemetry
Run it. For a long time. Only after seeing real failures add another layer.
Not architecture first → hope it works.
But simple system → observe → measure → find failure → fix → measure → only then add complexity.
Feels slower. In practice, probably faster.
What the experiment actually taught me
The hard part of an AI agent isn't making the model answer.
The hard part is maintaining a stable environment around the model while everything keeps changing.
User changes topic. Context grows. Provider changes. Cache behaves differently. Tool times out. Router makes a different decision. A previous answer was wrong. A correction arrives six turns later.
And somehow the assistant is expected to behave as if none of that happened.
That's the actual engineering problem.
The test I trust now
Not the number of agents.
Not the number of tools.
Not the biggest benchmark.
Not even the highest cache hit rate.
Just:
Does it still work when a real person uses it tomorrow?
Top comments (0)