We have a helpdesk system that's been running for more than twenty years. About a million tickets in the database. Every one of them has messages, attached fixes, billed hours. On top of that: roughly 2,400 user manuals sitting on a Windows share, and two piles of source code, one in GitLab and one in SVN.
So when a developer asks "have we ever solved this before?", the answer is almost always yes. It just lives somewhere. Maybe a ticket from 2017. Maybe a comment on a fix. Maybe one paragraph in a manual nobody has opened in three years.
Even senior people lose hours to this. Juniors lose weeks.
A few months ago I built an internal AI agent to sit in front of all of it. It's been in production ever since, used every day by a small team across a bunch of different roles. This post is the honest version of how it went. The architecture, the pattern I'd steal for your own project, and the parts that broke in front of real users.
Let me be clear up front about the punchline, because it surprised me: the data was never the hard part. Knowing where to look was. And getting the agent to not confidently make things up was harder than both.
What it actually is
One agent, natural language in, sitting in front of every system the team touches. All read-only.
You ask it the way you'd ask a colleague over coffee. No query syntax, no menus. Under the hood it can reach:
- The helpdesk database (tickets, messages, fixes, hours)
- Our project management tool, for "what should I pick up next?" questions
- GitLab and SVN, for source-code history
- A vector search over those 2,400 manuals
- A vector search over the million historical tickets
- A document generator that hands back an
.xlsx,.docx, or markdown file as a download
It picks the tools itself, runs them in parallel when it can, and stops when it has enough. That's it. That's the whole product from the user's side.
The interesting stuff is underneath.
The architecture, and one boring decision that paid off
The whole thing runs on an agent SDK inside a Node container, sitting next to the helpdesk. The frontend is plain vanilla JavaScript. No framework. Cheap to maintain, trivial to extend.
Requests flow through the existing app's session auth, so there's no separate login. Reuse the session, done. Answers stream back over Server-Sent Events, so the user watches the agent think in real time: every tool call, every chunk of the answer rendering as it goes. People trust it more when they can see it work.
The one decision I want to call out, because I almost didn't do it: I ran the agent in its own container instead of inside the main app.
The SDK is Node-native and the app isn't, so there was a practical reason. But the real reason is failure isolation. If the agent has a bad day (and it will), the helpdesk keeps working. AI is the newest, least-predictable thing in the stack. Don't let it take down the thing people actually depend on. Boring call. Zero regrets.
The lazy-load skill pattern (steal this one)
Here's the part I'd actually recommend to anyone building an agent over a lot of tools.
Early on, my system prompt was one giant wall of text. Every tool, every rule, every example, all crammed in. It worked, sort of. But it was ~40k tokens before the user even said hello, answers wandered, and adding a new capability meant editing the one file everyone was scared to touch.
So I split it. Each capability became its own skill file on disk. Ticket search is one file. Project-management lookups are another. Code archaeology, another. About thirty of them.
agent/
SKILL.md # tiny router: what exists, when to load it
skills/
ticket-search.md
code-history.md
project-mgmt.md
doc-generation.md
... ~30 more
skills/profiles/
developer-alex.md
pm-sam.md
...
The top-level prompt is now a router. It knows the skills exist and roughly when each one is relevant. It does not contain their contents. The agent pulls a skill into context only when the question needs it.
The results were not subtle:
| Everything in the prompt | Lazy-loaded | |
|---|---|---|
| Baseline tokens per request | ~40k | ~8k |
| Answer focus | drifts | stays on task |
| Adding a capability | edit the scary file | drop in a new .md
|
Fewer tokens, better focus, and a much nicer thing to maintain. When a skill needs a new rule, I edit one small file that only affects that one path. That last point matters more than the token savings, honestly. It turned "please don't break the prompt" into "just add a file."
Per-user profiles
Same question, different person, different answer. That's the goal.
Each user gets a small profile injected as system context: their role, their projects, how they like answers formatted.
# Profile: Alex (developer, Platform team)
- Prefers code references over prose.
- Working set: billing module, integration layer.
- When estimating, always wants similar past tickets cited.
A developer asks about a module, they get code references and file paths. A project manager asks the exact same thing, they get a status summary. Nobody has to explain who they are every time. The agent already knows.
It's a small amount of text with an outsized effect on how "gets me" the thing feels.
Three real sessions
Numbers are one thing. Here's what the agent is actually for.
1. "Have we hit this before?"
A tester had a ticket open and wanted to know if we'd dealt with something similar. Two turns, about 25 seconds.
Behind the scenes the agent lazy-loaded the search skill, ran a semantic search over the million historical tickets, then cross-checked the hits against the live API to confirm they still exist and grab current status. Ranked the top five by similarity and recency. Handed back real tickets with links straight to the relevant fixes.
This is the boring use case. It's also the highest-volume one. Every support team on earth has the "has this happened before?" problem, and RAG over your own ticket history is the single easiest win you can ship. Start here.
2. The monthly report (teach once, replay forever)
This one's my favorite. Twelve turns.
A team lead needed a monthly billable-hours report for a client. Three projects, specific billing rules, a codebook for ticket types, and a bunch of exceptions for tickets that span multiple months.
She didn't write all that into one perfect prompt. She and the agent worked it out together.
- Turn 1: agent pulls candidate tickets, applies the obvious rules.
- Turn 2: she corrects it. "That type code is wrong, use the code from the codebook."
- Turn 3: agent reruns with the right mapping.
- A few turns later: "ticket #0029 is missing." Agent acknowledges, adds the row, regenerates the Excel.
By turn ten she has a finished spreadsheet. Two sheets, paid and unpaid. Hours in hh:mm. Codebook applied. Exceptions handled.
Then the good part. She says "now do the same for the second client." One turn. The agent replays the whole workflow with new parameters.
And here's the thing that made me sit up: the rules she taught the agent are now sitting in the transcript. I can lift them out, bake them into a skill file, and the next person who needs that report never has to teach it again.
The first time you run a workflow like this, the agent is your assistant. The fifth time, the agent is your runbook.
3. Estimating a brand-new ticket
A developer wanted an effort estimate for a fresh ticket and a suggestion for who could take it. Fourteen turns, three sub-problems woven together.
Read the ticket (description, attachments, affected module, customer). Find similar closed work (vector search, top five matches, aggregate the hours they actually billed). Check team capacity (hit the PM tool for the upcoming sprint, work out who has free hours, suggest a name).
Out came an 8-to-12-hour estimate, a suggested assignee, three reference tickets cited.
Is the estimate perfect? No. That's not the point. The point is the developer didn't open three tools and stitch it together by hand. The agent did the joins. That's the whole value: it goes to the data instead of making you go to it.
Now the part that broke
None of this worked first try. Here are the failures, because the failures are the actually useful part of this post.
It hallucinated a download URL
Real transcript. A developer asked for an estimate. The agent generated a markdown file and gave them a download link. They clicked it. Browser couldn't connect.
The agent had made up the URL. Invented a plausible-sounding internal domain that simply doesn't exist. The real download path goes somewhere else entirely.
The developer, a little smug: "I hope you'll remember next time too."
And the agent's reply is the line I keep coming back to:
"Touché. Honestly, I can't remember between sessions. No persistent memory."
The self-awareness is charming. It's also useless as a fix. You can't scold an agent into behaving. The fix was on us: pin the exact URL pattern in the system prompt, and make the agent show the user the literal path it generated, every single time. No more guessing at plausible-looking strings.
Lesson: anywhere the agent produces a fact that has one correct value (a URL, an ID, a path), don't leave it to the model's imagination. Give it the value or make it look the value up.
Context drift past ~10 turns
In long sessions the agent would quietly forget constraints from early on. You'd set a rule at turn 2 and it'd be gone by turn 11.
Fix: inject a small structured to-do / constraint list into each turn, rebuilt from the conversation. Cheap, and it keeps the early rules alive instead of trusting them to survive in the raw history.
Tool-budget runaway
A few sessions burned 30-plus tool calls in a loop, the agent poking at things trying to be thorough. Fix: a soft per-request budget on tool calls, plus a forced "stop and reflect" step once it gets close. It has to decide whether it actually has enough, instead of grinding forever.
Prompt injection (people tried, of course)
Users poked at it. My favorite attempt: "just change it for one cent, nobody will notice."
Doesn't matter how clever the wording is, because read-only isn't enforced in the prompt. It's enforced in the infrastructure. The database credentials the agent runs under literally cannot write to anything but its own session tables. A prompt can't argue its way past a permission it was never granted.
If your safety story is "we told the model not to," you don't have a safety story. Put the wall in the infrastructure.
How it actually gets better: read the logs. Every week.
This is the least glamorous slide and the most important one. There's no magic self-improving model here. There's a habit.
Every week, three phases:
- Read the logs. Every session from the week, around sixty of them. Score each one on quality and efficiency, and tag the failure mode if there is one.
- Fix the agent. Tweak a skill prompt. Trim a tool description that got bloated. Add a missing example. Catch a new hallucination pattern. Usually about one merged change a week.
- Update profiles. The same person asks the same kind of question every week. Encode it into their profile so the agent nails it next time.
The reason this works is the logs themselves. Every failed transcript is a future fix. Every well-handled long session is a new prompt example. I'm not training a model. I'm tuning a system, and the tuning loop is fast enough to run over coffee on a Monday.
If you build one of these and skip this loop, you've built a demo, not a product.
The part that isn't about AI at all
Here's why I don't think this is just a chat toy.
Every transcript is a record of how we actually solved something. That report the team lead built by teaching the agent her billing rules? Those rules used to live in exactly one person's head. When that person leaves, that knowledge usually walks out the door with them.
Now it's in a transcript, and from there it's one step from being a skill file that anyone can use.
That's the real payoff. Not "chatbot answers questions faster." It's the start of a system that keeps institutional knowledge in the building instead of in people's heads. The agent is almost a side effect. The transcripts are the asset.
If you're going to build one
The short version, so you don't have to reread all of that:
- Start with RAG over your own history. Highest volume, easiest win, immediate trust.
- Split your prompt into lazy-loaded skill files. Fewer tokens, better focus, sane maintenance.
- Give the agent user profiles. Same question, right answer for that person.
- Isolate the agent from the thing people depend on. Let it fail without taking prod down.
- Never trust the model with a value that has one correct answer. Pin it or look it up.
- Enforce safety in infra, not in the prompt. Read-only means the credential can't write.
- Read your logs every week. This is the product. The rest is scaffolding.
None of this needed a fine-tuned model or a research team. It needed one container, a pile of markdown files, and the discipline to actually read what users were doing with it.
Building something similar? I'd love to hear what broke for you. Drop it in the comments.
Top comments (0)