Having an agent these days is not that hard with Claude, Hermes, ChatGPT, etc.
Most people these days, I believe, only see the tip of the iceberg. In most cases, they build and communicate with these agents through natural language.
I don’t say there’s anything wrong with it. Instead, it’s a very convenient and useful way to make the most of the technology we have available today.
But the tradeoff is that they never really know how exactly the machine runs behind the scenes.
Do they even need to know?
For general tasks, def no. But when it comes to a very specific use case for a specific person, I bet the answer is yes. Or at least, it's necessary to have someone who understands it deeply enough to handle that for them. Individually.
How the machine is running behind these agents
I never understood this until I built my first ever agent.
I called this one a Crypto Research Assistant. It does a super simple task: receive a question about crypto, then answer it. If the question is not covered in its provided sources, it’ll say that it doesn’t know.
Its scale is 0.0001% of every current agent you can randomly name out there, but even with this small agent, the work behind the scenes is way more complicated than I thought before.
Let me unpack them for you.
1/ First, the core agent loop
Do you know the fact that it’s never the case that an agent receives a question and gives you the answer in the very first response?
In practice, they call multiple tools, receive the results from those tools, and input those results back into the model. They’ll do this multiple times until they have enough information to answer your question.
That’s the agent loop.
2/ Second, the RAG pipeline
This is the core structure to handle source ingestion and find the answer to your question. There are 4 main steps:
Chunking -> Embedding -> Retrieval -> Generation
The RAG pipeline will then be wired as one of the tools in the core agent loop, so the agent can actually use it.
3/ Third, evals (also one of the most important things)
At this point, the agent can answer your question. The thing is you don’t even know whether its answer is right or not.
Maybe you do because you know all the information in the sources you give it, but users won’t.
So you need to build a system to check:
- Leak: Is the agent’s answer actually from the sources, or is it guessing based on its memory?
- Over-refusal: Does the agent refuse to answer even when it knows the answer?
4/ Fourth, the deployment
Ok, now you finally have a basic agent running. But if you want others to use it, you have to deploy it somewhere.
Keep in mind, you don’t want to publish the source documents when you deploy the app to the cloud. So here’s the stack I chose:
- For code: my Github repo of course. This contains the source code only, no secrets or any private source documents
- For document storage: an S3 bucket, read by a least-privilege IAM key
- For UI: deploy to Streamlit Community Cloud from my public repo, password-gated
Yep, that’s the whole process.
That doesn’t even count the bugs that I encountered along the way. So if I could go back in time and build it again, here are the hardest lessons that I wish I knew earlier.
The Lessons
1/ Your evals won’t sustainably be 100% all the time
I used to spend a lot of time fixing the system prompt and tool description so that all my evals could sustainably pass. The benchmark I made was 3 times 100% in a row.
The reason why it’s so hard is that the LLM’s response is different every time, even when you don’t change the prompt.
So choose your target eval, improve only one thing at a time. When you achieve it, move on. Don’t ever fall into the rabbit hole like me.
2/ Watch out for your token usage
If it’s only a few LLM calls, that’s basically fine. But if you’re A/B testing your evals, watch out for it.
Shout out to the many devs in the comments who gave me advice on this.
Some of the solutions that you want to note about this:
- Set a cap for token usage
- Give the agent the latest response for context, so it won’t rerun the whole thing every time
- Only test the eval you want to fix, not the whole set every time
3/ Don’t make a chunker that overfits to any specific article
Chunking is one of the hardest parts I got in this entire agent-building process tbh.
The hard part of chunking an article is:
- It has to be not too long because the model has a token limit per call
- Or too short because the chunks list would be too fragmented. The model then can’t find the exact chunk it needs to answer your question
I used to build a chunker that divided a specific article based on its content. The mean rank improved significantly, but when it came to multi-doc ingestion, that chunker just didn’t work anymore.
So I had to spend another morning just rebuilding the entire chunker and retesting the evals to make sure everything still worked.
A lot of work and time wasted, but lesson learned.
Final Thoughts
That’s everything I’ve learned so far from building an agent.
It sounds so complicated, but my thesis is that to get further in building these agents for specific use cases, you have to understand deeply how they work.
Or else, you’ll never get the output you want.
If you find this article interesting, pls drop a follow. Also, if you have any thoughts or suggestions, I’d love to hear them and chat more with ya.
Hope you enjoyed the read!
- Aeron
Top comments (0)