At Opero we build agents, the sort of voicebots and chatbots technical staff use in the field or at the office while preparing for a job. They are built on the technical documentation of manufacturers, engineering labs, HVAC companies, and field service teams.
These agents are not demonstrations. A technician uses them when a machine is broken.
Many of my first beliefs were incorrect. Some were also expensive. These are the 13 things I learned along the way.
It would mean the world if this is useful to someone building their own agents.
The Stack
This is the software that we use:
- Agent and API: Python, FastAPI
- Orchestration: LangGraph for the graph, LangChain for the components
- Observability: Langfuse
- Retrieval: OpenAI embeddings, Milvus (hybrid dense and sparse search), zerank-2 for reranking
- Structured data: Postgres
- Models: Different providers with automatic failover. If one provider gives an API error during operation, we send the request to a different provider.
- Communications: agentic phone calls, SMS, and email with Hail.
- Infrastructure: servers on Hetzner, storage and some pipeline components on AWS
Part 1. Ingestion
Lesson 1: Build Your Own Document ETL Pipeline
We started with a commercial document processing platform called "Unstructured".
The platform was easy to start. We had a system in operation in less than one day. Commercial platforms are good for this.
Then we found problems.
The extraction quality was low. The platform gave structured output, but the structure was correct for Unstructured, not for us. We changed our system to agree with their format. This is the incorrect sequence. It's an anti-pattern. We had to twist our pipeline to make it work.
Then we started working with Zeppelin (a major Caterpillar dealer), and the first document they shared was 8,000 pages of dense technical documentation. Drawings, schematics, backlinks that cross-reference other pages, complex identifiers, etc.
The pipeline on Unstructured failed 20 times. We paid for each failure.
At that time, we made a decision. The cost was one problem, but the larger problem was different: our document processing logic was in a system that we could not examine, repair, or improve.
We benchmarked the available platforms and libraries, and settled on an open source library called Docling. We built a prototype in one weekend. The prototype gave better results than the commercial platform.
We continue to improve the pipeline. It is now fully automatic and the results are good.
The lesson is not "do not use commercial tools". Start with a commercial tool. Release your product MVP. Learn your true requirements. But move key architecture components like document processing to your own system as soon as possible.
Data processing is the base of all other functions. You need flexibility, cost control, and the ability to repair your own failures.
Lessons 2 to 11 are possible only because we control the ETL pipeline.
Do you want a full article about the ETL pipeline, the architecture, the models, and the orchestration? Tell us and I will write it.
Lesson 2: Owning the Pipeline Cut Our Costs 15 to 20x
Unstructured charged 20 USD for 1,000 pages at the start. The price then increased to 30 USD.
Per-page pricing does not reflect real cost. The true processing cost changes with the document. One page of simple text and one page of rotated engineering drawings are not equivalent. But the provider charges the same price for the two pages.
We operated our own pipeline for some weeks. We measured the cost with real customer documents. Our cost was 15 to 20 times less.
We did not use low quality models to get this result. We used good models. We operated the models on our own GPU infrastructure with our own routing.
For a small company, this is not just an improvement — it is months of extra runway.
Lesson 3: Extraction Quality Sets the Maximum Performance of the System
You cannot correct bad extraction with a better retriever, a more intelligent agent, or a larger model.
A spec sheet has a key-value layout. If the extraction makes this layout into unstructured text, the data is lost. If the extraction ignores a rotated page, the data is lost. If a table loses its column alignment, the data is lost. No subsequent process can recover this data.
Technical documentation has these conditions frequently: rotated pages, dense tables, spec sheets with key-value layouts, scanned manuals that are 30 years old, and diagrams with important text in the image.
All our improvements in accuracy start with correct extraction.
Lesson 4: Chunking is a Strategy, Not a Default Setting
We use hybrid chunking. This method divides the document by its structure and its hierarchy. It then merges the parts by token count.
Many developers use a recursive character splitter with a 512 token window. They do not change this setting again. Then they ask why the retrieval quality is low.
Your chunk boundaries control the possible results of the retriever. If a procedure is divided between two chunks, no retriever can give the full procedure to a technician.
Lesson 5: Extract Taxonomy and Tags in the ETL Pipeline
We define a taxonomy during ingestion: manufacturer, model, and custom tags for each organization. The pipeline extracts the tags during preprocessing. We keep the tags as scalar filters with the vectors.
This looks like a small administrative task. It becomes a product function in Lesson 13. It is also a good example of a function that a closed commercial platform does not permit.
Part 2. Retrieval
Lesson 6: A Simple Top-K Semantic Search is Not Sufficient
The first version of the agent at Opero did a simple top-K semantic retrieval. It found 10 to 20 documents by semantic similarity. It put the documents in the context. Then it generated an answer.
There was no reranking. There was no relevance filter. There was no procedure to find if the documents were best available for the user question.
We hoped that cosine similarity would find useful data. Then we gave the result to the user as an answer.
This method worked well enough for a demonstration. It failed often enough to be dangerous. This is the most dangerous failure condition in this field.
Lesson 7: Add a Reranking Step
We added a reranking step. The system finds many candidate documents. A dedicated model then gives a relevance score to each candidate for that query.
The improvement was immediate and large. If you apply only one lesson from this list, apply this one.
We also use the relevance score in the user interface. If the best result has a low score, we tell the user. We do not give a confident answer from low quality context.
Lesson 8: Rerank Scores Have No Absolute Scale — Pick a Model That Calibrates Them
We started with Cohere Rerank. It operates correctly, but its relevance scores do not use an absolute scale.
So we had to tune the thresholds for each organization and each industry type. We had to decide which score was high, medium, or low. These thresholds were estimates. They also changed when the document collection changed.
We changed to zerank-2 from ZeroEntropy. This model gives standardized relevance scores.
This looks like a small change. It is not a small change. You can define a fixed relevance scale one time. You can then build product logic on this scale. The scale stays correct when your data changes. You do not have to keep a calibration procedure.
Lesson 9: Use Hybrid Search If Your Users Type Serial Numbers
We added BM25 keyword matching with the semantic search.
Engineers and technicians do not write complete questions. They type "E-047". They type a part number from a label. They type a serial number.
Vector search has low accuracy with these exact terms. This is not a defect. Embeddings find meaning, and an error code has no meaning.
Semantic search finds the answer for "why does the compressor short cycle".
BM25 finds the answer for "SCR-4471-B".
Users need the two methods, frequently in the same question.
Lesson 10: Multiple Languages Are Usual
Most of our documentation is in English. But we also have documents in German, Danish, Swedish, and Chinese. European technical customers have documents in these languages.
Plan for multiple languages from the first day. It is difficult to add this function later.
Part 3. Orchestration
Lesson 11: Start with a Workflow. Change to an Agent Later.
Our first version was not an agent. It was a DAG. We built it with LangGraph, and we still use LangGraph for all orchestration.
- Node 1: classify the input. Is it a question or a greeting? The system answers greetings directly and at low cost.
- Node 2: query the RAG system, format the result, and give it to the user.
That was the full system. This was the time of GPT-4o and Claude 3.5 Sonnet.
You can debug a deterministic workflow. You know which step failed. Agentic loops are more difficult to analyze. If you give autonomy to a loop that uses an unreliable retriever, the loop fails in unusual ways instead of predictable ways.
Repair retrieval first. Add agency second. Use this sequence.
Lesson 12: Give the Agent Permission to Try Again
We changed to a ReAct loop. The agent decides if the documents are sufficient. If they are not sufficient, the agent rephrases the query and does a new search.
This function operates only because of Lessons 7 and 8. The agent reads a standard relevance score. It finds that the results are not sufficient. It writes a new query. Then it does the search again.
Reranking gives the agent a signal. Standard scores give the agent a threshold. Agency without these two pieces is only an expensive random search.
We also added a pre-retrieval layer. This layer finds context quickly and sends it to the primary node. So the usual case is fast. The ReAct agent keeps its tools for a more complete search when necessary.
Lesson 13: When the Agent Cannot Find an Answer, Give Control to the User
This is my preferred part of the system. It is the result of Lesson 5.
If the first query gives results with a high relevance score, the agent responds immediately. If it does not, the agent requeries with a different angle.
But if the agent gets stuck in a loop and cannot find sufficient results, we do two things. We do not generate an incorrect answer. We also do not show only the message "no results found".
We show a filter interface. We build this interface from the taxonomy tags. The user selects a manufacturer, a model, or other data that the user knows. We then do the query again automatically with these filters.
The user knows data that the retriever does not know. Let the user nudge the agent.
Degrading gracefully is better than a confident incorrect answer.
Part 4. Communications
Bonus. MCP Makes Integration Simple
The primary loop is now reliable. So integration is configuration work instead of architecture work.
We added MCP servers for customer system integrations. The agent can now get live data, not only documentation, using Nango.
We added web search. The user can now include internet sources.
We also added voice, SMS, and email in one API/MCP layer using Hail.
This last item changes the function of the agent. The agent is no longer a chatbot that answers questions. It can call a technician. It can send a part number by SMS. It can send a report by email. The interface is no longer a text box. The interface is the system that the user has.
The Primary Lesson
There is one pattern in all 13 items.
Every important improvement comes when we take control of a layer that we initially rented.
The ETL pipeline. The chunking. The taxonomy. The retrieval strategy. The orchestration logic. Each layer was initially an abstraction from a different company. Each layer becomes better when we can examine it and change it.
Start with the commercial platform. Release a product. Learn what is important. Do not build infrastructure for a product that has no users.
But you will get to the limits of that platform. Know which layer you will take control of first.
I build Opero, which supplies AI agents for technical field service, engineering teams, OEMs, HVAC, and manufacturing. I write about the problems we hit.
Which lesson do you want more information about? Readers ask about the ETL pipeline most frequently. Tell us and I will make it the next article.
Top comments (0)