I built an MCP server so AI agents can find and query our company data. Most of my time did not go into the server. It went into one question: why does the agent pick the wrong table?
The answer is almost always the dbt docs. Here is what I learned.
1. The agent only sees the start of your description
When I indexed our dbt models for search, I could not embed everything. Long text gets cut. So only the first line of a model description and the first two sentences of each column count.
Everything after that never reaches the search. Put the point in sentence one: what this table is, at what grain, and what it is not. Same for NULLs. If a column is null for a special reason, say it in the first sentence.
2. Never let AI write your YAML from the model
This is the biggest trap. You have 300 models, no time, so you ask an AI to write the descriptions from the SQL.
You get things like user_id: Identifier of user.
That says nothing. The agent can already see the column is called user_id. What it needs to know is how to use it: which table to join it to, whether it can be null, whether it is the player or the account, and which other user_id columns it must not be mixed with.
An AI that reads the model can only repeat the model. Then the agent reads that text and learns nothing new. You are in a loop: the code explains itself to itself. Descriptions have to come from someone who knows how the data is used.
3. Not every table belongs in the index
You can answer the same question from five different tables. Raw events, a staging view, a cleaned model, a mart, a dashboard table. If all five are searchable, the agent will pick any of them, and some give a different number.
I keep raw, monitoring and technical schemas out of the index on purpose. The agent should only find the layer I want people to query: the core and mart models, where the logic is already decided. The cleaned, shared layer in the middle of a medallion setup matters most here, because that is where the business definitions live.
Less in the index means fewer wrong answers.
4. More chunks is not more recall
I used to split wide tables into several chunks, one per group of columns. On a set of 224 real questions, those chunks took 33% of the top-5 results and produced only 6.6% of the correct answers. They carried column names and no context, so they matched everything.
One chunk per table, with the description first, fixed it. Hit rate went from 0.879 to 0.893 and noise from unrelated tables dropped a lot (30 → 9 on one group of questions).
5. Test what your users really send
My Turkish test scored 0.52 against 0.91 in English. Then I checked the logs: the agent translates the question to English before it calls the tool, every time. The Turkish test measured something nobody does.
This is the first post in a short series on running a data platform that AI agents can use, so a whole product team can make data-driven decisions at any hour without waiting for an analyst.
Coming up:
- The key building blocks of an AI-centered data platform, and how to get a whole product team using it day to day
- Tricks that improve the quality of what the agent gives back
Top comments (0)