TL;DR: Motley lets you see what data questions your users actually ask, how they get answered, and what is wrong or missing. This makes it easier to improve the semantic layer, by hand or with AI agents.
Of all the reasons to have a semantic layer, the main one today is increasingly letting AI agents answer data questions correctly. How do you know whether your semantic layer is doing a good job of that, and how do you improve it?
There are many possible friction points. Do the models cover everything users ask? Are the descriptions good enough for the agent to discover and understand the metrics? Are there gaps in context, unexplained gotchas in the data?
A semantic layer is an ever-changing system: it evolves and expands over time, getting better and better at answering questions – and this can happen very fast, because users will ask a lot of increasingly complex things, giving you feedback and usage data. How do you make the most of it?
There are two main ways to judge how well your semantic layer performs: looking at real user sessions, and running offline evals before problems reach users. This post covers the first, arguably the more powerful one.
See what your users actually ask
Out of the box, Motley lets you browse your users’ sessions and see what questions they ask their agents and how the agents answer them. You can see what the semantic layer is actually used for and where to improve it.
Every question gets a report
For each user question, Motley asks the agent to file feedback: what was asked and whether an answer was found. If the question was answered without issues, the agent also adds a short explanation of how it got there; otherwise, it writes a problem report.
Each report falls into one of these categories:
Answered – when an answer was found without issues.
Missing data – when the agent couldn’t find the data to answer the user’s question.
User asked for churn by plan; there is no subscription end date in any model.Unclear docs – when descriptions or memories were missing, ambiguous or contradicted the data, so the agent couldn’t work out how to proceed.
orders.amount doesn’t say whether refunds are included; I assumed they are.Wrong result – when the result of a query was incorrect or looked incorrect.
Revenue for March came back 3× higher than in the previous answer; the join to payments likely fans out.Other – anything else the agent thinks you should know.
A single session can have several reports, because users often ask several questions. Reports in any category other than “Answered” arrive with the status “Open”, so you can track which ones you’ve addressed. Once you’ve fixed the cause, mark the report as “Resolved” and it is excluded from the overview on your Home page.
The category usually tells you the fix: missing data means adding a column or measure, unclear docs means correcting a description or adding a memory, and a wrong result means opening the query and checking the joins and filters.
Trace any answer back to its query
As an admin, you can open any session, along with the feedback attached to it and all the tool calls the agent made – what it searched for, what it queried, what SQL Motley executed and what numbers came back. This lets you trace each answer back to the query and the reasoning behind it, and debug any problem that appears. Any query can be opened in the Playground, so you can test your changes by re-running it.
Let your agent do the triage
You can also connect your agent to these sessions – for admins, the Motley MCP server provides tools to explore the sessions and feedback. Ask your agent “what went wrong in session #01afbc7…” or even “explore the recent sessions with negative feedback, propose changes to the semantic layer based on our dbt project and open a PR for each”. The agent can also close the loop on its own: list the open problems, inspect the session, fix the model, description or memory, and resolve the report.
See our Usage docs to learn more, or try out Motley Cloud to see how it works (demo workspace available).
Analyzing usage and catching problems as users hit them is great, but would you rather prevent them in the first place? In Part 2, we’ll talk about using offline evals for that.
Top comments (0)