Why We Never Let an LLM Talk Directly to Our Database (A Teams + Copilot + MySQL Case Study)

TLDR

Letting a chatbot generate SQL and fire it straight at a production database works great in a demo and falls apart the first week in production. This post walks through the setup we actually use: Microsoft Teams as the chat window, a Copilot Agent that turns a question into a draft query, a custom API sitting in between that decides what's actually allowed to run, and MySQL underneath that never talks to anything except that API.

Four things break naive "LLM-to-SQL" setups almost immediately:

• Plain English is full of unstated assumptions ("this month," "region," "sales" all mean different things to different people)
• The model doesn't know your schema's history it has no idea orders was replaced by orders_v2 last year
• Nothing stops a bad prompt from becoming a full table scan or a query that crosses a tenant boundary
• A stateless bot forgets what "that" referred to the moment you ask a follow-up

None of these get fixed by writing a better prompt. They get fixed by architecture.

The problem with "just wire an LLM to the database"

Every analytics team has fielded the same request: "can I just ask instead of filing a ticket?" Self-service BI has been promising this for years, and most attempts fall over the first time someone asks a question the tool wasn't built to expect.

Here's what usually happens: a team grabs a chatbot template, points it at an LLM, hands the LLM a database connection string, and ships it. It works beautifully for the first ten questions the ones from the demo script. Then a real user asks something ambiguous, references a conversation from yesterday, or asks for something they probably shouldn't be able to see, and the whole thing quietly produces a wrong answer with total confidence.

That's arguably worse than no tool at all. A business user who gets a confidently wrong number stops trusting every number the system gives them afterward.

Where it actually breaks

Ambiguity hiding in plain language. "Total sales for this month by region" sounds unambiguous until you ask: calendar month or fiscal month? Region as in sales territory, shipping address, or billing address? A model that silently picks one interpretation isn't being helpful it's guessing and hoping.

Schema knowledge nobody fed it. Real production databases aren't the clean three-table examples in a tutorial. They're full of renamed columns, soft-deleted rows nobody cleaned up, and a _v2 table that quietly became the source of truth eighteen months ago. An LLM has zero visibility into any of that unless you hand it that context on every single call.

No ceiling on what the query can do. Once natural language becomes executable SQL, you're one bad interpretation away from a DELETE, an unindexed scan across a huge table, or a join that leaks data across a tenant boundary. If the model's judgment is the only thing standing between a typo and your production data, that's not a safety mechanism that's crossing your fingers.

No memory of the conversation. "Now break that down by product line" only makes sense if the system remembers what "that" was. A stateless request/response bot loses this instantly, and once it does, people stop trusting it with anything more than a single isolated question.

What we build instead: four layers, each with exactly one job

The fix isn't a cleverer prompt. It's separating concerns so the layer closest to the data is also the strictest one not the one that trusts the model the most.

[ Microsoft Teams ]  →  [ Copilot Agent ]  →  [ Custom API ]  →  [ MySQL ]
   chat surface          intent + SQL          the actual gate     system of record

1. Teams: the front door, and nothing more

Teams is where people already are, so it's the natural place to ask a question but its job should stop at rendering the chat, holding conversation/tenant context, and displaying results as tables or cards. No reasoning happens here. Keep it thin enough and the exact same backend can sit behind Slack or a web app later without a rewrite.

2. The Copilot Agent: figure out intent before writing SQL

This is the layer that turns "show me sales by region" into an actual query but only after it's resolved what the person is really asking, using domain context and whatever came earlier in the conversation. Once intent is nailed down, it drafts a query that's read-only by default, gets validated, and gets optimized before it ever leaves the agent. From there an orchestrator hands the request to the API layer and is responsible for failing gracefully a clarifying question in chat, not a stack trace.

3. The custom API: this is where the actual safety lives

This is the part no demo ever shows, and it's the one that matters most. A handful of endpoints something like /executeQuery, /metadata, /health sit between the agent and the database. Every request gets authenticated, scoped to the right tenant, validated, and logged before anything executes. Only then does a query executor run against MySQL and format the result. If the agent is where SQL gets written, this is where it has to earn permission to run.

4. MySQL: the system of record, reachable one way only

The database itself tables, views, stored procs is only ever touched through a secured connection from the query executor. It never sees the agent directly, and it never sees natural language. By the time a query reaches MySQL, it's already validated, authenticated, and read-only.

The fifth "layer" that runs through everything else

Every hop in this chain Teams to agent, agent to API, API to database, and the response flowing back up runs over HTTPS with read-only access enforced at every step. Logging and monitoring aren't an afterthought bolted on later; they're threaded through each layer so that when something does go wrong, there's an actual trail back to the request that caused it.

The rule we keep coming back to: a natural-language interface is only as trustworthy as the layer the model isn't allowed to talk its way around.

Getting the agent layer right

The temptation on any project like this is to let the model do too much write the SQL, decide who's allowed to see the result, and run it directly. That collapses three separate jobs (understanding, authorization, execution) into a single prompt. Prompts are not access control.

The pattern that actually holds up splits this cleanly:

• The agent understands language and drafts SQL. That's it.
• The API owns authentication, tenant authorization, and validation and it's the only thing holding database credentials.
• The database executes a query that's already been checked, scoped, and forced read-only.

Each layer trusts the one below it exactly as much as it's actually verified never more.

This also tells you where retry logic belongs. If a generated query fails validation or times out, the right move is a refined attempt or a clarifying question back in chat ("did you mean the fiscal month or the calendar month?") not a silent fallback to a broader, unbounded query that happens to return something.

What to build vs. what to rent

Microsoft's Copilot Studio and the Teams SDK hand you the conversational plumbing chat UI, session handling, adaptive cardsessentially for free. That part is commodity infrastructure, and renting it is the obviously correct call.

What's not commodity is everything inside your custom API: validation rules, tenant isolation, read-only enforcement, the audit trail. That logic encodes how your org actually governs access to data, and it shouldn't live inside a generic connector you don't control.

A simple test: if a layer's job is "let someone type a message and get a response," buy it. If a layer's job is "decide whether this request is allowed to touch this data," build it and keep it.

What "production-ready" actually looks like

Pulling all of this together, a system worth trusting shares a few traits:

• Intent gets resolved before SQL gets generated, so ambiguity gets caught early instead of baked into a wrong query
• Every generated query defaults to read-only and gets validated before it goes anywhere
• One dedicated API layer authenticates, authorizes by tenant, and logs every call and it's the only thing with database credentials
• MySQL only ever runs what's already been checked
• The entire path runs over HTTPS with a monitoring trail a security team would actually recognize

None of these pieces are exotic on their own. The real engineering is refusing to let a model's fluency substitute for an access-control layer you only need to build once.

FAQ

Why not just point the LLM at the database directly?
Because at that point nothing stands between a user's phrasing and production data except the model's judgment. A dedicated API layer between the agent and MySQL is what actually enforces read-only access, authentication, and tenant isolation. The model should never hold credentials.

Does the agent decide what data someone's allowed to see?
It shouldn't, and in this setup it doesn't. Authorization lives in the API layer, which checks identity and tenant before a query ever runs. The agent's job is understanding intent and drafting SQL not gatekeeping access.

How do follow-up questions like "now break that down by product" work?
Session and conversation context live in the Teams layer and get passed to the agent, which keeps enough history that a follow-up resolves against the prior query instead of starting from a blank slate.

Could this move to Slack or a plain web app instead of Teams?
Yes as long as Teams stayed thin. If all the actual reasoning lives in the agent and API layer rather than leaking into Teams-specific code, swapping the front end is a delivery-layer change, not a rebuild.

We build AI agent and data-integration systems for enterprise teams at Bitcot. If you're wiring an LLM up to a real database and want a second opinion on where the guardrails should live, let's talk.