I'm building a Rails engine that lets SaaS customers ask questions about their own data in plain language. I don't want an LLM writing SQL or having direct access to my database. Not in a production SaaS application. This is why, and what this series will build, test and measure.
Context
My last series, DataPorter, was about engineering a gem: a DSL, an orchestrator, generators, a UI. This one has a second goal. I want to learn LLM integration and retrieval (RAG) properly, at a low level, with numbers instead of vibes.
So here is the deal. I'm building a real gem, in the open, and the series follows the order in which I solve the problems. Each part covers code that exists and is tested, every decision comes with the options I turned down, and once the LLM is in, the numbers come from evaluation runs, not from impressions.
The problem
Picture a SaaS that sells order management to online shops. A shop owner opens the app and wonders:
"How many orders do I have to ship tomorrow?"
Today, that owner needs a dashboard someone built, or a support ticket. The obvious idea in 2026 is to put an LLM in front of the database: let it write the SQL, run it, show the result. It's called text-to-SQL, it demos beautifully, and for an internal analyst it can be great.
Giving an LLM access to a database sounds convenient at first. But the moment you put this in front of actual customers, things get more complicated :
- The LLM writes raw SQL. Safety then relies on keyword blocklists and read-only transactions. That is a defence you hope works.
- Tenant isolation becomes the LLM's job. "Only look at account 42" is an instruction in a prompt. One prompt injection away from leaking another customer's data.
- Your schema, and often your results, go to an AI provider.
If you run a multi-tenant app, the second point alone should make you pause
| Text-to-SQL | What I want | |
|---|---|---|
| What the LLM produces | Raw SQL | A structured query, validated against what the developer declared |
| Tenant isolation | Asked of the model | Applied by the engine, outside the model's reach |
| What leaves your app | Schema, sometimes rows | The question and declared labels, never result rows |
| Who it is for | Analysts and admins | The SaaS's own customers |
What we're building
Here is where it's going. This is a mockup: a fake shop, fake data and scripted answers, made to show the idea. It only shows what is planned: a number or a table, how the question was understood, a clarifying question with fixed choices, a refusal. No chart, and no sentence written by the LLM.
DataTeller is a mountable Rails engine. The developer declares what can be asked about: datasets, metrics, dimensions, segments, with business names and synonyms. The LLM never sees the database. It sees the question and the declared vocabulary, and it fills in a structured query:
{
"version": 1,
"intent": "answer",
"dataset": "orders",
"metric": "count",
"period": { "dimension": "ship_on", "range": { "relative": "tomorrow" } },
"filters": [ { "segment": "paid" } ]
}
That JSON is validated against the declarations, then compiled by the engine into ActiveRecord, with the tenant filter added by the compiler. This is the SQL it runs on the dummy app :
SELECT COUNT(*) AS c0 FROM "orders"
WHERE "orders"."status" = $1 -- the "paid" segment, as the developer declared it
AND "orders"."ship_on" >= $2 AND "orders"."ship_on" < $3 -- "tomorrow", in the tenant's time zone
AND "orders"."account_id" = $4 -- the tenant filter, added by the compiler
Every value is a bound parameter, and every column name comes from the declarations.
The tenant filter is added by the engine, independently of the LLM-generated query.
The compiler is responsible for preserving that boundary. How to guarantee this, and how to test it against hostile inputs, is one of the central challenges of this project.
The whole pipeline:
Step 6 is worth a remark: the explanation shown to the user is generated from the validated query, not written by the LLM. No sentence the customer reads comes from the model. That is a deliberate choice, and I'll defend it in a later part.
Why not use what exists?
There are already tools solving parts of this problem.
Blazer provides BI capabilities for Rails, with AI extensions such as SQL generation. QueryLens offers a Rails engine built around natural-language queries and SQL execution. Outside Rails, Cube provides a more comprehensive semantic layer with governed metrics, dimensions and access policies.
These are valid approaches. But DataTeller starts from a different assumption: the person asking questions is an end customer of the SaaS, not an administrator with database access.
I don't want an LLM exploring the database schema and generating SQL. I want developers to explicitly define what users can query, while the engine enforces authorization and tenant isolation independently of the model.
That's the focus of DataTeller:
- No text-to-SQL. The LLM produces structured queries, never executable SQL.
- Explicit data exposure through developer-defined datasets, metrics and dimensions.
- Host application authorization, evaluated server-side.
- Tenant isolation enforced during compilation.
- Native ActiveRecord execution.
I'm not trying to reinvent natural-language analytics. I'm exploring a safer, Rails-native way to expose application data to end users.
Three decisions made up front
| Decision | Chosen | Rejected | Why |
|---|---|---|---|
| LLM output | A validated structured query | Raw SQL | The engine controls the translation to ActiveRecord |
| LLM tooling | Raw HTTP, my own retrieval | An orchestration framework | The point is to understand each layer |
| What is exposed | An explicit, versioned config | Inference at runtime | A new column becomes queryable only through a visible diff |
What this series covers
Roughly in this order, the order in which I solve the problems:
- Scaffolding the engine and a multi-tenant dummy app
- The query format: six design decisions
- The declaration DSL
- Validator, compiler, and why multi-tenant isolation is not an LLM problem
- Execution guards, permissions, aggregate disclosure
- How I test AI data access security
- An evaluation suite, written before the first prompt
- Adding the LLM, and a measured baseline
- RAG: vectors versus hybrid retrieval, with numbers
- And then clarification, explanations, prompt injection, publishing
From the evaluation part onward, the articles carry numbers. Before that, the proof is in the specs
What I don't know yet
- Expressiveness is limited on purpose: one metric per query, filters combined with AND only, no sub-queries. Some real questions won't fit. The evaluation measures how often, and the articles will show it.
- "No result rows go to the provider" is not "no data goes to the provider". The question itself can contain a customer's name. I'll document exactly what leaves the app.
- Security claims need evidence. That is why a test kit is part of the plan, not an afterthought.
Recap
- Text-to-SQL is fine for analysts and worrying in front of customers: raw SQL, isolation delegated to a prompt, schema sent out.
- DataTeller has the LLM fill a structured query that the engine validates and compiles, with the tenant scope injected by the engine.
- This series builds it in the open, and measures what can be measured.
I don't know yet how far I can take this approach, but that's part of the fun. I'll document the decisions, mistakes and trade-offs along the way.
Next up
In part 1, we scaffold the engine: a mountable gem, a dummy app with several tenants and fake data, and Postgres with pgvector, before a single line of AI.
This is part 0 of the **Building DataTeller* series, where I document the process of building a secure AI data access engine for Rails.*
The source code will become publicly available with Part 1.
Next: Part 1 — [Coming soon]
Source code: github.com/SerylLns/data_teller (repository currently private)
Top comments (0)