Last month, we were working on a feature for one of our projects. The requirement was to build a natural-language interface that would allow users to query a fairly large MySQL dataset without having to know anything about the underlying database structure.
The idea was simple enough. A user asks a question in plain English, the application understands what information they're looking for, and the corresponding records or analytics are returned.
Now, as most crash courses and quick development reels would suggest, we could simply pass the database schema and the user's question to an LLM, ask it to generate SQL, execute the query and return the result.
And for a POC, that would probably be enough to demonstrate the concept.
But there's always been a striking difference between validating an idea through a POC and making sure the same feature continues to behave reliably when it becomes part of an application used by different people.
We were looking at a similar situation. The underlying data was changing regularly, users had different access scopes, and the questions themselves could vary considerably. Some users might only need an aggregate count, while others might want to inspect records across different operational units.
Before moving towards wider usage, we wanted to understand how we could approach this without making every database interaction completely dependent on the LLM.
Understanding what we were trying to solve
Before getting into the implementation, I want to clarify something.
We weren't trying to discover a faster way to retrieve records from MySQL. Database indexing, query planners, pagination, caching and application-level data access are already established approaches. We would need those regardless of whether an LLM was involved.
What interested us was the additional layer introduced by natural language.
Consider a conventional API. The frontend might send a request containing a region identifier, a status filter and a date range. The backend knows what those parameters mean and can construct a corresponding query.
With a natural-language interface, the user might write something like, "How many requests were pending by the end of last month?"
There are no predefined query parameters being supplied by the user. The application now has to understand the requested metric, interpret the time period, determine the appropriate data scope and prepare an execution request.
We can look at the overall application flow like this:
Once we isolated this part of the application, the discussion became less about which LLM to use and more about how much responsibility we should give it.
Looking into NL-to-SQL
We started exploring existing natural-language-to-SQL approaches to understand how similar requirements were being addressed.
The basic idea behind NL-to-SQL is straightforward: translate a question expressed in natural language into a SQL query that can retrieve the requested information.
The interesting part comes when we start thinking about what the model needs in order to generate that query.
An LLM understands language reasonably well, but it doesn't automatically understand our database. It needs context about available tables, columns, relationships and sometimes the business meaning behind those structures.
Take a question such as:
"Show me the requests updated this week that are still pending review."
A model could generate syntactically valid SQL for this. But which table should it use? What does pending review mean in this application? Which timestamp represents the last update? Does the user have access to every operational unit represented in that table?
These questions can't be answered reliably just by asking the model to write better SQL.
One approach would be to provide the entire database schema with every request. But for a large relational database, this introduces its own problems. Most of that schema may be irrelevant to the user's question, and the model still has to identify the appropriate tables and relationships from a much larger context.
This led us towards a more controlled approach using a schema card.
We built a schema-card layer to give the LLM only the database context relevant to the question. Instead of passing the entire schema, we used our baseline schema definition to identify role-permitted tables and rank them based on their relevance to the user's question.
The card included column information, saved query examples, and metadata identifying protected fields and scope-related columns. We also kept live database records out of this step, since the model only needed enough context to propose SQL.
The current selection uses lexical matching, which works with known terminology but can struggle when users describe the same business concept differently. Since the card also depends on a baseline schema, keeping it aligned with database changes is another area we have to account for.
What happens after the LLM generates SQL?
Suppose the model successfully generates a query. What should happen next?
In a simple demonstration, we might execute the generated SQL immediately and display the results.
But for our feature, we had to consider more than whether the statement was syntactically correct.
A SQL statement can be perfectly valid and still reference a restricted table, return sensitive information or perform an operation that the interface was never supposed to allow.
For this reason, we limited the model's responsibility to proposing the query.
The proposal followed a structured output contract. Instead of returning arbitrary text containing SQL somewhere in the response, the model was expected to produce JSON identifying the response type and, where appropriate, containing a SQL statement.
We then parsed that response and evaluated the generated SQL using a SQL parser and classifier.
The classifier inspected the statement's structure, determined its operation category and applied restrictions around permitted database operations. The surrounding policy checks also considered the tables referenced by the query and limits on query complexity.
We used sqlglot for these checks, which allowed us to inspect the SQL structurally rather than relying on string matching.
There is a practical difference between checking whether a string starts with SELECT and parsing the statement to understand what it contains. SQL can include nested expressions, common table expressions, functions and multiple kinds of statements. A parser gives the application a better foundation for enforcing the supported query contract.
Of course, passing those checks doesn't prove that the model selected the correct business records or interpreted every filter correctly. Structural SQL validation and semantic correctness are separate concerns, and the latter still needs evaluation.
Once the proposed query passed the initial checks, we showed it to the user.
The user could review the statement and explicitly choose to run it. The application then reclassified and rechecked the stored SQL before sending it through our existing governed read-execution path.
That path already applied execution controls such as result limits, a statement watchdog, response masking and audit records.
I think the separation between proposing SQL and executing SQL deserves a little attention.
From a frontend perspective, we can still make the interaction feel like a single chat conversation. The user asks a question, sees the proposed query and chooses whether to execute it.
But internally, proposal and execution are separate operations.
That gives the application an opportunity to persist the proposal, retain an audit record and recheck the statement when execution is requested. The model doesn't have to remain active while the user is deciding.
It also doesn't necessarily mean that the model is called twice. The LLM may be involved only in generating the SQL, after which the application handles execution and presents the returned data.
We still need the backend restrictions even with human review. A user approving a query shouldn't become a substitute for proper access control.
Where the grammar layer became useful
While exploring the NL-to-SQL approach, we also had another category of questions to deal with.
Consider:
"How many requests are pending in this region?"
The question is written in natural language, but the requested information is something the application may already understand as a supported metric.
We don't necessarily need the model to construct arbitrary SQL for that.
We built a deterministic grammar layer backed by controlled vocabulary to interpret supported analytics questions and convert them into structured intents containing the required metric, dimensions, scope and time information.
Once an intent was resolved and validated, the analytics service could identify the relevant data partitions and read the corresponding snapshot aggregates.
This is where we began separating natural-language analytics from flexible NL-to-SQL.
For known analytics questions, the system could resolve the request without calling an LLM. If grammar couldn't understand an otherwise permitted request, a configured model provider could be used as a fallback to propose the structured intent.
That proposal still passed through vocabulary, scope and permission validation before execution. A permission refusal wasn't supposed to become an opportunity to try the question again through the model.
The grammar layer naturally has its limitations. It can only interpret questions covered by the supported vocabulary and expressions. Maintaining that vocabulary also becomes part of the application work.
So we weren't attempting to handle every possible question through deterministic rules. We were using them for questions whose meaning and execution contract were already known.
There was also a difference in how results came back.
For analytics, the service returned structured aggregate values from the database rather than passing the result into another model call for narration. It could also include information about the contributing data partitions and errors encountered during retrieval.
This was particularly relevant because the analytics service could retrieve information from more than one backend partition. If one target failed, the response could preserve the available result along with information about the failure rather than silently presenting it as a complete answer.
We also had to consider questions that belonged outside structured database querying. A question about an operational procedure, for example, would be better suited to document retrieval and possibly RAG than SQL generation over operational records. That distinction helped us think about where the natural-language querying feature should have its boundaries.
What we learned from the implementation review
After understanding the query flow, we also looked at what happened further downstream.
One observation was that our natural-language analytics path queried its backend partitions sequentially. The NL-to-SQL execution path already had a concurrent multi-target read mechanism, but the deterministic analytics path hadn't adopted the same approach.
This raised a practical performance question.
Even if grammar resolution avoids an LLM call, the total response time still depends on everything that happens after the intent has been resolved. Multiple database connections, sequential reads and network latency can contribute significantly to the overall request duration.
We already had functional tests covering grammar resolution, generated SQL classification and the successful execution paths. However, we hadn't yet measured how the complete interface would behave under higher concurrency or across a larger number of backend partitions.
Another area worth examining was the absence of a request-wide deadline. Individual database statements had timeout controls, but a request involving multiple operations could still spend considerable time progressing through those individually bounded calls.
These observations gave us a better picture of the work beyond the model interaction itself.
Understanding the performance of this feature would require measurements around grammar resolution rates, model fallback frequency, database execution time and partial failures.
We didn't have a clean benchmark that would let us claim a particular latency or cost improvement from the design, so I'd rather leave those numbers out than estimate them.
Where this leaves me
Looking at the completed query flow, I found it interesting how little of the overall execution actually needed to depend on the LLM. The model still helped us handle flexible questions, but we had enough information within the application to manage known analytics, SQL validation and database execution separately.
There's still more to understand about how this behaves under load, particularly across multiple data partitions. For now, the experience gave me a better appreciation of the engineering decisions that sit between a working NL-to-SQL demo and an application feature.# What Building an LLM-Offloaded Query Layer Taught Me
Last month, we were working on a feature for one of our projects. The requirement was to build a natural-language interface that would allow users to query a fairly large MySQL dataset without having to know anything about the underlying database structure.
The idea was simple enough. A user asks a question in plain English, the application understands what information they're looking for, and the corresponding records or analytics are returned.
Now, as most crash courses and quick development reels would suggest, we could simply pass the database schema and the user's question to an LLM, ask it to generate SQL, execute the query and return the result.
And for a POC, that would probably be enough to demonstrate the concept.
But there's always been a striking difference between validating an idea through a POC and making sure the same feature continues to behave reliably when it becomes part of an application used by different people.
We were looking at a similar situation. The underlying data was changing regularly, users had different access scopes, and the questions themselves could vary considerably. Some users might only need an aggregate count, while others might want to inspect records across different operational units.
Before moving towards wider usage, we wanted to understand how we could approach this without making every database interaction completely dependent on the LLM.
Understanding what we were trying to solve
Before getting into the implementation, I want to clarify something.
We weren't trying to discover a faster way to retrieve records from MySQL. Database indexing, query planners, pagination, caching and application-level data access are already established approaches. We would need those regardless of whether an LLM was involved.
What interested us was the additional layer introduced by natural language.
Consider a conventional API. The frontend might send a request containing a region identifier, a status filter and a date range. The backend knows what those parameters mean and can construct a corresponding query.
With a natural-language interface, the user might write something like, "How many requests were pending by the end of last month?"
There are no predefined query parameters being supplied by the user. The application now has to understand the requested metric, interpret the time period, determine the appropriate data scope and prepare an execution request.
We can look at the overall application flow like this:
[Diagram 1 — End-to-end HLD: Natural-language interface → Application interpretation layer ↔ Optional LLM → Governed execution → Database → Response.]
Once we isolated this part of the application, the discussion became less about which LLM to use and more about how much responsibility we should give it.
Looking into NL-to-SQL
We started exploring existing natural-language-to-SQL approaches to understand how similar requirements were being addressed.
The basic idea behind NL-to-SQL is straightforward: translate a question expressed in natural language into a SQL query that can retrieve the requested information.
The interesting part comes when we start thinking about what the model needs in order to generate that query.
An LLM understands language reasonably well, but it doesn't automatically understand our database. It needs context about available tables, columns, relationships and sometimes the business meaning behind those structures.
Take a question such as:
"Show me the requests updated this week that are still pending review."
A model could generate syntactically valid SQL for this. But which table should it use? What does pending review mean in this application? Which timestamp represents the last update? Does the user have access to every operational unit represented in that table?
These questions can't be answered reliably just by asking the model to write better SQL.
One approach would be to provide the entire database schema with every request. But for a large relational database, this introduces its own problems. Most of that schema may be irrelevant to the user's question, and the model still has to identify the appropriate tables and relationships from a much larger context.
This led us towards a more controlled approach using a schema card.
We built a schema-card layer to give the LLM only the database context relevant to the question. Instead of passing the entire schema, we used our baseline schema definition to identify role-permitted tables and rank them based on their relevance to the user's question.
The card included column information, saved query examples, and metadata identifying protected fields and scope-related columns. We also kept live database records out of this step, since the model only needed enough context to propose SQL.
The current selection uses lexical matching, which works with known terminology but can struggle when users describe the same business concept differently. Since the card also depends on a baseline schema, keeping it aligned with database changes is another area we have to account for.
What happens after the LLM generates SQL?
Suppose the model successfully generates a query. What should happen next?
In a simple demonstration, we might execute the generated SQL immediately and display the results.
But for our feature, we had to consider more than whether the statement was syntactically correct.
A SQL statement can be perfectly valid and still reference a restricted table, return sensitive information or perform an operation that the interface was never supposed to allow.
For this reason, we limited the model's responsibility to proposing the query.
The proposal followed a structured output contract. Instead of returning arbitrary text containing SQL somewhere in the response, the model was expected to produce JSON identifying the response type and, where appropriate, containing a SQL statement.
We then parsed that response and evaluated the generated SQL using a SQL parser and classifier.
The classifier inspected the statement's structure, determined its operation category and applied restrictions around permitted database operations. The surrounding policy checks also considered the tables referenced by the query and limits on query complexity.
We used sqlglot for these checks, which allowed us to inspect the SQL structurally rather than relying on string matching.
There is a practical difference between checking whether a string starts with SELECT and parsing the statement to understand what it contains. SQL can include nested expressions, common table expressions, functions and multiple kinds of statements. A parser gives the application a better foundation for enforcing the supported query contract.
Of course, passing those checks doesn't prove that the model selected the correct business records or interpreted every filter correctly. Structural SQL validation and semantic correctness are separate concerns, and the latter still needs evaluation.
Once the proposed query passed the initial checks, we showed it to the user.
The user could review the statement and explicitly choose to run it. The application then reclassified and rechecked the stored SQL before sending it through our existing governed read-execution path.
That path already applied execution controls such as result limits, a statement watchdog, response masking and audit records.
[Diagram 2 — Zoomed DFD: Grammar-first analytics, optional LLM fallback, governed SQL proposal and execution.]
I think the separation between proposing SQL and executing SQL deserves a little attention.
From a frontend perspective, we can still make the interaction feel like a single chat conversation. The user asks a question, sees the proposed query and chooses whether to execute it.
But internally, proposal and execution are separate operations.
That gives the application an opportunity to persist the proposal, retain an audit record and recheck the statement when execution is requested. The model doesn't have to remain active while the user is deciding.
It also doesn't necessarily mean that the model is called twice. The LLM may be involved only in generating the SQL, after which the application handles execution and presents the returned data.
We still need the backend restrictions even with human review. A user approving a query shouldn't become a substitute for proper access control.
Where the grammar layer became useful
While exploring the NL-to-SQL approach, we also had another category of questions to deal with.
Consider:
"How many requests are pending in this region?"
The question is written in natural language, but the requested information is something the application may already understand as a supported metric.
We don't necessarily need the model to construct arbitrary SQL for that.
We built a deterministic grammar layer backed by controlled vocabulary to interpret supported analytics questions and convert them into structured intents containing the required metric, dimensions, scope and time information.
Once an intent was resolved and validated, the analytics service could identify the relevant data partitions and read the corresponding snapshot aggregates.
This is where we began separating natural-language analytics from flexible NL-to-SQL.
For known analytics questions, the system could resolve the request without calling an LLM. If grammar couldn't understand an otherwise permitted request, a configured model provider could be used as a fallback to propose the structured intent.
That proposal still passed through vocabulary, scope and permission validation before execution. A permission refusal wasn't supposed to become an opportunity to try the question again through the model.
The grammar layer naturally has its limitations. It can only interpret questions covered by the supported vocabulary and expressions. Maintaining that vocabulary also becomes part of the application work.
So we weren't attempting to handle every possible question through deterministic rules. We were using them for questions whose meaning and execution contract were already known.
There was also a difference in how results came back.
For analytics, the service returned structured aggregate values from the database rather than passing the result into another model call for narration. It could also include information about the contributing data partitions and errors encountered during retrieval.
This was particularly relevant because the analytics service could retrieve information from more than one backend partition. If one target failed, the response could preserve the available result along with information about the failure rather than silently presenting it as a complete answer.
We also had to consider questions that belonged outside structured database querying. A question about an operational procedure, for example, would be better suited to document retrieval and possibly RAG than SQL generation over operational records. That distinction helped us think about where the natural-language querying feature should have its boundaries.
What we learned from the implementation review
After understanding the query flow, we also looked at what happened further downstream.
One observation was that our natural-language analytics path queried its backend partitions sequentially. The NL-to-SQL execution path already had a concurrent multi-target read mechanism, but the deterministic analytics path hadn't adopted the same approach.
This raised a practical performance question.
Even if grammar resolution avoids an LLM call, the total response time still depends on everything that happens after the intent has been resolved. Multiple database connections, sequential reads and network latency can contribute significantly to the overall request duration.
We already had functional tests covering grammar resolution, generated SQL classification and the successful execution paths. However, we hadn't yet measured how the complete interface would behave under higher concurrency or across a larger number of backend partitions.
Another area worth examining was the absence of a request-wide deadline. Individual database statements had timeout controls, but a request involving multiple operations could still spend considerable time progressing through those individually bounded calls.
These observations gave us a better picture of the work beyond the model interaction itself.
Understanding the performance of this feature would require measurements around grammar resolution rates, model fallback frequency, database execution time and partial failures.
We didn't have a clean benchmark that would let us claim a particular latency or cost improvement from the design, so I'd rather leave those numbers out than estimate them.
Where this leaves me
Looking at the completed query flow, I found it interesting how little of the overall execution actually needed to depend on the LLM. The model still helped us handle flexible questions, but we had enough information within the application to manage known analytics, SQL validation and database execution separately.
There's still more to understand about how this behaves under load, particularly across multiple data partitions. For now, the experience gave me a better appreciation of the engineering decisions that sit between a working NL-to-SQL demo and an application feature.
Top comments (1)
This is my first technical blog on this platform, and I'm looking forward to documenting more of my engineering experiences here.
I'd really appreciate any feedback, suggestions, or different perspectives on the approach. Always happy to learn and improve through discussions!
Thanks for reading! :)