Modern LLMs are already surprisingly good at generating SQL. The real challenge is giving them enough context to generate SQL that people actually trust.
The Question That Changed Our Approach
Like many teams working on enterprise AI, we spent a lot of time trying to improve prompts.
We experimented with different prompt templates.
We added more examples.
We adjusted temperatures.
We tried different models.
Sometimes the results improved.
Sometimes they didn't.
But one thing became obvious after working with real enterprise databases.
The model wasn't failing because it couldn't write SQL.
It was failing because it didn't understand the organization.
That realization completely changed how we approached enterprise AI.
Instead of asking:
"How can we make the model smarter?"
we started asking:
"How can we make our data easier for AI to understand?"
That turned out to be a much more interesting engineering problem.
Demo Databases Hide the Real Problem
Most AI SQL demos look fantastic.
A clean schema.
Simple foreign keys.
Consistent naming.
Twenty tables.
One customer table.
One orders table.
Everything joins naturally.
Ask:
Which customers spent the most last month?
The model generates SQL.
The SQL executes.
Everyone applauds.
Production systems don't look anything like that.
One customer table becomes twelve.
Revenue exists in multiple systems.
Finance has one definition.
Sales has another.
Marketing has a third.
Historical migrations leave duplicate entities everywhere.
Some relationships exist only because one senior engineer remembers them.
That's the environment enterprise AI actually operates in.
SQL Isn't the Hard Part Anymore
Five years ago, SQL generation itself was a research challenge.
Today, it's becoming a solved problem.
Give GPT-4, Claude, Gemini, or another modern LLM a reasonably organized schema, and they'll usually generate valid SQL.
The bottleneck has moved.
Today's bottleneck looks more like this:
- Which table should the model use?
- Which definition is officially trusted?
- Which join path is correct?
- Which records should be excluded?
- Which business rule overrides the default logic?
Those questions have nothing to do with SQL syntax.
They're questions about organizational knowledge.
We Started Looking at Failed Queries
One exercise turned out to be incredibly useful.
Instead of reviewing successful demos, we collected failed enterprise queries.
Not queries with syntax errors.
Queries that returned technically correct answers nobody trusted.
Patterns appeared almost immediately.
Pattern 1 — Multiple Sources of Truth
The database contained several revenue tables.
Each one existed for a legitimate reason.
Historical reporting.
Operational reporting.
Finance adjustments.
Regional systems.
The AI selected one.
Finance expected another.
Nobody considered the answer reliable.
Pattern 2 — Hidden Business Rules
The SQL was valid.
The numbers were wrong.
Why?
Because experienced analysts always excluded cancelled transactions after reconciliation.
That rule wasn't stored anywhere.
It simply lived inside institutional knowledge.
The AI had no way to discover it.
Pattern 3 — Relationship Ambiguity
The schema suggested one join.
Senior engineers always used another.
Not because the database required it.
Because years of production experience had proven it was safer.
Again, nothing in the schema explained that.
Pattern 4 — Business Language Doesn't Match Database Language
Users ask:
Active customers
The database stores:
customer_status = 7
Business users understand "active."
The database understands integers.
Someone has to bridge that gap.
Better Prompts Didn't Solve These Problems
Our first instinct was exactly what most teams try.
Improve the prompt.
Add examples.
Explain business terminology.
Increase context length.
Eventually we realized we were embedding organizational knowledge inside prompts.
That doesn't scale.
Every application duplicates the same context.
Every prompt becomes longer.
Every update requires editing multiple systems.
Eventually prompts become documentation.
And documentation always drifts.
We Needed Shared Context Instead
The breakthrough came when we stopped thinking about prompts as the primary source of intelligence.
Instead, we began thinking about reusable context.
Things every AI application should understand before generating SQL.
Examples include:
- trusted business definitions
- approved metrics
- validated relationships
- preferred join paths
- business terminology
- governance rules
Instead of rebuilding this knowledge for every prompt, why not maintain it once?
That idea changed our engineering priorities.
The Architecture Started Looking Different
Originally our architecture looked familiar.
User
↓
LLM
↓
Database
Simple.
Elegant.
Wrong.
Eventually it evolved into something closer to this.
User
↓
AI Application
↓
Context Layer
• Semantic Definitions
• Relationship Discovery
• Business Metrics
• Governance Rules
↓
LLM
↓
Database
Notice something interesting.
The model didn't disappear.
It simply stopped carrying all the responsibility.
Context Is Becoming Infrastructure
I think many engineering teams still treat context as application logic.
Every AI assistant maintains its own prompts.
Its own examples.
Its own business rules.
Its own metadata.
That works when you have one assistant.
It becomes a maintenance nightmare when you have ten.
Or fifty.
Or hundreds.
Context shouldn't live inside applications.
It should live inside infrastructure.
Just like authentication.
Just like monitoring.
Just like APIs.
Shared.
Governed.
Reusable.
Data Engineering Is Quietly Changing
This has also changed how I think about data engineering.
Traditionally, data engineers focused on things like:
- ingestion
- transformation
- storage
- performance
- orchestration
Those responsibilities still matter.
But AI introduces another responsibility.
Preparing data for machine reasoning.
That includes questions like:
Can AI understand this metric?
Can AI safely join these tables?
Can AI explain where this number came from?
Can AI distinguish between similar concepts?
Those weren't traditional data engineering problems.
They're becoming increasingly important now.
The New Question I Ask
Whenever someone tells me they're building enterprise AI, I no longer ask:
Which model are you using?
Instead I ask:
Where does your AI get its business context?
Sometimes the answer is:
"Our prompts."
Sometimes it's:
"Our documentation."
Sometimes nobody knows.
That's usually where the biggest opportunity exists.
What I Think Will Matter Over the Next Five Years
Every year models become better.
That's almost guaranteed.
But every enterprise is using roughly the same foundation models.
Competitive advantage probably won't come from choosing Model A instead of Model B.
It will come from something much harder to copy.
The quality of organizational context.
Companies that organize business knowledge into reusable infrastructure will build AI systems that are more reliable, easier to maintain, and far more scalable.
Companies that don't will continue fighting the same problems with increasingly sophisticated prompts.
Final Thoughts
Looking back, the biggest shift wasn't technical.
It was conceptual.
We stopped treating enterprise AI as a language problem.
We started treating it as a data infrastructure problem.
LLMs are becoming exceptional reasoning engines.
What they still lack is trusted organizational context.
And I increasingly believe that's where the next generation of enterprise engineering will focus.
Not writing better prompts.
Not switching models every six months.
But building data infrastructure that allows every AI application to reason from the same trusted foundation.
When that foundation exists, SQL generation becomes almost the easy part.
Without it, even the smartest model is simply making educated guesses.
`
Top comments (0)