DEV Community: Nilesh Kumar

Vanna AI Alternatives: When RAG-Trained Text-to-SQL Needs Governance

Nilesh Kumar — Wed, 26 Aug 2026 14:09:20 +0000

Vanna proved you can get surprisingly far by retrieving similar queries and letting the model pattern-match.

Then someone asks a question that resembles nothing in the training set.

How RAG-based text-to-SQL works, and where that ends

Find queries like yours, adapt them. When the question is close to something seen before, it's fast and often right. That's a legitimate architecture and it explains the traction.

Limit	Why it's structural
Corpus coverage	Accuracy tracks how well history covers the question space
Novel joins	No precedent to retrieve, so the model improvises
No entitlement concept	Similarity doesn't know who's asking
Silent corpus decay	Schema evolves; old queries stay in the index

The entitlement row is the one that blocks enterprise deployment. Retrieval has no notion of authorisation — you can filter the result, but the query already ran with whatever credentials the service holds.

The alternative isn't a bigger corpus

More examples raise the floor and don't move the ceiling. The ceiling is that pattern-matching has no model of meaning, so it cannot distinguish "no precedent exists" from "improvise something plausible."

Resolve-then-prove instead of retrieve-then-guess:

Parse language into a typed intent
Resolve entities and metrics against a versioned semantic graph
Prove a join path exists — fail compilation if it doesn't
Inject policy predicates for the caller
Emit dialect-perfect SQL

Step 3 is what a retrieval corpus fundamentally cannot provide.

When Vanna is still the right call

Open source, fast to pilot, full visibility, no procurement. For a proof of concept on a bounded schema it's a reasonable starting point — just budget for the fact that production governance is yours to build.

The full breakdown — the scored alternatives comparison, and the architectural difference between retrieval and resolution — is here:

👉 Vanna AI Alternatives: When RAG-Trained Text-to-SQL Needs Governance

Originally published at colrows.com/blogs/vanna-ai-alternatives

Why Power BI Copilot Delivers Wrong Answers (and What It Costs You)

Nilesh Kumar — Wed, 19 Aug 2026 13:24:44 +0000

Power BI Copilot rarely says "I don't know."

It says a number. The number looks right. That's the expensive part.

The failure isn't syntax

The generated DAX or SQL usually runs fine. The failure is semantic — the right query shape over the wrong interpretation.

Failure	What happens
Ambiguous column	Two tables have `revenue`; it picks one
Misread relationship	Exists in the model, means something else
Lost filter context	Visual-level filters don't survive into the generated logic
No provenance	Nobody can tell afterwards which definition produced the number

An answer you can't reproduce isn't an answer. It's a suggestion with a decimal point.

Why prompting doesn't fix it

Copilot inherits the Power BI semantic model, which means it inherits everything that model leaves ambiguous. If two measures could plausibly answer the question, no amount of prompt engineering tells it which one your CFO means — because that information was never in the model.

You cannot prompt your way past a missing definition. You can only supply the definition.

What closes the gap

Entities and metrics resolved against a typed, versioned graph before any query exists
A proven join path — no valid path, no query, rather than a plausible guess
Policy compiled into the query per person, so different entitlements produce different SQL
An audit trail that reproduces the answer exactly, months later

On real enterprise schemas that difference is not marginal. Raw schema access scored 14.5% in our benchmark; the same model with compiled, governed context scored 98.2%.

The model was never the variable.

The full breakdown — the specific failure modes with examples, what a wrong number costs, and the architectural fix — is here:

👉 Why Power BI Copilot Delivers Wrong Answers (and What It Costs You)

Originally published at colrows.com/blogs/power-bi-copilot-wrong-answers

Snowflake vs. Databricks: Why You Need an Autonomous Semantic Layer

Nilesh Kumar — Wed, 12 Aug 2026 15:40:52 +0000

Your warehouse vendor now sells the semantic layer too. Convenient.

Right up until half the business turns out to live in the other warehouse.

Both shipped something real

Snowflake Semantic Views and Databricks Metric Views are genuine improvements. Definitions live next to the data, governed by the platform's own model, queryable by that platform's AI features.

If your entire estate is on one platform, either will take you a long way.

Almost nobody's estate is on one platform

There's a warehouse, a lakehouse someone piloted, an operational Postgres, and a SaaS system that never got migrated. A warehouse-native semantic layer can only see its own warehouse.

Consequence	What it looks like
Cross-platform joins	Fall back to hand-written SQL
Duplicate definitions	Same metric defined twice, drifting
Per-platform governance	Coverage is per-platform, so gaps are structural
Partial answers	Agent returns a confident number computed over two thirds of the data

That last one is the dangerous failure. Nothing errors. The number is just wrong in a way nobody can see.

The walled garden problem

There is also an incentive question worth naming: a warehouse vendor's semantic layer is, among other things, a reason to move more data into that warehouse. That's not sinister — it's product strategy — but it means "should meaning live inside the warehouse?" is being answered by parties who benefit from one answer.

Meaning has to sit above the warehouse, not inside one of them: one versioned graph across the estate, join paths proven regardless of which engine holds the tables, compile-time policy applied uniformly, and dialect-perfect SQL emitted per target.

Adopt the native features. Just don't mistake them for the layer that spans your business.

The full breakdown — what each platform's feature does and where it stops, plus the cross-estate architecture — is here:

👉 Snowflake vs. Databricks: Why You Need an Autonomous Semantic Layer

Originally published at colrows.com/blogs/snowflake-databricks-semantic-layer

Semantic Layer vs. Knowledge Graph: Choosing Your AI Data Foundation

Nilesh Kumar — Fri, 07 Aug 2026 17:18:30 +0000

Knowledge graph people and semantic layer people keep arguing past each other.

One models how things relate. The other proves what a number means. You probably need both.

The honest division of labour

	Knowledge graph	Semantic layer
Models	Entities and relationships	Metrics, grain, filters, policy
Strong at	Traversal, inference, causality	Aggregation, definition, governance
Weak at	Math and consistent aggregates	Anything not modelled as a metric
Typical failure	Ontology drifts, nobody notices	Question falls outside the metric set
Answers	"How is X connected to Y?"	"What is X, exactly, for this person?"

Ask an agent "why did EMEA churn spike after the pricing change" and you need both: the causal path through the graph, and a metric definition precise enough that the number is defensible.

Where the argument actually goes wrong

Treating them as competing purchases. They operate on different layers of the same problem — one is topology, the other is semantics plus policy.

The real question is neither. It's which layer executes. A graph and a definition are both inert until something compiles them into governed SQL and runs it against your warehouse.

The execution gap

That's the piece most architectures are missing. You can have an immaculate ontology and a complete metric catalogue and still have no answer to:

Does a valid join path exist for this specific question?
Is this caller entitled to these rows?
Can I reproduce this answer in six months?

Intent → context resolution → constrained planning → governed execution. The graph feeds step two. The semantic layer feeds steps two and four. Neither performs step three.

The full breakdown — the division of labour in detail, a worked customer-risk example, and how the two compose — is here:

👉 Semantic Layer vs. Knowledge Graph: Choosing Your AI Data Foundation

Originally published at colrows.com/blogs/semantic-layer-vs-knowledge-graph

Governing AI Agents: Why Compile-Time Security is Mandatory

Nilesh Kumar — Wed, 05 Aug 2026 14:28:24 +0000

The moment you hand an agent warehouse credentials, your governance model is already obsolete.

Prompts aren't policy. Output filters aren't controls. Logs aren't proof.

Every runtime guardrail shares one flaw

It fires after the query ran. The rows moved. You are not preventing exposure — you're annotating it.

That ordering problem is not solvable by making the classifier better. It's solvable by moving the decision earlier.

What actually governs an agent

Step	What happens	What it prevents
Intent parsing	Language → typed intent	Prompt injection reaching SQL
Context resolution	Entities and metrics resolved against a versioned graph	Guessed column semantics
Join proof	Path proven, or compilation fails	Invented relationships
Policy injection	RBAC + ABAC + row/column predicates	Unauthorised rows ever being read
Audit emission	Question → SQL → predicates → result	An unanswerable post-incident review

The agent never sees a table it wasn't entitled to. Not because it behaved well — because the query was never compiled.

The test to run on any vendor

Ask what happens when a user requests something they're not entitled to.

If the answer involves filtering the result set, the data already left the warehouse and you're being sold a logging feature as a security control.

If the answer is "compilation fails," you're looking at governance.

Why compile-time is the only defensible position

Regulators don't accept "the model said so" as a control, and they never have. A number that informed a decision has to be reconstructable months later with the definitions that were in force at the time.

That means policy has to be part of query construction, not a wrapper around it — and the audit artefact has to be the SQL itself, not a description of intent.

The full breakdown — the governance architecture step by step, policy composition across scopes, and the audit format — is here:

👉 Governing AI Agents: Why Compile-Time Security is Mandatory

Originally published at colrows.com/blogs/how-to-govern-ai-agents-that-query-enterprise-data