DEV Community: ArisynData

Why RBAC Alone Isn't Enough for Enterprise Data Agents

ArisynData — Thu, 10 Sep 2026 02:21:17 +0000

A user can be blocked from a sensitive column and still receive sensitive information derived from data they are allowed to access.

That changes the authorization problem for enterprise data agents.

Traditional access control asks:

Can this user read this database object?

An AI analytics system also needs to ask:

Is this user allowed to receive what the system can infer from those objects?

Consider a simple example.

A user cannot access:

employee.salary

But the same user can access:

department.total_cost
department.employee_count

A capable data agent can derive:

estimated_average_salary
=
department.total_cost
/
department.employee_count

No forbidden salary column was queried.

The database permission model may have worked perfectly.

The answer may still disclose information the policy intended to protect.

This is why:

Table access ≠ Answer access.

RBAC Still Matters

This is not an argument against role-based access control.

RBAC remains a critical foundation.

A typical model might define:

Role: Sales Manager

ALLOW:
  customer
  sales_order
  product
  regional_revenue

DENY:
  employee.salary
  payroll
  compensation_detail

At the database layer, those controls should continue to be enforced.

The problem is that an AI agent introduces several stages above the database:

Natural Language
      ↓
Intent Resolution
      ↓
Semantic Resolution
      ↓
Context Retrieval
      ↓
Relationship Planning
      ↓
SQL Generation
      ↓
Execution
      ↓
Answer Generation

Authorization therefore has more surfaces than a traditional application issuing predefined SQL.

The Inference Gap

Let's formalize the salary example.

Suppose policy says:

{
  "resource": "employee.salary",
  "action": "read",
  "effect": "deny"
}

But:

{
  "resource": "department.total_cost",
  "action": "read",
  "effect": "allow"
}

and:

{
  "resource": "department.employee_count",
  "action": "read",
  "effect": "allow"
}

The agent creates:

f(total_cost, employee_count)
→ estimated_average_salary

Every input is authorized.

The derived concept may not be.

Call this the inference gap:

Authorized Inputs
      ↓
Reasoning / Aggregation
      ↓
Restricted Information

Traditional object-level authorization may not express that boundary.

Add Semantic Authorization

Users ask questions in business concepts.

So policy should increasingly understand business concepts too.

Instead of governing only:

employee.salary

define a semantic concept:

{
  "concept": "employee_compensation",
  "direct_access": "deny",
  "derived_access": "deny"
}

Now a request such as:

What is the average salary of the engineering team?

can be resolved first:

Intent
→ Employee Compensation

Then evaluated:

Employee Compensation
→ DENY

before SQL generation begins.

This is semantic authorization.

It lets policy operate at the same abstraction level as the user's question.

Authorization Should Start Before SQL Generation

A common architecture is:

Question
   ↓
Retrieve Schema
   ↓
Generate SQL
   ↓
Database Permission Check
   ↓
Execute

The problem is that the model may already have received context it should not use.

A stronger pipeline is:

Question
      ↓
Identity
      ↓
Intent Resolution
      ↓
Semantic Policy
      ↓
Authorized Context
      ↓
Authorized Relationship Graph
      ↓
Query Planning
      ↓
SQL Generation
      ↓
Database Enforcement
      ↓
Answer Policy

The important change is:

Authorization constrains reasoning context before it constrains execution.

Build an Authorized Context Resolver

Imagine the enterprise semantic layer contains:

Revenue
Gross Margin
Customer Risk
Payroll Cost
Employee Compensation
Product Profitability

A generic context retriever might return all concepts semantically related to the question.

That is risky.

Instead:

Candidate Context
      ↓
Identity + Policy
      ↓
Authorized Context
      ↓
LLM

Pseudocode:

def resolve_authorized_context(question, user):
    intent = resolve_intent(question)

    candidates = retrieve_semantic_context(intent)

    allowed = [
        item for item in candidates
        if policy.can_use(user, item)
    ]

    return allowed

The real implementation will need stronger policy semantics, but the architectural boundary matters.

Do not give the model unauthorized context and hope the final SQL check fixes everything.

Relationships Need Authorization Too

Suppose relationship discovery finds:

Employee
   ↓
Department
   ↓
Cost Center
   ↓
Financial Cost

The path is structurally valid.

But a Sales user may not be allowed to traverse it.

So distinguish:

Trusted Relationship

from:

Authorized Relationship

A relationship object could carry policy metadata:

{
  "source": "department",
  "target": "cost_center",
  "status": "trusted",

  "policy": {
    "allowed_roles": [
      "finance",
      "hr"
    ]
  }
}

Then query planning uses a user-specific graph:

def authorized_graph(graph, user):
    return graph.filter(
        lambda edge: policy.can_traverse(user, edge)
    )

This gives us another useful rule:

Valid relationship ≠ Authorized relationship.

Query Planning Should Operate on the Authorized Graph

Assume the full relationship graph contains:

Customer ─ Order ─ Payment
Employee ─ Department ─ Cost Center
Supplier ─ Contract ─ Pricing

For a Sales user:

Customer ─ Order ─ Payment

may be available.

But:

Employee ─ Department ─ Cost Center

may be removed from the planning graph.

The SQL generator never sees that path.

That is safer than generating the query first and rejecting it later.

Direct Access and Derived Access Are Different Policies

Some concepts need two policy dimensions.

Example:

{
  "concept": "customer_credit_risk",

  "direct_access": {
    "roles": ["risk", "finance"]
  },

  "derived_access": {
    "roles": ["risk", "finance"]
  }
}

Why distinguish them?

Because an organization might allow:

Department Cost

but restrict:

Individual Compensation

or permit individual operational metrics while restricting a derived risk score.

The derived concept may have different sensitivity from its inputs.

Answer-Level Policy Is the Final Boundary

Even with pre-query authorization, a final result check is useful.

The pipeline may produce:

SQL Valid               ✓
Database Access         ✓
Relationship Valid      ✓
Execution               ✓
Answer Policy           ✕

The system should not return the result.

Conceptually:

result = execute(sql)

answer_concepts = classify_result_semantics(
    question=question,
    plan=query_plan,
    result=result
)

for concept in answer_concepts:
    if not policy.can_receive(user, concept):
        raise PolicyDenied(concept)

This is not simply SQL validation.

It is answer validation.

Why Result Classification Is Hard

A result rarely arrives labeled:

"This is sensitive compensation information."

The system needs evidence from:

Original Intent
Resolved Semantic Concepts
Selected Metrics
Query Plan
Aggregations
Relationship Path
Output Columns

That means answer governance should not be implemented as a disconnected moderation step.

It should preserve semantic provenance throughout query execution.

Preserve a Semantic Query Plan

Instead of storing only SQL:

SELECT ...

store a structured plan:

{
  "intent": "average employee compensation",

  "concepts": [
    "employee_compensation"
  ],

  "metrics": [
    "department_total_cost",
    "employee_count"
  ],

  "derived_metric": {
    "name": "estimated_average_salary",
    "expression": "department_total_cost / employee_count"
  },

  "relationships": [
    "employee -> department"
  ]
}

Now authorization has something meaningful to evaluate.

This is another reason production Text-to-SQL should not be treated as:

Question → SQL

The intermediate query plan matters.

Add Policy to the Query Plan

A policy-aware plan might look like:

{
  "concept": "employee_compensation",

  "authorization": {
    "semantic_access": false,
    "data_access": true,
    "relationship_access": true,
    "answer_access": false
  },

  "decision": "deny"
}

The system can stop before execution.

For a different user:

{
  "role": "HR Partner",
  "decision": "allow"
}

The same natural-language question can therefore produce different authorized query plans.

Policy-Aware Clarification

Authorization can also affect clarification.

Suppose a user asks:

Show employee cost.

The system resolves two candidates:

Department Operating Cost
Employee Compensation

The user is authorized for the first but not the second.

A naive clarification UI might reveal both options.

That itself may leak sensitive semantic structure.

Instead, candidate generation should be policy-filtered:

Candidate Concepts
      ↓
Policy Filter
      ↓
Allowed Clarification Options

Authorization therefore affects not only execution but also what the system is allowed to discuss.

Explain Denials in Business Terms

A natural-language interface should not return:

SQLSTATE 42501
permission denied

when the real issue is semantic.

A better response might be:

This question would reveal restricted employee compensation information. You can query department-level operating cost, but not derived salary information.

This improves both security and user experience.

The system can explain:

What category is restricted
What level is allowed
What alternative question is permitted

without exposing sensitive details.

Don't Try to Solve Every Possible Inference

There is an important practical limit.

If two harmless numbers can theoretically be combined into sensitive information, trying to enumerate every possible derivation can become impossible.

So focus governance on high-impact semantic concepts.

Examples:

Compensation
Protected Personal Information
Credit Risk
Confidential Pricing
Sensitive Forecasts
Health Information

Then model known derivation patterns and business policies around those concepts.

The goal is not mathematical prevention of all inference.

It is business-risk-aware governance.

A Practical Policy Model

One possible abstraction:

concept: employee_compensation

sensitivity: restricted

direct_access:
  allow:
    - hr
    - executive

derived_access:
  allow:
    - hr
    - executive

related_metrics:
  - salary
  - bonus
  - estimated_average_salary

restricted_derivations:
  - department_cost / employee_count

Another:

relationship:
  source: employee
  target: cost_center

trusted: true

traverse:
  allow:
    - finance
    - hr

This makes policy part of semantic and relationship metadata rather than an afterthought.

Audit the Reasoning Path

When a query is allowed or denied, log why.

Example:

{
  "user": "sales_manager",
  "question": "What is the average salary in engineering?",

  "resolved_intent": "employee_compensation",

  "policy_decision": "deny",

  "reason": "derived_access_not_allowed"
}

For allowed queries, record:

Semantic concepts used
Relationship path
Metrics selected
Policy decisions
Generated SQL

This creates an audit trail that is much more useful than logging SQL alone.

Test Authorization With Adversarial Questions

Enterprise data-agent security testing should include inference cases.

For example:

Direct request

Show individual employee salaries.

Expected:

DENY

Derived request

Divide engineering payroll cost by headcount.

Expected:

DENY

Allowed aggregate

Show total engineering operating cost.

Expected:

ALLOW

Relationship traversal

Join employee records with cost-center financials.

Expected:

ROLE DEPENDENT

These tests reveal whether the system governs meaning or only columns.

What to Measure

Useful authorization metrics could include:

Direct Sensitive Query Block Rate
Derived Sensitive Query Block Rate
False Denial Rate
Unauthorized Relationship Block Rate
Policy Explanation Accuracy

A secure system that denies every complex query is not useful.

The goal is:

Maximum useful access
within authorized semantic boundaries

A Reference Architecture

Putting everything together:

                 Natural Language
                        ↓
                     Identity
                        ↓
                Intent Resolution
                        ↓
                Semantic Policy
                        ↓
               Authorized Context
                        ↓
          Authorized Relationship Graph
                        ↓
                 Query Planning
                        ↓
                Policy-Aware Plan
                        ↓
                 SQL Generation
                        ↓
              Database Enforcement
                        ↓
                    Execution
                        ↓
              Answer Policy Check
                        ↓
             Return / Explain / Deny

RBAC remains underneath this architecture.

The new layers do not replace database security.

They extend governance into the reasoning process.

Final Thoughts

Enterprise data agents make databases easier to use because users no longer need to know schemas or SQL.

That abstraction is powerful.

It also means users can ask for information without knowing which fields, tables, joins, or calculations the agent will use.

So authorization has to follow the same abstraction upward.

From:

Who can access this table?

to:

Who can use this business concept?

and finally:

Who can receive this derived answer?

That is why RBAC alone is not the whole solution for AI-powered analytics.

Keep RBAC.

Keep row- and column-level controls.

But add governance around:

Intent
Semantics
Relationships
Derivations
Answers

Because:

Table access ≠ Answer access.

And:

Valid relationship ≠ Authorized relationship.

A production data agent should not only know how to find an answer.

It should know whether it is allowed to reveal it.

The Semantic Cold Start Problem in Enterprise Data Agents

ArisynData — Mon, 07 Sep 2026 22:00:00 +0000

Connecting an AI agent to a database is easy. Bootstrapping enough enterprise context for it to use that database correctly is the real engineering problem.

A new data agent can inspect a schema almost immediately:

tables
columns
data types
primary keys
sample values

Then a user asks:

What was revenue by product code last quarter?

The schema alone does not tell the agent:

What does "Revenue" mean here?
Which "product code" does the business use?
Which fields are authoritative?
How should the required tables be connected?
Which relationship candidates are trusted?

This gap between data access and business understanding is what I call the semantic cold start problem.

For production data agents, reducing that cold-start cost may matter as much as improving the underlying LLM.

1. Model the Problem as Knowledge Bootstrapping

A useful bootstrap pipeline looks like:

Connect Data
     ↓
Schema Discovery
     ↓
Relationship Discovery
     ↓
Semantic Candidate Generation
     ↓
Existing Knowledge Import
     ↓
Human Confirmation
     ↓
Runtime Clarification
     ↓
Knowledge Persistence
     ↓
Reuse

The important point is that no single step is expected to magically understand the enterprise.

Instead, the system progressively converts different forms of evidence into reusable context.

2. Start With a Machine-Readable Inventory

The first layer is deterministic.

For each data source, collect technical metadata such as:

{
  "table": "finance_revenue",
  "columns": [
    {
      "name": "recognized_amount",
      "type": "decimal"
    },
    {
      "name": "recognition_date",
      "type": "date"
    },
    {
      "name": "order_id",
      "type": "bigint"
    }
  ]
}

This makes the agent schema-aware.

But schema awareness is only Level 1.

It tells the system what exists, not what the business means.

3. Discover Relationship Candidates

The next problem is connectivity.

Enterprise databases often have incomplete foreign-key definitions, so relationship discovery may need to combine several signals:

Declared constraints
Column-name similarity
Compatible data types
Uniqueness
Value overlap
Value inclusion
Existing metadata

For example:

account.customer_id
customer.customer_id

may be a strong relationship candidate even if no FK constraint exists.

A candidate could be represented as:

{
  "source": "account.customer_id",
  "target": "customer.customer_id",
  "evidence": {
    "type_compatible": true,
    "name_similarity": 0.91,
    "inclusion": 0.98,
    "target_uniqueness": 0.99
  },
  "confidence": 0.96,
  "status": "candidate"
}

The important distinction is:

discovered relationship
≠
trusted relationship

Discovery reduces manual work.

Validation turns a candidate into enterprise knowledge.

4. Generate Semantic Candidates, Not Semantic Truth

Now consider the business term:

Product Code

The schema may contain:

product_master.material_id
inventory.item_code
sales_detail.sku_no
product_dim.prod_master_id

An LLM or semantic retrieval system can rank candidates:

{
  "term": "Product Code",
  "candidates": [
    {
      "field": "product_master.material_id",
      "score": 0.88
    },
    {
      "field": "inventory.item_code",
      "score": 0.84
    },
    {
      "field": "sales_detail.sku_no",
      "score": 0.79
    }
  ]
}

This is useful.

But it should not automatically become:

Product Code = material_id

A semantic candidate is evidence.

A business mapping is a governed decision.

That separation prevents model confidence from being mistaken for organizational truth.

5. Import What the Enterprise Already Knows

Cold start should not mean starting from zero.

Useful knowledge may already exist in:

Business glossaries
Metric definitions
Data catalogs
BI models
Dashboard logic
Documentation
Existing SQL

For example, Finance may already define:

metric:
  name: Revenue

definition:
  Recognized revenue for financial reporting

source:
  table: finance_revenue
  field: recognized_amount

time_field:
  recognition_date

If this definition already exists, the agent should reuse it instead of asking users to redefine Revenue.

A good bootstrap system therefore needs both:

Discovery

and:

Import

6. Use Humans for Confirmation, Not Exhaustive Modeling

Some mappings cannot be inferred safely.

Instead of asking a data team to document everything manually, let the system narrow the decision first.

Example:

"Product Code" has 3 likely mappings:

1. product_master.material_id
2. inventory.item_code
3. sales_detail.sku_no

A domain expert selects:

product_master.material_id

Persist the decision:

{
  "knowledge_type": "semantic_mapping",
  "term": "Product Code",
  "target": "product_master.material_id",
  "source": "human_confirmation",
  "status": "trusted",
  "confidence": 1.0
}

This changes the human role from:

Author everything

to:

Review high-value uncertainty

That is much more scalable.

7. Runtime Clarification Is Also a Learning Channel

Some ambiguity only appears when users ask real questions.

Suppose:

Show revenue by product code.

The semantic resolver returns:

{
  "term": "Product Code",
  "status": "ambiguous",
  "candidates": [
    "Master Product Code",
    "SKU Code",
    "Internal Product ID"
  ]
}

The agent asks:

Which product code do you mean?

The user chooses:

Master Product Code

A weak implementation uses the answer only for the current query.

A stronger implementation creates a knowledge event:

{
  "event": "semantic_confirmation",
  "user_phrase": "product code",
  "resolved_concept": "Master Product Code",
  "physical_field": "product_master.material_id",
  "source": "runtime_clarification"
}

Now the interaction improves future queries.

Every Correction Should Make the Next Question Cheaper

This is the core learning property.

Without persistence:

User A asks
   ↓
System clarifies
   ↓
Answer

User B asks same concept
   ↓
System clarifies again

With reusable knowledge:

User A asks
   ↓
System clarifies
   ↓
Knowledge stored
   ↓
User B asks
   ↓
Known mapping
   ↓
Answer

The second query requires less reasoning and less human effort.

That is what enterprise learning should look like.

8. Separate Runtime State From Persistent Knowledge

Not every user choice should become global enterprise truth.

Suppose a user says:

Use invoice amount for this analysis.

That may be a query-specific preference, not a new definition of Revenue.

So distinguish:

Session State
User Preference
Workspace Default
Governed Enterprise Knowledge

A useful persistence policy might look like:

def persist_resolution(resolution):
    if resolution.scope == "query":
        return save_session_state(resolution)

    if resolution.source == "user_selection":
        return propose_workspace_knowledge(resolution)

    if resolution.approved_by_domain_owner:
        return save_governed_knowledge(resolution)

The exact workflow varies, but the principle matters:

Learning requires scope and governance.

Otherwise one user's temporary choice can corrupt the shared semantic layer.

9. Track Provenance

Every knowledge object should answer:

Where did this come from?
Who confirmed it?
When was it created?
How confident are we?
Is it still active?

Example:

{
  "term": "Revenue",
  "target": "finance_revenue.recognized_amount",
  "source": "finance_metric_catalog",
  "owner": "finance",
  "status": "active",
  "confidence": 1.0,
  "version": 4
}

For discovered relationships:

{
  "relationship": "account.customer_id -> customer.customer_id",
  "source": "relationship_discovery",
  "confirmed_by": "data_team",
  "status": "trusted"
}

Without provenance, accumulated knowledge becomes difficult to trust.

10. Build a Context Resolver Above the Raw Knowledge

The query runtime should not dump every known object into the prompt.

Instead:

User Question
      ↓
Intent Extraction
      ↓
Context Resolver
      ↓
Relevant Business Terms
Relevant Metrics
Relevant Fields
Relevant Relationships
      ↓
Compact Query Context
      ↓
SQL Generation

For:

Revenue by product code last quarter

the context resolver might return:

{
  "metric": {
    "name": "Revenue",
    "field": "finance_revenue.recognized_amount"
  },

  "dimension": {
    "name": "Product Code",
    "field": "product_master.material_id"
  },

  "relationship_path": [
    "product_master -> sales_order",
    "sales_order -> finance_revenue"
  ],

  "time_range": "last_quarter"
}

The model reasons over a smaller, better-defined space.

11. Treat Knowledge Acquisition as an Event Stream

A useful architecture is to treat semantic learning as events:

schema_discovered
relationship_candidate_found
relationship_confirmed
metric_imported
semantic_mapping_confirmed
clarification_resolved
mapping_deprecated

For example:

{
  "event_type": "semantic_mapping_confirmed",
  "timestamp": "2026-09-07T10:30:00Z",
  "payload": {
    "term": "Product Code",
    "field": "product_master.material_id"
  }
}

This gives you:

Auditability
Versioning
Rollback
Analytics
Learning metrics

It also avoids hiding important enterprise knowledge inside opaque prompt history.

12. Measure Time to Intelligence

Connection latency is easy to measure:

Database connected in 4 minutes.

But a better enterprise metric is:

Time to Intelligence

Possible operational measurements include:

Time until first trusted answer

% of common business terms resolved

% of critical metrics governed

% of required relationships trusted

Clarification rate

Repeated clarification rate

Manual confirmations per 100 queries

A healthy system should improve over time.

For example:

Week 1
Semantic coverage: 42%
Repeated clarification: 31%

Month 1
Semantic coverage: 68%
Repeated clarification: 17%

Month 3
Semantic coverage: 84%
Repeated clarification: 6%

The exact targets depend on the enterprise.

The direction is what matters.

13. Measure Enterprise Learning Rate

Another useful concept is:

Enterprise Learning Rate

How quickly does real usage produce reusable trusted knowledge?

You could track:

New trusted mappings / week
New trusted relationships / week
Clarifications converted to reusable knowledge
Repeated ambiguity reduction
Human confirmation workload

The goal is not maximum knowledge creation.

It is reducing repeated uncertainty.

A useful signal might be:

Repeated Clarification Rate ↓

If users keep clarifying the same concepts month after month, the system is not learning effectively.

14. A Maturity Model

A data agent can be viewed as progressing through several levels.

Level 0 — Connected

Can access data

Level 1 — Schema-Aware

Understands tables and fields

Level 2 — Relationship-Aware

Understands how relevant data connects

Level 3 — Semantic-Aware

Understands business terms,
metrics, and dimensions

Level 4 — Business-Aware

Uses governed definitions,
trusted mappings, and business context

Level 5 — Learning

Turns corrections,
confirmations, and clarifications
into reusable knowledge

Most enterprise AI demos prove Levels 0 and 1.

Production value increasingly appears in Levels 3 through 5.

15. The Goal Is Not Fully Automatic Semantics

There is a tempting but unrealistic objective:

Connect database
↓
AI automatically understands everything

Some business meaning cannot be inferred from data.

If a company decides that Revenue means:

Recognized Revenue

rather than:

Invoice Amount

that is an organizational decision.

The system should not pretend otherwise.

A more realistic architecture is:

Machine discovers
      ↓
AI proposes
      ↓
Humans confirm high-impact meaning
      ↓
System remembers

The goal is not zero human input.

It is:

Minimum repetitive human work with preserved human authority over business meaning.

16. Keep Enterprise Knowledge Model-Agnostic

The underlying LLM will change.

Enterprise knowledge should survive those changes.

Avoid storing important semantics only inside:

Prompt templates
Few-shot examples
Conversation history
Model-specific instructions

Instead, keep durable knowledge as structured objects:

Business Terms
Metrics
Dimensions
Mappings
Relationships
Rules
Provenance

Then different models can consume the same enterprise context.

Models are replaceable.

Enterprise knowledge compounds.

17. A Practical Bootstrap Architecture
Putting the pieces together:

              Enterprise Data
                     ↓
             Metadata Discovery
                     ↓
          Relationship Discovery
                     ↓
      Existing Knowledge Import
                     ↓
       Semantic Candidate Layer
                     ↓
            Human Confirmation
                     ↓
          Trusted Knowledge Store
                     ↑
                     │
User Question → Runtime Resolver
                     │
                     ↓
              Clarification
                     │
                     ↓
              Knowledge Event
                     │
                     └──────────→ Trusted Knowledge Store

Trusted Context
      ↓
Query Planning
      ↓
SQL Generation
      ↓
Validation
      ↓
Execution

The system becomes better not because the model retrains after every question, but because the enterprise context around the model improves.

Final Thoughts

The semantic cold start problem is easy to underestimate.

A database connection gives an agent access to:

Data

Production analytics requires:

Data
+
Meaning
+
Relationships
+
Authority
+
Learning

So the engineering goal should not simply be:

Connect the database faster.

It should be:

Reduce the amount of work required for the system to become business-aware.

That means discovering what can be discovered, importing what already exists, asking humans only where judgment matters, and turning useful corrections into durable knowledge.

The most important property is simple:

Every correction should make the next question cheaper.

That is how a generic data agent starts becoming an enterprise data agent.

How I Would Benchmark a Text-to-SQL System for Production

ArisynData — Wed, 02 Sep 2026 22:26:00 +0000

A clean question against a clean schema is a demo. A production benchmark should deliberately test ambiguity, missing relationships, competing metrics, and SQL that executes successfully but answers the wrong business question.

Most Text-to-SQL evaluations start with questions like:

What was revenue last quarter?
Show sales by region.
List the top 10 customers.

These tests are useful for checking whether the basic pipeline works.

They are not enough to tell you whether the system is ready for enterprise production.

In production, the difficult cases are rarely caused by SQL syntax alone. They come from business language, incomplete metadata, ambiguous metrics, undocumented relationships, aggregation grain, and assumptions that look reasonable but are wrong.

So if I were benchmarking a Text-to-SQL system, I would build the benchmark around failure paths, not just happy paths.

## Why Execution Accuracy Is Not Enough

A typical evaluation might measure:

Question
   ↓
Generated SQL
   ↓
Expected SQL

or:

Generated SQL
   ↓
Execute
   ↓
Compare Result

Execution-based evaluation is better than exact SQL matching because multiple SQL statements can produce the same correct answer.

But enterprise reliability requires additional questions:

Did the system understand the business term correctly?

Did it choose the authoritative metric?

Did it use the right relationship path?

Did it preserve the correct aggregation grain?

Did it recognize ambiguity?

Did it know when not to query?

A query can execute successfully and still fail every one of these tests.

# Build a Failure-Path Benchmark

I would create at least seven test categories.

A. Ambiguous Intent
B. Business Term Mapping
C. Competing Metrics
D. Missing Relationships
E. Executable-but-Wrong SQL
F. Insufficient Intent
G. Environment Change

Each category tests a different part of the system.

## Test A: Ambiguous Intent

Question:

Show me our best customers.

Possible interpretations:

Revenue
Profit
Growth
Retention
Lifetime Value

The benchmark should not define best in the question.

The correct behavior depends on enterprise context.

If a governed definition exists:

term: best_customer
metric: customer_lifetime_value
status: active

the system can resolve it.

If multiple valid definitions remain, the expected behavior should be:

CLARIFY

not:

GENERATE_SQL

### Example benchmark case

{
  "id": "ambiguity_001",
  "question": "Show me our best customers.",
  "valid_metrics": [
    "revenue",
    "profit",
    "growth",
    "retention"
  ],
  "governed_default": null,
  "expected_action": "clarify"
}

### What to score

Ambiguity detected?
Correct candidates identified?
Clarification triggered?
Clarification options useful?

This gives you a Clarification Accuracy metric rather than only SQL accuracy.

## Test B: Business Term Mapping

Question:

What was revenue by product code this quarter?

Now make sure the schema does not contain a field called:

product_code

Instead:

product_master.material_id
sales_detail.sku_no
inventory.item_code
product_dim.prod_master_id

The benchmark should contain a governed semantic mapping:

business_term:
  name: Product Code

maps_to:
  table: product_master
  field: material_id

### Failure mode

A schema-similarity system may select:

inventory.item_code

because the name looks closest.

But the benchmark expects:

product_master.material_id

because that is the governed enterprise definition.

### What to score

Business term resolved correctly?
Correct physical field selected?
Governed mapping preferred over lexical similarity?

Call this:

# **Semantic Mapping Accuracy**

## Test C: Competing Metrics

Create several plausible financial fields:

sales_order.total_amount
invoice.invoice_amount
finance_revenue.recognized_amount
payment.received_amount

Question:

What was revenue last quarter?

All four fields are relevant to money.

Only one represents the governed Revenue metric.

Benchmark definition:

metric:
  name: Revenue

source:
  table: finance_revenue
  field: recognized_amount

time_field:
  finance_revenue.recognition_date

### What to inspect

Do not only inspect the final number.

Inspect the query plan.

Expected:

Metric
→ Revenue

Field
→ finance_revenue.recognized_amount

Time
→ finance_revenue.recognition_date

### What to score

Metric resolution accuracy
Source selection accuracy
Time-field accuracy

This catches systems that accidentally return the right number from the wrong business definition.

## Test D: Missing Relationships

Production databases often lack perfect foreign keys.

Build:

customer
account
sales_order
finance_revenue

Expected relationship path:

Customer
   ↓
Account
   ↓
Order
   ↓
Revenue

Then deliberately remove some database constraints.

The benchmark should still contain enough data evidence for a relationship system to infer or retrieve valid relationships.

For example:

account.customer_id
sales_order.account_id
finance_revenue.order_id

### Useful relationship evidence

A relationship engine might consider:

Column names
Data types
Uniqueness
Value overlap
Inclusion
Known metadata
Validated relationships

One useful inclusion signal is:

Inclusion(A → B)
=
|distinct(A) ∩ distinct(B)|
---------------------------
|distinct(A)|

where A is a candidate foreign-key-like column and B is a candidate referenced column.

### What to score

Correct tables selected?
Correct relationship path found?
Unsupported direct joins avoided?

Call this:

# **Relationship Path Accuracy**

## Test E: Executable-but-Wrong SQL

This is the test I would care about most.

Suppose the schema allows:

Customer → Order

and:

Customer → Account → Order

Both can generate executable SQL.

But for consolidated customers, only:

Customer → Account → Order

preserves the correct business grain.

Create two candidate queries.

### Query A

SELECT
    c.customer_name,
    SUM(o.amount)
FROM customer c
JOIN sales_order o
    ON c.customer_id = o.customer_id
GROUP BY c.customer_name;

### Query B

SELECT
    c.customer_name,
    SUM(o.amount)
FROM customer c
JOIN account a
    ON c.customer_id = a.customer_id
JOIN sales_order o
    ON a.account_id = o.account_id
GROUP BY c.customer_name;

Both may execute.

Only Query B is business-valid for the benchmark scenario.

### What to score

SQL executable?
Business relationship valid?
Aggregation grain valid?
Result semantically correct?

This creates an important distinction:

Execution Accuracy
≠
Business Accuracy

## Test F: Insufficient Intent

Question:

Show our best-performing products recently.

The benchmark intentionally leaves unresolved:

best-performing
recently

Possible metric candidates:

Revenue
Profit
Units Sold
Growth

Possible time candidates:

7 Days
30 Days
Current Month
Current Quarter

If no governed defaults exist, the expected result is not SQL.

It is:

{
  "action": "clarify",
  "unresolved": [
    "metric",
    "time_range"
  ]
}

### What to score

Did the system detect missing intent?
Did it avoid premature SQL generation?
Did it ask the minimum useful clarification?

This measures:

# **Safe Failure Accuracy**

A production system should not be rewarded for answering every question.

## Test G: Environment Change

Static benchmarks miss one of the largest production costs: maintenance.

Start with a working environment.

Then introduce changes:

Add a table
Rename a field
Add a new metric
Deprecate an old metric
Add a new relationship
Change a semantic mapping

For example:

Before:
Product Code → product_master.material_id

After:
Product Code → product_dim.product_code

Then rerun the benchmark.

Measure:

How many cases break?

How much manual configuration is required?

How quickly does the system recover?

Which knowledge objects need updating?

This produces a metric that benchmark leaderboards rarely show:

# **Semantic Maintenance Cost**

# Score More Than SQL

A useful benchmark report could look like:

Metric	Score
Semantic Mapping Accuracy	94%
Metric Resolution Accuracy	96%
Relationship Path Accuracy	91%
Clarification Accuracy	93%
SQL Execution Accuracy	98%
Business Answer Accuracy	89%
Safe Failure Accuracy	95%
Maintenance Effort	3.2 min/change

The exact metrics depend on the product.

The important point is that:

SQL Accuracy

should be one row, not the entire benchmark.

# Use a Multi-Stage Evaluation Harness

Instead of treating Text-to-SQL as one black box:

Question
→ SQL

capture intermediate artifacts.

For example:

{
  "question": "Revenue by customer last quarter",

  "intent": {
    "metric": "recognized_revenue",
    "dimension": "customer",
    "time": "last_quarter"
  },

  "semantic_objects": [
    "metric.revenue",
    "dimension.customer"
  ],

  "tables": [
    "customer",
    "account",
    "sales_order",
    "finance_revenue"
  ],

  "relationship_path": [
    "customer.customer_id -> account.customer_id",
    "account.account_id -> sales_order.account_id",
    "sales_order.order_id -> finance_revenue.order_id"
  ],

  "generated_sql": "...",

  "action": "execute"
}

Now failures become diagnosable.

If the final answer is wrong, you can determine whether the failure came from:

Intent
Semantic Mapping
Table Selection
Relationship Resolution
Metric Selection
SQL Generation
Execution

That is much more useful than:

FAILED

# Build Adversarial Test Pairs

A strong benchmark should include pairs of questions that look similar but require different interpretations.

Example:

Q1

What was our revenue last quarter?

Expected metric:

recognized_revenue

Q2

What was our invoiced amount last quarter?

Expected metric:

invoice_amount

Another pair:

Q1

Sales by customer region.

Expected dimension:

customer_region

Q2

Sales by billing region.

Expected dimension:

billing_region

These pairs test whether the system is actually resolving semantics or simply reusing the most common mapping.

# Include Negative Tests

Traditional benchmarks mostly contain questions that should be answerable.

Production benchmarks should include questions that should not.

For example:

Question:
"Show profitability by happiness score."

No governed metric:
happiness_score

Expected:
Cannot resolve metric / clarification required

Or:

Question:
"Join employee salary with public customer data."

Expected:
Rejected by policy

The exact safety rules depend on the deployment.

The principle is:

A system should be tested on its ability to refuse invalid query plans, not only generate valid ones.

# Test With Messy Schemas

Do not benchmark only:

customer
orders
products

Use schemas that look more like enterprise reality:

t_cust_m
cust_master_old
acct_rel
f_ord_h
ord_detail_v2
fin_rev_rec
inv_hdr
inv_line

Add:

Deprecated tables
Duplicate concepts
Missing descriptions
Inconsistent naming
Cross-system IDs
Partial foreign keys

If the product claims to work with enterprise data, the benchmark should look like enterprise data.

# Measure Human Setup Cost

Before the first benchmark run, track setup effort.

For example:

Schema descriptions:       4 hours
Metric definitions:        3 hours
Relationship configuration: 6 hours
Example SQL:                5 hours
Prompt tuning:              2 hours

Total:

20 hours

Another system may achieve slightly lower raw accuracy but require:

4 hours

of setup.

That difference matters.

So benchmark:

Accuracy

and:

Cost to achieve that accuracy

together.

# A More Useful Production Score

I would think about production value approximately as:

                  Accuracy × Trust × Coverage
Value  ≈  ─────────────────────────────────────
                Setup + Maintenance Cost

Again, this is not a universal mathematical formula.

It forces the benchmark to consider what enterprises actually pay for.

A system that scores 97% on 50 carefully prepared questions may be less valuable than one scoring 93% across thousands of messy tables with far less manual configuration.

# What I Would Put in a POC

If I were designing a vendor POC, I would include:

10 clear questions
10 ambiguous questions
10 semantic-alias questions
10 competing-metric questions
10 missing-relationship questions
10 multi-path join questions
10 executable-but-wrong traps
10 questions that require clarification
10 unanswerable questions
10 post-schema-change regressions

That gives:

100 cases

with intentionally different failure modes.

Then evaluate both:

Can it answer correctly?

and:

Can it recognize when it should not answer yet?

# Final Thoughts

A production Text-to-SQL benchmark should not primarily test whether an LLM knows SQL.

Modern models are already good at SQL generation under clean conditions.

The harder questions are:

Did it understand the business term?

Did it select the right metric?

Did it find the right data?

Did it use the right relationship?

Did it preserve the right grain?

Did it detect ambiguity?

Did it fail safely?

How much human work was required?

Those are system-level questions.

So when evaluating a Text-to-SQL platform:

Don't benchmark the easiest route from question to SQL.

Deliberately break the assumptions.

Remove the foreign key.

Add three plausible revenue fields.

Use business terminology that does not match the schema.

Create two executable join paths where only one is correct.

Ask a question that should trigger clarification.

Then change the data model and run everything again.

That is where you start learning whether you have a good demo—or a production system.

Building a Trustworthy Semantic Layer for Natural Language Analytics

ArisynData — Mon, 31 Aug 2026 22:10:00 +0000

Natural language analytics looks simple from the outside:

User asks a question.

The system generates SQL.

The database returns an answer.

In real enterprise environments, this breaks quickly.

The reason is not just SQL generation. The reason is semantic ambiguity.

When a user asks:

Show sales by product code for last month.

The system needs to resolve several questions before it can safely generate a query:

What does "sales" mean?
Is it gross sales, net sales, recognized revenue, bookings, or paid order amount?
What does "product code" mean?
Is it internal product code, SKU, marketplace product code, barcode, or ERP material number?
Which date field should represent "last month"?
Which tables contain the right facts and dimensions?
Which join path is trusted?
Is the current user allowed to access the required data?

This is why a semantic layer is becoming a critical part of natural language analytics.

But not all semantic layers are designed for the same job.

Databricks is building semantic capabilities natively inside its lakehouse platform. Semora is building an independent semantic layer for complex, cross-platform enterprise data. From an engineering perspective, the interesting question is not "Which product has a better chat interface?"

The better question is:

What system architecture makes natural language analytics both fluent and accurate?

## The Core Problem: Language Models Should Not Guess the Data Model

A large language model can usually understand that these phrases are related:

product code
product ID
item number
SKU
material number

But in a real company, those phrases may not be interchangeable.

One table may contain:

product_code
product_id
sku_id
barcode
platform_item_id
material_number

If the system maps "product ID" to product_code just because the words sound close, the answer may be fluent but wrong.

On the other hand, if the system asks the user to clarify every similar term, the experience becomes painful.

The engineering challenge is to build a runtime that can decide:

When can we answer directly?
When should we show the assumption?
When should we ask a lightweight clarification?
When must we require explicit confirmation?
How do we save the user's answer so we do not ask again?

This is where Semora can differentiate.

## Databricks: Platform-Native Semantic Context

Databricks has been expanding its semantic layer and natural language analytics stack through Unity Catalog business semantics, metric views, Genie, and Genie Agents.

The Databricks approach is powerful because semantic assets live close to the governed data platform.

A Databricks-style flow can look like this:

User question
  -> Genie Agent scope
  -> Unity Catalog metadata
  -> Metric views
  -> Table and column descriptions
  -> Trusted queries and examples
  -> SQL generation
  -> Query inspection
  -> Governed answer

This works especially well when:

The company already uses Databricks as the main analytical platform.
Tables, permissions, lineage, and metrics are managed in Unity Catalog.
Business domains are curated into focused Genie Agents.
Data teams maintain metric views, examples, and trusted assets.

The key engineering lesson is that Databricks does not treat the language model as the only source of intelligence. It surrounds the model with governed context.

That context reduces the search space.

If a user is asking inside a Sales Genie Agent, "revenue" can be interpreted using sales-domain semantics. If they ask inside a Finance context, the default meaning may be different.

This domain scoping is one of the most important reasons a natural language analytics experience feels smooth.

## Semora: Independent Semantic Runtime

Semora has a different opportunity.

Many enterprises do not have one clean analytical platform. They have multiple warehouses, operational databases, business intelligence tools, spreadsheets, and legacy systems. Field names are inconsistent. Foreign keys are missing. Business terms vary by department.

In this environment, an independent semantic layer is valuable because it can sit above the data platforms instead of being tied to only one.

A Semora-style flow can look like this:

User question
  -> Intent classification
  -> Role and permission scope
  -> Business domain routing
  -> Synonym and semantic retrieval
  -> Business concept mapping
  -> Certified field selection
  -> Relationship discovery
  -> Trusted question template matching
  -> Query generation
  -> Query validation
  -> Answer with lineage
  -> Clarification memory update

The important word here is runtime.

It is not enough to have pages for synonyms, business semantics, data definitions, question templates, and knowledge bases. Those assets must be actively retrieved, ranked, and applied while the user is asking a question.

Otherwise, the product can have many semantic assets and still feel rough.

## Runtime Component 1: Intent Classification

The first step should not be SQL generation.

The first step should be intent classification.

For example, the user may ask:

What questions can you answer for me?

This is not a data query. It is a capability discovery question.

A poor system routes it into the query pipeline, fails to find a metric, and asks for clarification.

A better system recognizes the intent and answers based on the user's role, permissions, business domains, and available question templates.

Useful intent classes include:

DATA_QUERY
CAPABILITY_DISCOVERY
METRIC_EXPLANATION
FIELD_EXPLANATION
DOMAIN_EXPLORATION
TEMPLATE_RECOMMENDATION
PERMISSION_SCOPE_EXPLANATION
PRODUCT_HELP

For CAPABILITY_DISCOVERY, the response should be generated from available semantic assets:

Accessible business domains
Certified metrics
Question templates
Frequently used questions
Available dimensions
Permission boundaries

Example response:

Based on your current access, I can answer questions about sales, products, inventory, and channels. You can ask about sales trends, product rankings, inventory shortages, channel comparisons, and customer repeat purchase behavior.

This small routing layer can dramatically improve first-time user experience.

## Runtime Component 2: Business Domain Routing

Even if Semora keeps one universal question box, the backend should not treat all data as one global space.

The system should route each question into one or more business domains.

For example:

"sales by product code"
  -> Sales domain
  -> Product domain

"inventory risk by SKU"
  -> Inventory domain
  -> Product domain

"recognized revenue by customer"
  -> Finance domain
  -> Customer domain

Domain routing should use:

User role
Permission scope
Question text
Matched metrics
Matched templates
Historical user behavior
Business glossary terms

The user can still see one clean interface. Internally, the system should load a scoped semantic context.

This reduces ambiguity before the language model starts planning the query.

## Runtime Component 3: Multi-Path Semantic Retrieval

A common failure mode is exact or weak lexical matching.

If the physical field is called product_code, and the user says "product number," the system may fail unless that synonym was manually configured.

This does not scale.

Semora should use multi-path retrieval:

User phrase
  -> Exact match
  -> Token match
  -> Synonym match
  -> Embedding similarity
  -> Field description match
  -> Sample value match
  -> Historical clarification match
  -> Question template match
  -> Domain default match

For the phrase "product code," candidates might include:

Business concept: Product Code
Fields:
  - dim_product.product_code
  - order_items.product_code
  - inventory_sku.sku_code
  - marketplace_items.platform_item_id
  - product_barcodes.barcode

The retrieval layer should not immediately decide. It should return candidates with evidence.

Example evidence:

{
  "candidate": "dim_product.product_code",
  "business_concept": "Product Code",
  "evidence": {
    "synonym_match": ["product code", "item code"],
    "domain": "Product",
    "field_description": "Internal product code used across sales and inventory",
    "historical_confirmations": 17,
    "certification_status": "certified"
  },
  "confidence": 0.93
}

This gives the language model structured context instead of asking it to guess from raw schema names.

## Runtime Component 4: Confidence-Based Clarification

Clarification should be a decision, not a default.

A useful policy could be:

confidence >= 0.85
  -> answer directly and show the assumption

0.60 <= confidence < 0.85
  -> answer with a lightweight change option, or ask a short clarification

confidence < 0.60
  -> require clarification

high-risk metric or high-risk join
  -> require confirmation even if confidence is moderate

For a high-confidence case:

I interpreted "product code" as the certified Product Code concept and used the internal product code field.

For an ambiguous case:

Which product identifier do you mean?

Internal product code

SKU number

Marketplace product code

Barcode

ERP material number

This is better than showing raw column names to a business user.

Technical users can still inspect the underlying fields, joins, and SQL.

## Runtime Component 5: Trusted Question Templates

Question templates should not be simple prompt examples.

They should be trusted analytical paths.

For example, a template for:

Top-selling products in a time period

Should include:

template_id: top_selling_products
intent: ranking
metric:
  name: Sales Quantity
  certification: certified
dimensions:
  - Product Code
  - Product Name
time:
  default_field: Paid Time
filters:
  - Paid Orders Only
relationships:
  - order_items -> dim_product by product_code
parameters:
  - time_range
  - channel
  - category
validation:
  - sales_quantity >= 0
  - product_code is not null
owner: Sales Analytics Team
status: certified

When the user asks:

Show last month's product code sales ranking.

The system should match the template first, extract parameters, and reuse the certified query path.

The language model should not rebuild the business logic from scratch.

## Runtime Component 6: Relationship Discovery and Join Safety

Natural language analytics systems often fail silently when joins are wrong.

A query can run successfully and still be incorrect because:

It joined at the wrong grain.
It created duplicate rows.
It used a weak relationship.
It joined through the wrong entity.
It mixed internal identifiers with external identifiers.

For Semora, relationship discovery should be a major differentiator.

A relationship engine should evaluate:

Field name similarity
Data type compatibility
Value overlap
Inclusion ratio
Cardinality
Null rate
Uniqueness
Historical usage
Human approval
Domain relevance

The output should be explainable:

{
  "join_path": [
    "order_items.product_code",
    "dim_product.product_code"
  ],
  "confidence": 0.91,
  "evidence": {
    "value_overlap": "high",
    "right_side_uniqueness": "strong",
    "approved_by": "data_owner",
    "fanout_risk": "low"
  }
}

This is where independent semantic layers can create real value. Platform-native semantic layers work well when relationships are already modeled. Independent semantic layers can help discover and govern relationships where metadata is incomplete.

## Runtime Component 7: Query Validation

SQL generation is not the end of the pipeline.

The system should validate the generated query before presenting the answer.

Validation checks may include:

Does the user have permission to access every field?
Is the selected metric certified?
Is the date field appropriate for the metric?
Are filter values valid?
Is the join path approved?
Is there fanout risk?
Is the aggregation grain correct?
Is the result within a reasonable historical range?
Did the query return no rows because of a value mapping issue?

This validation layer is one of the most important differences between a demo and a production-ready product.

## Runtime Component 8: Clarification Memory

Every clarification should become a reusable asset.

If a user confirms:

By "product code," I mean the internal product code.

The system should save:

user_phrase: product code
standard_concept: Product Code
field_mapping: dim_product.product_code
business_domain:
  - Product
  - Sales
confirmed_by: user_or_owner
scope: user | team | organization
status: draft | reviewed | certified
usage_count: 1

Over time, repeated confirmations should create candidate semantic rules. Data owners can review and promote them.

This makes the system smoother with use.

The goal is not to ask better clarification questions forever. The goal is to ask fewer clarification questions over time because the semantic layer is learning.

## A Practical Architecture

Here is a practical architecture for a natural language analytics runtime:

                         +----------------------+
                         |   User Question      |
                         +----------+-----------+
                                    |
                                    v
                         +----------------------+
                         | Intent Classifier    |
                         +----------+-----------+
                                    |
              +---------------------+---------------------+
              |                                           |
              v                                           v
   +------------------------+                 +------------------------+
   | Non-query Intents      |                 | Data Query Intent      |
   | Help, scope, examples  |                 | Metrics, filters, SQL  |
   +------------------------+                 +-----------+------------+
                                                          |
                                                          v
                                           +---------------------------+
                                           | Domain Router             |
                                           +------------+--------------+
                                                        |
                                                        v
                                           +---------------------------+
                                           | Semantic Retrieval        |
                                           +------------+--------------+
                                                        |
                                                        v
                                           +---------------------------+
                                           | Template Matching         |
                                           +------------+--------------+
                                                        |
                                                        v
                                           +---------------------------+
                                           | Confidence Decision       |
                                           +------------+--------------+
                                                        |
                             +--------------------------+--------------------------+
                             |                                                     |
                             v                                                     v
                 +------------------------+                           +------------------------+
                 | Clarification          |                           | Query Planning         |
                 +-----------+------------+                           +-----------+------------+
                             |                                                    |
                             v                                                    v
                 +------------------------+                           +------------------------+
                 | Semantic Memory        |                           | SQL Generation         |
                 +------------------------+                           +-----------+------------+
                                                                                  |
                                                                                  v
                                                                      +------------------------+
                                                                      | Query Validation       |
                                                                      +-----------+------------+
                                                                                  |
                                                                                  v
                                                                      +------------------------+
                                                                      | Answer + Lineage       |
                                                                      +------------------------+

The language model appears in multiple places:

Intent classification
Query planning
Clarification generation
Parameter extraction
Result explanation
Follow-up question handling

But the model is not operating alone. It is grounded by semantic assets, trusted templates, permissions, relationship evidence, and validation rules.

That is the difference between natural language analytics as a demo and natural language analytics as enterprise infrastructure.

## Where Semora Can Stand Apart

Databricks has a strong platform-native story. If a company has standardized its data and governance inside Databricks, that integration is hard to beat.

Semora should not try to be a smaller version of Databricks.

It should focus on the problems that appear when the enterprise data world is messy:

Business terms vary across teams.
Field names do not match user language.
Relationships are not fully documented.
Metrics conflict across systems.
Users need answers across multiple platforms.
Clarification should become governed semantic memory.
Natural language queries need explainable lineage.

This is a valuable and distinct engineering problem.

The best semantic layer is not the one that lets the model guess more freely. It is the one that gives the model enough context, evidence, and constraints to answer correctly.

## Conclusion

Natural language analytics is not solved by connecting a language model to a database.

It requires a semantic runtime.

That runtime must classify intent, route by business domain, retrieve semantic assets, match trusted templates, evaluate confidence, clarify only when necessary, generate governed queries, validate results, and save human feedback as reusable semantic knowledge.

Databricks approaches this through a platform-native lakehouse stack.

Semora's opportunity is to build an independent semantic intelligence layer for enterprises whose data, terminology, and relationships span many systems.

For developers and data engineers, the lesson is clear:

Do not make the model guess the enterprise data model. Build the semantic system that lets it reason with evidence.

Your AI Data Agent Needs a “Do Not Use” Layer

ArisynData — Tue, 25 Aug 2026 02:08:09 +0000

The missing layer in many enterprise AI systems is not more metadata. It is a machine-readable record of the mistakes the organization already knows not to make.

A surprising amount of enterprise data expertise sounds like this:

Don't use that table after the migration.

Don't use invoice_amount as recognized revenue.

Don't join those two tables directly.

Don't use created_at for financial reporting.

Don't aggregate those rows before deduplicating accounts.

These rules are rarely visible in the database schema. They may not exist in the catalog. Often they live only in the heads of experienced analysts, engineers, and business owners.

That worked when humans were the primary consumers of enterprise data. It becomes a serious problem when AI agents start querying data directly.

The usual response is to give the model more context.

But what if the missing context is not another description of what the data is?

What if the missing layer is a structured description of what the AI must not do?

## The Data Model Contains Less Knowledge Than We Think

Consider:

CREATE TABLE invoice (
    invoice_id      BIGINT,
    customer_id     BIGINT,
    invoice_amount  DECIMAL(18,2),
    invoice_date    DATE,
    status          VARCHAR(20)
);

An AI can identify a customer reference, an amount, and a date. It can probably generate valid SQL for:

What was recognized revenue last quarter?

But the enterprise may already know:

invoice_amount is not recognized revenue.

That fact is not a schema fact.

It is organizational knowledge.

Without it, the agent can be technically competent and still wrong.

## “Do Not Use” Is a Real Data Asset

Data platforms traditionally capture positive knowledge:

What tables exist?
What do columns mean?
What is the metric definition?
How are entities related?

Expert users also carry negative knowledge:

Which source is misleading?
Which join is dangerous?
Which field looks correct but isn't?
Which timestamp should not be used?
Which aggregation creates double counting?

This knowledge is valuable precisely because the wrong choice often looks reasonable.

A production data agent therefore needs:

Positive Knowledge
→ What can I use?

Negative Knowledge
→ What must I avoid?

## A “Do Not Use” Layer

Imagine structured constraints attached to enterprise data objects.

For a field:

object:
  type: column
  name: invoice.invoice_amount

meaning:
  business_term: Invoice Value

usage:
  valid_for:
    - invoice_analysis

  invalid_for:
    - recognized_revenue

reason:
  Invoice value may include amounts that have
  not yet met revenue recognition rules.

owner:
  finance

status:
  active

For a relationship:

object:
  type: relationship

from:
  customer.customer_id

to:
  order.customer_id

constraint:
  invalid_when:
    account_structure: consolidated

required_path:
  - customer
  - account
  - order

reason:
  Direct joins can duplicate orders across child accounts.

For a time field:

object:
  type: time_rule

metric:
  recognized_revenue

required_field:
  finance_revenue.recognition_date

do_not_use:
  - invoice.invoice_date
  - sales_order.created_at

This is not a blacklist.

It is contextual enterprise knowledge.

## Why a Blacklist Is Too Primitive

Marking:

invoice.invoice_amount = forbidden

would be wrong.

The field is perfectly valid for Invoice Value. It is invalid only when interpreted as Recognized Revenue.

So the real rule is:

Object + Business Intent + Context → Valid / Invalid

For example:

field: invoice.invoice_amount

valid_for:
  - invoice_value

invalid_for:
  - recognized_revenue
  - cash_received

The same physical field can be valid or invalid depending on the analytical task.

## Put Constraints Before the LLM

A naive pipeline:

Question
   ↓
Schema Retrieval
   ↓
LLM
   ↓
SQL

A stronger pipeline:

Question
   ↓
Business Intent Resolution
   ↓
Candidate Retrieval
   ↓
“Do Not Use” Constraint Check
   ↓
Trusted Relationship Resolution
   ↓
Allowed Query Context
   ↓
LLM
   ↓
SQL
   ↓
Constraint Validation

The constraint layer appears twice.

First, it prevents known-invalid data from entering the model's decision space.

Second, it validates that generated SQL did not violate a known rule.

## Filter Invalid Candidates Before Generation

Suppose retrieval returns:

candidates = [
    "finance_revenue.recognized_amount",
    "invoice.invoice_amount",
    "sales_order.total_amount",
    "payment.received_amount"
]

For a recognized-revenue question:

def filter_by_constraints(candidates, intent, knowledge):
    allowed = []
    excluded = []

    for candidate in candidates:
        rule = knowledge.evaluate(
            object=candidate,
            intent=intent
        )

        if rule.is_hard_violation:
            excluded.append({
                "object": candidate,
                "reason": rule.reason
            })
            continue

        allowed.append(candidate)

    return allowed, excluded

The model now receives a much cleaner problem.

But keep the exclusion reason:

excluded_context:
  - object: invoice.invoice_amount
    reason: not valid for recognized revenue

That helps explanation and debugging.

## Relationship Constraints May Matter Even More

Wrong joins are dangerous because they can return plausible results.

Imagine:

Customer ───── Order
    │
    └──── Account ───── Order

Both paths may technically exist.

For consolidated accounts, the direct path may duplicate orders.

Represent the rule:

constraint:
  type: relationship_path

  from: customer
  to: order

  when:
    account_structure: consolidated

  forbidden_path:
    - customer
    - order

  required_path:
    - customer
    - account
    - order

Then build a task-specific graph:

Customer ──X── Order
    │
    └──── Account ───── Order

Conceptually:

valid_graph = relationship_graph.apply_constraints(
    business_context=context
)

path = valid_graph.shortest_trusted_path(
    source="customer",
    target="order"
)

This is safer than asking an LLM to infer join topology from names every time.

## Hard Rules and Soft Rules

Not every constraint should block execution.

### Hard Constraint

severity: hard

rule:
  do_not_use: customer_legacy

when:
  query_date >= 2026-01-01

Violation should stop or regenerate the query.

### Soft Guidance

severity: soft

rule:
  prefer: billing_region

fallback:
  registered_region

Violation may lower ranking or trigger review.

This avoids two extremes: an inflexible rule engine and a prompt that treats every rule as optional advice.

## Compile Governance Into Runtime Constraints

Suppose a governed metric says:

metric:
  id: recognized_revenue

source:
  finance_revenue.recognized_amount

time_field:
  finance_revenue.recognition_date

invalid_sources:
  - invoice.invoice_amount
  - sales_order.total_amount

required_filters:
  - status=recognized

Compile it:

policy = {
    "required_source":
        "finance_revenue.recognized_amount",

    "required_time_field":
        "finance_revenue.recognition_date",

    "forbidden_sources": [
        "invoice.invoice_amount",
        "sales_order.total_amount"
    ],

    "required_filters": [
        "status=recognized"
    ]
}

The same policy can shape retrieval, context construction, generation, and validation.

One governed definition drives multiple runtime controls.

## Validate What Must Not Appear

After generation, parse the SQL and check the policy:

def validate_query(ast, policy):
    violations = []

    for source in policy["forbidden_sources"]:
        if ast.references(source):
            violations.append({
                "type": "forbidden_source",
                "object": source
            })

    if not ast.references(policy["required_time_field"]):
        violations.append({
            "type": "wrong_time_field"
        })

    return violations

If violations exist:

if violations:
    regenerate_or_escalate(
        sql=sql,
        violations=violations
    )

The agent is no longer trusted to police itself.

## Where Does Negative Knowledge Come From?

One of the best sources is historical failure.

Example:

Problem:
Revenue dashboard overstated.

Root Cause:
Orders joined directly to customers.

Fix:
Use account-level relationship.

Lesson:
Never use direct Customer → Order join
for consolidated accounts.

Most organizations stop after fixing the dashboard.

A stronger system captures the lesson:

Failure
   ↓
Root Cause
   ↓
Data Rule
   ↓
Structured Constraint
   ↓
Future Prevention

Operational experience becomes AI infrastructure.

## Ask Experts for Mistakes, Not Documentation

If you ask a senior analyst:

Please document these 50 columns.

you may get accurate but generic descriptions.

Instead ask:

What are the five mistakes a new analyst is most likely to make with this data?

You may hear:

Don't use that amount as revenue.
Don't use this table after migration.
Don't join those IDs directly.
Don't use created_at for monthly reporting.
Don't SUM before deduplicating accounts.

Those answers encode experience, not just definitions.

They are often extremely valuable for AI.

## Version and Own the Constraints

A constraint can become outdated.

It should include:

ID
Version
Owner
Status
Effective Date
Reason
Validation State

For example:

constraint:
  id: C-REV-014
  version: 3

  rule:
    invalid_for:
      object: invoice.invoice_amount
      metric: recognized_revenue

  owner:
    finance

  status:
    active

  effective_from:
    2026-01-01

  reason:
    revenue recognition follows finance_revenue

Now the system knows whether the rule is still authoritative.

## Make the Layer Observable

Useful metrics include:

Constraint Coverage
% of critical metrics with explicit negative rules

Prevented Violations
Invalid candidates removed before LLM generation

Generated SQL Violation Rate
% of queries violating known constraints

Known-Mistake Recurrence
How often a known incident repeats after its rule is encoded

The last metric captures the real purpose.

The goal is not to create more documentation.

It is to stop known mistakes from recurring.

## Do Not Turn Everything Into a Rule

Not every piece of human judgment should become deterministic policy.

Some decisions are ambiguous. Some rules have many exceptions. Some knowledge is uncertain.

Encode negative knowledge when it is:

Known
Repeated
High-impact
Business-specific
Stable enough to govern
Hard for the model to infer from schema alone

Leave genuinely ambiguous reasoning to the model.

The goal is not:

Replace AI reasoning with rules.

The goal is:

Stop using AI reasoning for mistakes
the enterprise already knows how to prevent.

## A More Mature Data Agent Runtime

Putting it together:

User Question
      ↓
Business Intent
      ↓
Candidate Retrieval
      ↓
Positive Knowledge
      +
Negative Knowledge
      ↓
Constraint Filtering
      ↓
Trusted Relationship Planning
      ↓
Allowed Context
      ↓
LLM Reasoning / SQL Generation
      ↓
Constraint Validation
      ↓
Execution
      ↓
Answer

The LLM no longer operates over everything that looks relevant.

It operates inside a data space shaped by enterprise knowledge.

## The Bigger Engineering Principle

A wrong enterprise answer is not always a hallucination.

Sometimes it is:

valid computation over an invalid business assumption.

A bad join can execute.

A wrong metric can aggregate.

A deprecated table can return real rows.

A wrong date field can produce a believable trend.

That is why more context alone is not enough.

The system also needs boundaries.

## Final Thoughts

The most valuable enterprise data knowledge may not be another description of what a field means.

It may be:

Don't use this field for that metric.

Or:

Don't take that join path.

Or:

Don't use that timestamp for reporting.

Those warnings represent accumulated organizational experience.

If they remain in people's heads, every new AI agent can repeat old mistakes.

If they become structured, versioned, and enforceable, they become something more useful:c

A “Do Not Use” layer for enterprise AI.

And that may be one of the missing pieces between an AI agent that can query data and one that can use enterprise data reliably.

Beyond SQL Accuracy: Building Evidence Chains for AI Data Agents

ArisynData — Thu, 20 Aug 2026 03:10:07 +0000

AI data agents are getting good at producing executable SQL.

That is useful, but executable SQL is not the same thing as a correct business answer.

A query can compile, run successfully, return real rows, and still answer the wrong question because the agent selected the wrong metric definition, source table, relationship, grain, time field, or filter.

For production systems, this creates a different engineering requirement:

An AI data agent should not only generate an answer. It should preserve the evidence that produced it.

This article explores how to treat that evidence as a first-class artifact rather than an explanation generated after the fact.

## 1. Why SQL Accuracy Is an Incomplete Target

Consider this question:

What was net revenue by region last quarter?

The agent generates:

SELECT
    c.region,
    SUM(i.invoice_amount) AS net_revenue
FROM invoices i
JOIN customers c
    ON i.customer_id = c.customer_id
WHERE i.invoice_date >= '2026-04-01'
  AND i.invoice_date < '2026-07-01'
GROUP BY c.region;

The query is valid.

But suppose the enterprise definition is:

Net Revenue
=
Recognized Revenue
-
Refunds
-
Credits

and regional attribution is based on the billing account rather than the customer master.

The SQL engine cannot detect this error.

From the database's perspective, the query is correct.

From the business's perspective, it is wrong.

That gives us two different validation layers:

SQL Validity
=
Can this query execute?

Business Validity
=
Does this query represent the intended business question?

Production AI analytics needs both.

## 2. The Failure Surface Is Larger Than SQL Generation

A useful way to model an AI data query is:

Question
   ↓
Semantic Resolution
   ↓
Source Selection
   ↓
Relationship Selection
   ↓
Filter / Time Resolution
   ↓
SQL Generation
   ↓
Execution
   ↓
Answer

An error can occur at any stage.

For example:

Semantic Resolution
Revenue → Invoice Amount       ❌

Source Selection
invoices                       ✓

Relationship Selection
invoice.customer_id → customer ✓

Time Resolution
invoice_date                   ❌

SQL Generation
Valid SQL                      ✓

Execution
Success                        ✓

If we evaluate only the final SQL syntax or execution status, the system appears healthy.

The real failure happened much earlier.

This is why AI data agents need evidence chains, not just query logs.

## 3. What Is an Evidence Chain?

An evidence chain records the artifacts and decisions that connect a user question to the final answer.

Conceptually:

User Question
      │
      ▼
Business Definition
      │
      ▼
Source Data
      │
      ▼
Relationships
      │
      ▼
Filters / Time Rules
      │
      ▼
Generated SQL
      │
      ▼
Execution Result
      │
      ▼
Final Answer

The goal is not to create another verbose chain-of-thought log.

The goal is to preserve verifiable system artifacts.

That distinction matters.

Internal model reasoning is probabilistic and may not be suitable as an audit artifact.

A metric identifier, relationship identifier, SQL query, data-source identifier, and execution result are inspectable.

## 4. Evidence Should Be Structured

Instead of storing only:

question
sql
answer

store a structured evidence object.

For example:

{
  "question": "What was net revenue by region last quarter?",

  "semantic_resolution": {
    "metric_id": "net_revenue",
    "metric_version": "3.2",
    "dimensions": ["billing_region"]
  },

  "sources": [
    "orders",
    "refunds",
    "billing_accounts"
  ],

  "relationships": [
    {
      "relationship_id": "rel_orders_billing_account",
      "from": "orders.billing_account_id",
      "to": "billing_accounts.account_id",
      "status": "trusted"
    }
  ],

  "time": {
    "field": "settlement_date",
    "period": "last_quarter"
  },

  "sql": {
    "query_id": "q_8271"
  },

  "execution": {
    "result_id": "r_4412",
    "status": "success"
  },

  "validation": {
    "status": "passed"
  }
}

Now the answer has provenance.

## 5. Capture Evidence During Execution, Not Afterward

A common anti-pattern is:

1. Agent generates answer
2. User asks "Why?"
3. LLM generates an explanation

That produces a narrative, not necessarily provenance.

The model may describe what it believes happened.

A stronger implementation captures evidence as each stage executes.

For example:

evidence = Evidence(question=question)

metric = resolve_metric(question)
evidence.metric_id = metric.id
evidence.metric_version = metric.version

dimension = resolve_dimension(question)
evidence.dimensions.append(dimension.id)

path = relationship_service.get_trusted_path(
    metric=metric,
    dimensions=[dimension]
)
evidence.relationship_ids.extend(path.relationship_ids)

sql = generate_sql(
    question=question,
    metric=metric,
    dimensions=[dimension],
    relationship_path=path
)
evidence.sql = sql

result = execute(sql)
evidence.execution_status = result.status
evidence.result_id = result.id

The evidence object is built from actual system events.

That makes it much more useful for debugging and audit.

6. Semantic Evidence

The first layer is semantic resolution.

If the user asks:

What was revenue last quarter?

the system should be able to show:

User Term:
Revenue

Resolved Metric:
Net Revenue

Version:
3.2

Definition:
Recognized Revenue - Refunds - Credits

This matters because business terms are often ambiguous.

The SQL may be technically perfect while using the wrong definition.

A metric version is particularly important.

If Revenue v3.1 and v3.2 differ, the evidence chain should preserve which version generated the answer.

## 7. Source Evidence

The next question is:

Which data actually contributed to the answer?

For example:

orders
refunds
billing_accounts

A production evidence record may also include fields:

{
  "source": "orders",
  "fields": [
    "billing_account_id",
    "recognized_amount",
    "settlement_date"
  ]
}

This makes it possible to inspect whether the agent used an authoritative source rather than merely a semantically similar table.

## 8. Relationship Evidence

Multi-table queries require another layer of provenance.

Suppose the query joins:

orders
   ↓
billing_accounts
   ↓
region

The evidence should preserve why that path was selected.

At minimum:

{
  "relationship_id": "rel_orders_billing_account",
  "from": "orders.billing_account_id",
  "to": "billing_accounts.account_id",
  "status": "trusted"
}

More advanced systems could include relationship evidence such as:

Database Constraint
Naming Similarity
Value Inclusion
Uniqueness
Business Validation

The important point is that the agent is not silently inventing a join.

The relationship is inspectable.

## 9. Filter and Time Evidence

Filters are easy to overlook because they often appear as simple SQL predicates.

But they can completely change the business answer.

For example:

"last quarter"

might resolve to:

2026-04-01 → 2026-06-30

and use:

settlement_date

rather than:

invoice_date

The evidence record should preserve both.

For example:

{
  "time_resolution": {
    "business_period": "last_quarter",
    "start": "2026-04-01",
    "end": "2026-06-30",
    "field": "settlement_date"
  }
}

This makes temporal interpretation auditable.

## 10. Query Evidence

The SQL itself remains critical evidence.

But do not treat SQL as the entire explanation.

SQL is the executable consequence of upstream decisions.

A useful query record might include:

{
  "query_id": "q_8271",
  "sql_hash": "sha256:...",
  "generated_at": "2026-08-20T09:15:00Z",
  "metric_version": "3.2",
  "relationship_ids": [
    "rel_orders_billing_account"
  ]
}

This makes the query reproducible and connects it to the semantic and relationship state used during generation.

## 11. Execution Evidence

Execution success is useful evidence, just not sufficient evidence.

Capture:

Execution Status
Duration
Rows Returned
Data Source
Query Timestamp
Result Identifier

Potentially:

{
  "status": "success",
  "duration_ms": 842,
  "rows_returned": 8,
  "result_id": "r_4412"
}

This lets the system distinguish:

Bad SQL

from:

Valid SQL + Wrong Business Interpretation

## 12. Validation Should Check the Chain

Instead of asking only:

Did the SQL execute?

validation can inspect several layers.

### Semantic Validation

Was an approved metric definition used?

### Source Validation

Were authorized and current data sources used?

### Relationship Validation

Were joins based on trusted relationships?

### Filter Validation

Were required business filters applied?

### Query Validation

Did the SQL pass syntax and safety checks?

This produces a richer validation object:

{
  "semantic": "passed",
  "sources": "passed",
  "relationships": "passed",
  "filters": "passed",
  "sql": "passed"
}

## 13. Why LLM-as-a-Judge Cannot Be the Only Validator

A second model can review generated SQL.

That is useful.

But consider:

Agent:
Revenue = invoice_amount

Judge:
The SQL correctly sums invoice_amount.

Both models agree.

Both can still be wrong relative to the enterprise definition.

Language models are good at evaluating logical consistency inside the context they receive.

They cannot recover enterprise truth that is missing from that context.

So:

You cannot verify a data answer with language alone.

Verification needs grounded evidence from semantic definitions, metadata, relationships, executed SQL, and query results.

## 14. Evidence Chains Improve Debugging

Suppose an answer is reported as incorrect.

Without structured evidence, the debugging workflow may be:

Reproduce Prompt
↓
Inspect Agent Logs
↓
Inspect SQL
↓
Guess What Went Wrong

With evidence:

Semantic Resolution      PASS
Source Selection         PASS
Relationship Selection   FAIL
Filter Resolution        PASS
SQL Validation           PASS
Execution                PASS

The engineering team immediately knows where to investigate.

This is especially valuable because many production NL2SQL failures happen before SQL generation.

## 15. Evidence Chains Improve Evaluation

Evidence also makes offline and online evaluation more granular.

Instead of only measuring:

Exact SQL Match
Execution Accuracy
Final Answer Accuracy

teams can measure:

Metric Resolution Accuracy
Source Selection Accuracy
Relationship Selection Accuracy
Filter Resolution Accuracy
Trusted Relationship Usage
SQL Validation Pass Rate

This turns a monolithic accuracy score into a diagnostic system.

If final-answer accuracy drops, the team can determine which layer caused the regression.

## 16. Evidence as an API Object

Evidence should not exist only in the UI.

It can be part of the agent API.

For example:

{
  "answer": {
    "metric": "Net Revenue",
    "value": 27300000,
    "currency": "USD"
  },

  "evidence": {
    "metric": {
      "id": "net_revenue",
      "version": "3.2"
    },

    "sources": [
      "orders",
      "refunds",
      "billing_accounts"
    ],

    "relationships": [
      "rel_orders_billing_account"
    ],

    "query_id": "q_8271",

    "validation": "passed"
  }
}

Now downstream systems can choose how much evidence to expose.

## 17. Progressive Disclosure in the UI

Most business users do not want to read SQL after every question.

So the default UI can remain simple:

Net Revenue
$27.3M

✓ Evidence available

Then:

Why this answer?

can reveal:

Metric Definition
Source Data
Relationships
Filters
SQL
Validation

Different users can inspect different levels.

An executive may stop at the answer.

An analyst may inspect the metric and filters.

A data engineer may inspect joins and SQL.

## 18. Evidence Is Stronger Than Explanation

There is an important difference between:

Explainable

and:

Inspectable

An LLM can generate a convincing explanation.

An inspectable system exposes the artifacts that actually produced the answer.

For enterprise data, inspectability is often the stronger trust mechanism.

The user does not have to believe the explanation.

They can inspect the evidence.

## 19. A Practical Evidence Pipeline

A simplified architecture could look like:

User Question
      │
      ▼
Semantic Resolver
      │
      ├── metric_id
      └── dimension_ids
      │
      ▼
Relationship Resolver
      │
      └── relationship_ids
      │
      ▼
Context Builder
      │
      ▼
SQL Generator
      │
      └── query_id
      │
      ▼
SQL Validator
      │
      ▼
Query Executor
      │
      └── result_id
      │
      ▼
Answer Generator
      │
      ▼
Evidence Object

Every stage contributes structured evidence.

The evidence object becomes the trace that connects the original question to the final answer.

## 20. What Should Be Stored?

At minimum:

Question ID
User Question
Resolved Metric + Version
Resolved Dimensions
Source Tables / Fields
Relationship IDs
Filters
Time Interpretation
Generated SQL / Query ID
Execution Result ID
Validation Status
Timestamp

Depending on governance requirements, also consider:

Semantic Model Version
Relationship Model Version
Data Snapshot / Query Timestamp
Authorization Context
Agent Version
Model Version

This makes answers reproducible even as the system evolves.

## 21. A Better Definition of Production Readiness

For a prototype, this may be enough:

Question → SQL → Answer

For production enterprise AI, a stronger standard is:

Question
→ Governed Interpretation
→ Trusted Data Path
→ Executable Query
→ Traceable Result
→ Evidence-Backed Answer

The difference is not cosmetic.

It changes how the system can be trusted, evaluated, debugged, audited, and improved.

## Final Thoughts

AI data agents are rapidly improving at generating SQL.

But SQL generation is only one stage in a much larger correctness problem.

A production system should be able to answer two questions:

What is the answer?

and:

What evidence produced that answer?

That evidence should include the business definition, source data, relationships, filters, SQL, execution result, and validation state.

Not because every user wants to inspect every detail.

But because enterprise trust should be based on something stronger than model confidence.

A trustworthy data agent does not just return a number. It preserves the chain of evidence that makes the number verifiable.

The Reasoning Tax: Why AI Data Agents Waste Tokens Relearning Your Schema

ArisynData — Tue, 18 Aug 2026 02:52:49 +0000

If your data agent has to rediscover metric definitions, table relationships, and trusted query paths on every request, you are spending LLM reasoning on knowledge your system should already have.

AI data agents are becoming increasingly capable.

A modern agent can:

retrieve schemas;
interpret business terms;
identify candidate tables;
infer joins;
generate SQL;
validate queries;
execute them;
explain the result.

That looks like progress.

But from an engineering perspective, there is an uncomfortable question:

How much of this work is genuinely new reasoning, and how much is the agent repeatedly rediscovering facts the enterprise already knows?

That repeated work creates a hidden cost: the reasoning tax.

## A Typical Data Agent Does Too Much at Query Time

Consider this question:

What was revenue by customer last quarter?

A typical agent pipeline may look like:

User Question
      ↓
Intent Detection
      ↓
Schema Retrieval
      ↓
Business Term Resolution
      ↓
Candidate Table Selection
      ↓
Relationship Discovery
      ↓
Join Path Selection
      ↓
Metric Construction
      ↓
SQL Generation
      ↓
SQL Validation
      ↓
Execution

Now the user asks:

Show customer revenue for Q2.

The analytical intent is almost identical.

But many implementations repeat most of the pipeline.

The agent may again retrieve schemas, resolve Revenue, identify Customer, compare join paths, and choose the reporting date.

This is wasteful because much of that information is not query-specific.

## Separate Query-Time Reasoning From Reusable Knowledge

A useful engineering distinction is:

Reusable Enterprise Knowledge
vs.
Query-Specific Reasoning

### Reusable Enterprise Knowledge

Examples:

Revenue = Recognized Revenue

Customer = canonical customer entity

Revenue Date = recognition_date

Customer → Order = trusted relationship

Order → Revenue = validated query path

These should not be probabilistically reconstructed on every request.

### Query-Specific Reasoning

Examples:

What is the user asking?

Does "last quarter" mean fiscal or calendar quarter?

Should results be grouped by customer or customer segment?

Is the user asking for a comparison?

What should be investigated next?

These genuinely depend on the current request.

The design principle is:

Retrieve the known. Reason about the unknown.

## Why the Reasoning Tax Matters

There are three immediate engineering consequences.

### 1. Token Usage

Suppose schema retrieval returns 30 tables, each with:

table descriptions;
columns;
data types;
comments;
sample values.

The agent may receive thousands of tokens before reasoning even starts.

If every query repeatedly includes the same structural information, token consumption scales with query volume.

A rough model is:

Total Token Cost
≈
Requests
×
(Context Tokens + Reasoning Tokens + Output Tokens)

Reducing repeated context has a direct effect on cost.

### 2. Latency

Agent workflows often involve multiple model or tool calls:

Retrieve
→ Classify
→ Resolve
→ Plan
→ Generate
→ Validate

Even if each step takes only a small amount of time, the total latency accumulates.

If a trusted metric definition can be returned from a deterministic service in milliseconds, there is little value in asking an LLM to infer it again.

### 3. Inconsistency

Repeated reasoning also introduces variability.

One request may produce:

Revenue → invoice_amount

Another:

Revenue → recognized_revenue

Another:

Revenue → order_amount

The model may be behaving reasonably in all three cases.

The architecture is simply asking it to repeatedly solve an ambiguous problem.

Govern the definition once and retrieve it consistently.

## Schema Retrieval Is Necessary, but It Is Not Enough

A common NL2SQL architecture is:

Question
   ↓
Embedding Search
   ↓
Relevant Tables
   ↓
LLM
   ↓
SQL

This is much better than sending the entire warehouse schema.

But table retrieval still leaves unresolved questions:

Which metric is authoritative?

Which entity is canonical?

Which relationship is trusted?

Which date field should be used?

Which join path is safe?

Retrieval that returns only schema is still forcing the LLM to reconstruct business knowledge.

A richer query context should return something closer to:

question: "revenue by customer last quarter"

metric:
  name: revenue
  definition: recognized_revenue
  aggregation: SUM

dimensions:
  - customer

time:
  field: recognition_date

tables:
  - customer
  - sales_order
  - finance_revenue

trusted_path:
  - customer
  - sales_order
  - finance_revenue

Now the LLM is not discovering the data model.

It is using it.

## Precompute Relationship Knowledge

Relationship inference is a major source of unnecessary query-time reasoning.

Suppose the agent needs to connect:

Customer

to:

Payment

It may discover several candidate paths:

Customer → Order → Invoice → Payment

Customer → Account → Payment

Customer → Contract → Invoice → Payment

If the organization has already validated the first path for this analytical scenario, there is no reason to compare all three paths again.

Relationship knowledge can be discovered ahead of time using evidence such as:

Primary / Foreign Keys
Column Naming
Value Overlap
Uniqueness
Inclusion Ratio
Historical Query Patterns
Business Validation

For example:

A = order.customer_id
B = customer.customer_id

A useful signal is:

Inclusion(A → B)
=
|distinct(A) ∩ distinct(B)|
---------------------------
|distinct(A)|

Relationship candidates can then move through:

Discovered
    ↓
Candidate
    ↓
Validated
    ↓
Trusted

Query-time agents should preferentially retrieve the trusted result.

## Move Metric Resolution Out of the Prompt

Metric definitions are another common source of repeated reasoning.

Instead of embedding this in every system prompt:

When the user says revenue, use recognized_amount
from finance_revenue unless...

maintain a governed metric object:

{
  "metric": "revenue",
  "version": "2.1",
  "aggregation": "SUM",
  "table": "finance_revenue",
  "column": "recognized_amount",
  "time_field": "recognition_date",
  "status": "active"
}

The agent calls:

get_metric_definition("revenue")

and receives the current definition.

This has several advantages:

one definition across agents;
easier versioning;
easier auditing;
less prompt complexity;
fewer tokens;
less ambiguity.

## Build Query Context Before Calling the LLM

A useful architecture is:

                User Question
                     │
                     ▼
          ┌────────────────────┐
          │ Context Resolver   │
          ├────────────────────┤
          │ Business Terms     │
          │ Metrics            │
          │ Metadata           │
          │ Relationships      │
          │ Trusted Paths      │
          └─────────┬──────────┘
                    │
                    ▼
             Trusted Context
                    │
                    ▼
                  LLM
                    │
             Reason / Generate
                    │
                    ▼
                   SQL

The LLM receives only what is relevant.

This changes the role of the model.

Before:

LLM = Data Discovery + Business Interpretation + Reasoning + SQL

After:

Data Layer = Known Enterprise Facts

LLM = Intent + Reasoning + SQL / Analysis

## A Simple Context Resolver

Conceptually, the resolver could work like this:

def build_query_context(question):
    concepts = resolve_business_terms(question)

    metrics = get_metric_definitions(concepts)
    entities = get_business_entities(concepts)

    metadata = get_relevant_metadata(
        metrics=metrics,
        entities=entities
    )

    relationships = get_trusted_relationships(
        metadata.tables
    )

    return {
        "metrics": metrics,
        "entities": entities,
        "metadata": metadata,
        "relationships": relationships
    }

Then:

context = build_query_context(question)

sql = llm.generate_sql(
    question=question,
    context=context
)

The exact implementation will vary.

The architectural point is that the model does not have to infer every layer of enterprise knowledge itself.

## Cache Stable Knowledge at the Right Level

Not all knowledge changes at the same frequency.

For example:

Table Schema
→ changes occasionally

Metric Definition
→ changes occasionally

Trusted Relationship
→ changes occasionally

User Question
→ changes every request

This suggests different caching and refresh strategies.

A system might maintain:

Metadata Cache
Semantic Cache
Relationship Cache
Query Context Cache

with invalidation triggered by:

Schema Change
Metric Version Change
Relationship Update
Governance Update

This is more efficient than treating every query as a completely new reasoning problem.

## MCP Can Expose the Known Layer

For agent-based architectures, MCP can provide a clean interface to reusable enterprise knowledge.

For example:

get_metric_definition

get_business_entity

get_table_metadata

get_trusted_relationships

get_query_context

The agent workflow becomes:

Question
   ↓
Agent
   ↓
MCP Tools
   ↓
Enterprise Data Intelligence
   ↓
Trusted Context
   ↓
LLM Reasoning

MCP does not remove the need for semantic or relationship intelligence.

It gives agents a standardized way to access it.

## Do Not Confuse Precomputation With Hard-Coding

Moving knowledge out of query-time reasoning does not mean freezing the data model.

Enterprise knowledge changes.

Schemas evolve.

Metrics change.

Relationships change.

The reusable layer therefore needs:

Discover
   ↓
Detect Change
   ↓
Evaluate Impact
   ↓
Validate
   ↓
Version
   ↓
Publish
   ↓
Invalidate Cache

The objective is not:

Never reason about the data model again.

It is:

Do not reason about the same established data knowledge on every request.

## Measure the Reasoning Tax

Teams can make this problem observable.

Useful metrics might include:

### Context Tokens per Query

Average tokens sent before generation

### Reasoning Calls per Query

Average LLM calls required before SQL execution

### Schema Candidates per Query

How many tables / columns must the model evaluate?

### Relationship Resolution Rate

% of queries using prevalidated relationships

### Metric Resolution Rate

% of business metrics resolved without LLM inference

### Time to First SQL

Question received
→
Executable SQL generated

These metrics can reveal whether the agent is spending most of its time solving analytical problems or reconstructing the enterprise data model.

## The Optimization Target Changes

A common optimization question is:

Which model gives the best SQL accuracy?

That still matters.

But production systems should also ask:

How much unnecessary reasoning are we forcing the model to perform?

A more capable model may hide poor architecture by successfully reasoning through large amounts of noisy context.

That does not make the architecture efficient.

The better system may be the one that gives the model less to figure out.

## Final Thoughts

AI agents are good at reasoning.

Reasoning is also probabilistic, expensive, and slower than deterministic lookup.

So use it where it adds value.

If Revenue already has a governed definition:

retrieve it.

If Customer already has a canonical entity:

retrieve it.

If a join path is already trusted:

retrieve it.

Then let the model solve the genuinely new problem:

what the user is asking and how to analyze it.

The next generation of enterprise data agents may not win by thinking harder.

They may win by knowing what they no longer need to think about.

Why Every AI Agent Shouldn't Have to Rediscover Your Data Model

ArisynData — Fri, 14 Aug 2026 10:23:00 +0000

Enterprise AI is moving toward a multi-agent architecture.

Instead of one general-purpose assistant, organizations are starting to build specialized agents for:

sales;
finance;
operations;
customer support;
analytics;
management.

That creates an engineering problem that is easy to miss.

If every agent independently receives database schemas, retrieves metadata, learns metric definitions, discovers join paths, and builds its own interpretation of the business, then every new agent becomes another data-modeling project.

The result is not only duplicated engineering work.

It is duplicated enterprise understanding.

A better architecture is to separate agent reasoning from shared data intelligence.

## The Problem With Agent-Local Data Knowledge

A simple data agent often looks something like:

User
  ↓
Agent
  ├── System Prompt
  ├── Schema Retrieval
  ├── Metric Definitions
  ├── Join Instructions
  ├── SQL Tool
  └── LLM
        ↓
    Database

For a single proof of concept, this is reasonable.

Now imagine five teams building five agents:

Sales Agent
Finance Agent
Supply Chain Agent
Support Agent
Analytics Agent

Each team has to solve the same problems:

Which tables are relevant?

What does "Customer" mean?

Which definition of Revenue is authoritative?

How should Customer connect to Order?

Which data source should be trusted?

The implementation may be different, but the underlying enterprise knowledge is largely the same.

If that knowledge lives inside each agent, duplication begins immediately.

## Example: Five Agents, Three Definitions of Revenue

Suppose an enterprise contains:

sales_order.total_amount
finance_invoice.invoice_amount
finance_revenue.recognized_amount
payment.received_amount

A sales agent might map:

Revenue → sales_order.total_amount

A finance agent might use:

Revenue → finance_revenue.recognized_amount

An analytics agent might retrieve:

Revenue → finance_invoice.invoice_amount

All three agents may generate syntactically correct SQL.

All three queries may execute successfully.

But the organization now has three AI systems answering the same business question differently.

The problem is not LLM reasoning.

The problem is that business meaning was implemented locally inside each agent.

## Business Knowledge Should Be an Enterprise Dependency

Agent-specific knowledge and enterprise-wide knowledge should be separated.

An agent may legitimately own:

task instructions;
workflow logic;
persona;
tool selection;
planning strategy.

But definitions such as these should not be duplicated:

Customer
Revenue
Active Customer
Inventory Balance
Gross Margin

The same applies to data relationships:

Customer
   ↓
Order
   ↓
Invoice
   ↓
Payment

These are enterprise data assets.

They should be reusable dependencies.

## Thick Agents vs. Thin Agents

A thick agent contains a large amount of enterprise-specific intelligence:

Agent
├── LLM
├── Business Semantics
├── Metrics
├── Schema Knowledge
├── Relationship Knowledge
├── Business Rules
├── Query Logic
└── Tools

This creates coupling.

If the Revenue definition changes, multiple agents may need to change.

If a trusted join path changes, multiple prompts or tools may need to be updated.

If the organization switches models, important business knowledge may be buried inside model-specific implementation.

A thinner design looks different:

Agent
├── LLM
├── Task Logic
├── Planning
└── Tools
      ↓
Shared Data Intelligence
├── Business Semantics
├── Metrics & Dimensions
├── Metadata
├── Trusted Relationships
└── Query Context
      ↓
Enterprise Data

Now the agent does not need to rediscover the enterprise every time it answers a question.

## What Should the Shared Data Layer Expose?

The shared layer does not have to be one monolithic service.

It can expose several reusable capabilities.

### 1. Metadata Retrieval

For example:

{
  "table": "sales_order",
  "columns": [
    "order_id",
    "customer_id",
    "order_date",
    "total_amount"
  ]
}

But raw metadata alone is not enough.

### 2. Business Semantics

The agent should be able to resolve:

"recognized revenue"

into something like:

metric: revenue
definition: recognized revenue
aggregation: SUM
source:
  table: finance_revenue
  column: recognized_amount

This definition can then be reused by every authorized agent.

### 3. Relationship Discovery

Suppose the agent needs Customer and Payment data.

It should not have to guess the join path from raw schemas.

A relationship service could return:

Customer
   ↓ customer_id
Order
   ↓ order_id
Invoice
   ↓ invoice_id
Payment

along with evidence or confidence information.

For example:

{
  "path": [
    "customer",
    "sales_order",
    "invoice",
    "payment"
  ],
  "status": "trusted"
}

### 4. Query Context

Instead of sending an entire warehouse schema into the prompt, the agent can receive only the relevant context:

Question
   ↓
Resolve Business Concepts
   ↓
Retrieve Relevant Metadata
   ↓
Retrieve Trusted Relationships
   ↓
Generate Query Context
   ↓
LLM

This reduces unnecessary context and gives the model more targeted information.

## Why Schema Retrieval Alone Is Not Enough

A common pattern today is:

Question
   ↓
Embedding Search
   ↓
Relevant Tables
   ↓
LLM
   ↓
SQL

This is useful.

It solves the problem of sending thousands of tables to the model.

But it does not answer:

Which metric definition is authoritative?

Which relationship is trusted?

Which business entity does this table represent?

Retrieval answers:

What looks relevant?

A shared data intelligence layer must also answer:

What is valid for this business question?

That distinction matters more as the number of agents grows.

## Relationship Knowledge Should Be Evidence-Based

Enterprise relationships are often not fully represented by foreign keys.

A relationship engine can use several signals:

Database constraints
Naming similarity
Value overlap
Uniqueness
Inclusion relationships
Validated business mappings

For example, if:

A = order.customer_id
B = customer.customer_id

we can calculate:

Inclusion(A → B)
=
|distinct(A) ∩ distinct(B)|
---------------------------
|distinct(A)|

A high inclusion ratio can provide evidence that A references B.

But discovered relationships should not automatically become business truth.

A safer lifecycle is:

Discovered
    ↓
Candidate
    ↓
Validated
    ↓
Trusted

Agents should preferentially consume trusted relationships rather than independently inventing joins.

## MCP Fits Here — But It Is Not the Data Model

MCP gives agents a standardized way to discover and call tools.

That makes it a natural interface for shared data capabilities.

For example, an enterprise data MCP server could expose tools such as:

get_metric_definition()

get_table_metadata()

discover_relationships()

get_query_context()

An agent could then call:

Agent
  ↓
MCP
  ↓
Shared Data Intelligence Services

But MCP itself does not determine:

What Revenue means

Which customer definition is authoritative

Which join path is trusted

Those decisions come from the data intelligence behind the interface.

A useful distinction is:

MCP standardizes access. Shared data intelligence standardizes understanding.

## Centralization Does Not Mean Static Knowledge

Moving enterprise knowledge into a shared layer solves duplication, but it creates another responsibility:

the shared knowledge must stay current.

Enterprise systems continuously change:

New tables
New columns
New metrics
New business rules
New relationships

So the shared layer needs its own lifecycle:

Discover
   ↓
Detect Change
   ↓
Evaluate Impact
   ↓
Validate
   ↓
Version
   ↓
Publish

Otherwise the organization simply replaces many stale agent configurations with one stale central configuration.

## What Happens When a Metric Changes?

Suppose:

Revenue v1
=
SUM(invoice_amount)

changes to:

Revenue v2
=
SUM(recognized_amount)

With agent-local knowledge:

Sales Agent → update
Finance Agent → update
Analytics Agent → update
Management Agent → update
Support Agent → maybe update

Someone has to discover every dependency.

With shared semantics:

Revenue Definition
       ↓
Version Update
       ↓
Shared Data Intelligence
       ↓
All Authorized Agents

The enterprise changes the definition once.

Agents consume the updated version.

That is a much cleaner dependency model.

## Shared Data Intelligence Also Reduces Agent Coupling

There is another benefit.

Agent frameworks are changing quickly.

Organizations may switch between:

models;
orchestration frameworks;
agent runtimes;
application interfaces.

Business knowledge should survive those changes.

If metric definitions, data relationships, and business mappings are independent of the agent implementation, agents become easier to replace.

That suggests an architectural principle:

Agents = Replaceable Compute / Reasoning

Enterprise Data Intelligence = Durable Knowledge

The enterprise should own its understanding of itself.

Not the current agent framework.

## A Practical Multi-Agent Data Architecture

A simplified architecture might look like:

               Sales Agent
                    │
              Finance Agent
                    │
             Analytics Agent
                    │
             Operations Agent
                    │
                    ▼
        ┌──────────────────────┐
        │ Shared Data          │
        │ Intelligence Layer   │
        ├──────────────────────┤
        │ Business Semantics   │
        │ Metrics & Dimensions │
        │ Metadata             │
        │ Relationships        │
        │ Trusted Query Paths  │
        └──────────────────────┘
                    │
                    ▼
             Enterprise Data

The shared layer does not replace agents.

It makes them thinner.

It does not replace the LLM.

It gives the LLM a consistent representation of the enterprise.

## Final Thoughts

The first generation of enterprise agents focused on making individual agents more capable.

The multi-agent era creates a different engineering problem:

How do many agents share one consistent understanding of enterprise data?

If every agent independently discovers schemas, defines metrics, and guesses relationships, organizations will create duplicated logic and inconsistent answers.

The better pattern is separation of concerns:

Agents reason.

Shared data intelligence defines what the enterprise means.

Models will change.

Agents will change.

Frameworks will change.

Your enterprise should not have to rediscover its own data model every time they do.

Semantic Drift: The Hidden Failure Mode of Enterprise AI Analytics

ArisynData — Wed, 12 Aug 2026 13:18:00 +0000

Enterprise AI systems rarely fail only because the model is weak.

A more subtle failure happens when the model is working correctly, the SQL executes successfully, and the result looks reasonable — but the system is reasoning over an outdated representation of the business.

This is semantic drift.

As more teams put semantic layers between LLMs and enterprise data, maintaining those semantics becomes a production engineering problem rather than a one-time modeling task.

The database changes.

Business definitions change.

Relationships change.

If the AI's understanding does not change with them, accuracy degrades quietly.

A Semantic Layer Is Runtime Infrastructure

A common AI analytics architecture looks roughly like this:

User Question
     ↓
LLM / Query Agent
     ↓
Semantic Layer
     ↓
Enterprise Data

The semantic layer may contain:

business terms;
metric definitions;
dimensions;
mappings to physical tables and columns;
relationships used to generate queries.

This helps prevent an LLM from guessing directly from raw schemas.

But there is an important consequence:

Once AI depends on the semantic layer at query time, stale semantics become a runtime failure.

The problem is that semantic models are often maintained as if they were documentation.

Define them once.

Review them.

Publish them.

Then assume they remain correct.

Enterprise data does not behave that way.

Failure Mode 1: Schema Drift

Suppose a metric is mapped to:

customer.customer_id

After a warehouse migration, the organization introduces:

account_customer.customer_key

The old table may remain available for compatibility.

That creates a dangerous situation.

Nothing necessarily breaks.

The old query can still execute.

The semantic mapping is simply pointing to a representation that is no longer authoritative.

Traditional schema monitoring may tell you that a column was added.

AI analytics needs to answer a harder question:

Does this schema change invalidate any business meaning used by the AI?

That requires connecting physical metadata changes to semantic dependencies.

Failure Mode 2: Semantic Drift

Semantic drift happens even when the schema does not change.

Consider:

Active Customer

Version 1:

customer.status = 'ACTIVE'

Later, the business changes the definition:

customer with >= 1 completed order in the last 90 days

The database can remain exactly the same.

But the meaning has changed.

An AI system using the old definition may continue returning perfectly valid SQL and perfectly wrong business answers.

This is why execution success is a weak validation signal for enterprise AI.

SQL succeeded != business meaning is correct

Failure Mode 3: Relationship Drift

This problem becomes more interesting when queries span multiple tables.

Imagine the original analytical path is:

Customer
   ↓ customer_id
Order

After an ERP redesign:

Customer
   ↓
Account
   ↓
Order

The old join may still work because legacy identifiers remain populated.

But the organization has changed the business relationship.

An LLM that sees both paths now has multiple executable options.

Which one is correct?

This cannot be solved reliably with schema retrieval alone.

The system needs maintained relationship knowledge.

Why Embeddings Do Not Solve Drift

A common approach is to embed schema metadata and retrieve relevant tables for each question.

For example:

User Question
     ↓
Embedding Search
     ↓
Relevant Tables / Columns
     ↓
LLM
     ↓
SQL

This is useful for reducing schema size.

But semantic similarity does not tell us whether a definition is current.

Embedding search may find:

invoice_amount
recognized_revenue
payment_amount

for the term:

revenue

All three are semantically related.

Only business governance can determine which one currently represents the metric.

Likewise, embeddings may identify two similar identifiers, but similarity does not prove that they form the trusted join path.

Retrieval solves relevance.

It does not solve validity.

Treat Semantic Assets Like Versioned Code

If semantic definitions affect AI-generated answers, they should be treated more like production code than documentation.

A metric definition should have:

metric: revenue
version: 2.1
status: active
definition: recognized revenue
source:
  table: finance_revenue
  column: recognized_amount
valid_from: 2026-07-01

A relationship should also carry explicit evidence:

relationship:
  from: customer.customer_id
  to: account.customer_id
type: business_validated
confidence: 0.97
status: active

The exact format is less important than the engineering principle:

Meaning needs identity, state, history, and validation.

Without versioning, it becomes difficult to answer basic production questions:

Which definition generated this answer?
When did that definition change?
Which queries are affected?
Can we roll back?

Detect Changes Before They Become Wrong Answers

A living semantic system needs change detection.

At the physical layer, monitor:

New table
Column added
Column removed
Type changed
Constraint changed

At the relationship layer:

New candidate relationship
Join coverage changed
Identifier uniqueness changed
Relationship confidence changed

At the semantic layer:

Metric definition changed
Mapping became ambiguous
Business term changed
New semantic version published

The important part is not generating more alerts.

It is calculating impact.

For example:

finance_invoice.invoice_amount changed
                 ↓
revenue metric mapping affected
                 ↓
12 query templates affected
                 ↓
AI queries using Revenue require validation

Now schema monitoring becomes useful to AI governance.

Relationship Discovery Should Produce Candidates, Not Truth

Automatically discovering relationships can help keep a data model current.

Signals may include:

declared primary/foreign keys;
naming similarity;
value overlap;
uniqueness;
inclusion ratio;
historical joins.

For two columns A and B, a simple inclusion signal could be:

inclusion(A → B)
=
|distinct(A) ∩ distinct(B)|
---------------------------
|distinct(A)|

A high value can indicate a possible relationship.

But it should not automatically become trusted business logic.

A better lifecycle is:

Relationship Detected
        ↓
Candidate
        ↓
Evidence / Confidence
        ↓
Validation
        ↓
Trusted Relationship

This distinction matters.

Automated discovery improves coverage.

Governance establishes trust.

Human-in-the-Loop Is a Feature, Not a Failure

Enterprise semantics often cannot be inferred safely from technical metadata alone.

If the system finds:

Revenue

mapped plausibly to both:

sales_order.total_amount

and:

finance_revenue.recognized_amount

the correct behavior may be to ask for clarification.

A useful workflow is:

Ambiguity Detected
       ↓
Candidate Definitions
       ↓
Human Review
       ↓
Validate With Query
       ↓
Publish New Version

The goal of automation is not to eliminate domain experts.

It is to stop asking them to manually rediscover every schema and relationship change.

A Practical Living Data Model Loop

Putting the pieces together, the lifecycle looks like:

1. Discover metadata
        ↓
2. Detect schema and relationship changes
        ↓
3. Identify impacted semantic assets
        ↓
4. Generate candidate updates
        ↓
5. Validate ambiguous business meaning
        ↓
6. Version and publish
        ↓
7. Use validated semantics for AI queries
        ↓
8. Repeat

This is fundamentally different from:

Build semantic layer → Done

The semantic model becomes an operational system.

What Should Be Monitored?

A few useful signals include:

Semantic Coverage

How much of the active analytical surface has governed meaning?

governed metrics / queried metrics

Relationship Coverage

How many required multi-table query paths are backed by trusted relationships?

Ambiguity Rate

How often does a business term map to multiple plausible definitions?

Stale Mapping Rate

How many semantic mappings reference changed or deprecated physical assets?

Validation Failure Rate

How often do proposed semantic or relationship updates fail business validation?

These metrics tell you more about production readiness than simply measuring whether SQL execution succeeds.

The Key Engineering Shift

The first generation of LLM analytics focused heavily on query generation.

The next engineering challenge is maintaining the data knowledge used to generate those queries.

A stronger model cannot compensate for a stale business definition.

A larger context window cannot determine whether an old join path is still authoritative.

And better embeddings cannot decide when the organization changed the meaning of revenue.

The system needs a maintained layer of enterprise data knowledge.

Not static metadata.

Not a one-time semantic project.

A living model.

Final Thoughts

If your AI analytics system depends on business semantics, those semantics are production infrastructure.

Treat them accordingly.

Monitor changes.

Track relationships.

Version definitions.

Detect ambiguity.

Validate business meaning.

Because the most dangerous enterprise AI failure is not always a broken query.

Sometimes the query works perfectly — against yesterday's understanding of the business.

Building AI Analysts: Why Enterprise Analytics Needs More Than BI Dashboards

ArisynData — Mon, 10 Aug 2026 12:46:00 +0000

For decades, enterprises have relied on dashboards as the primary way to consume business data.

A typical workflow looks like:

Business Question
↓
Find the Right Dashboard
↓
Select Filters
↓
Review Metrics
↓
Human Analysis

This model works well for predefined questions.

But modern businesses increasingly ask questions that were never designed into dashboards:

Why did sales decline in East China last month?

Answering this requires combining:

sales data;
customer data;
product data;
regional information;
order history.

A dashboard can show what happened.

But understanding why it happened requires deeper analysis.

This is where AI Analysts become valuable.

From BI Dashboards to AI Analysts

AI changes the interaction model.

Instead of searching for reports, users can directly express business questions.

Business Question

↓

AI Understanding

↓

Data Discovery

↓

Multi-dimensional Analysis

↓

Business Insight

The goal is not simply generating charts.

The goal is helping users understand:

What happened?
Why did it happen?
What factors contributed?
What should be investigated next?

Why Building AI Analysts Is Difficult

Connecting an LLM to a database is not enough.

Enterprise analytics requires understanding the business behind the data.

An AI Analyst needs to understand:

business concepts;
data relationships;
analytical logic.

Understanding Business Concepts

Business users do not think in tables.

They ask:

Show me high-value customers.

But enterprises may have:

customer_table

account_table

crm_customer

billing_customer

The AI needs to understand:

Which customer definition is correct?
Which attributes define value?
Which data source should be trusted?

A business concept is not always a table.

It is a combination of entities, metrics, rules, and relationships.

Understanding Data Relationships

Enterprise analysis rarely depends on one table.

A typical path may involve:

Customer
↓
Order
↓
Contract
↓
Payment

However, enterprises often contain:

multiple applications;
duplicated entities;
different identifiers;
historical data models.

A database may allow many joins.

But only some relationships represent business logic.

The important question is not:

Can these tables be joined?

It is:

Is this the correct business relationship?

Moving From Answers to Analysis

Traditional systems answer:

What is the number?

AI Analysts should answer:

Why is this number changing?

For example:

Sales decreased last month. Why?

The system should analyze:

regional changes;
customer changes;
product changes;
historical trends.

This requires dynamic analysis instead of fixed dashboards.

The Architecture Behind AI-Native Analytics

A future analytics system is not simply:

User
↓
LLM
↓
Database

A more reliable architecture requires:

User Question

↓

AI Understanding

↓

Semantic Layer

↓

Relationship Intelligence

↓

Query Generation

↓

Analysis Result

AI needs to understand:

Business Semantics

What does the data mean?

Examples:

revenue definition;
active customer definition;
inventory calculation.

Data Relationships

How is enterprise data connected?

Examples:

customer to order;
order to invoice;
invoice to payment.

Analytical Context

What analysis should happen next?

Examples:

comparison;
trend analysis;
root cause exploration.

The Future of Enterprise Analytics

The next generation of analytics will combine:

large language models;
semantic understanding;
relationship intelligence;
enterprise data governance.

The future is not about creating more dashboards.

It is about creating AI systems that can become intelligent partners in business analysis.

Final Thoughts

Traditional BI helped enterprises understand what happened.

AI Analysts will help enterprises understand why it happened.

But this requires more than connecting an LLM to a database.

AI needs to understand business concepts, data relationships, and analytical context.

Only then can enterprise AI move from answering questions to supporting real analysis.

Beyond Data Lineage: Building Observability for AI-Generated Queries

ArisynData — Wed, 05 Aug 2026 11:20:00 +0000

Enterprise AI is moving from experimentation to production.

Organizations are deploying AI assistants that can query databases, generate reports, and analyze business data.

The first challenge was making AI capable of answering questions.

The next challenge is making those answers trustworthy.

When an AI system provides an answer, enterprises need to understand:

What data did AI use?
Why did AI choose those tables?
Which business definitions were applied?
How was the final answer generated?

This is where traditional data observability needs to evolve.

Traditional Data Observability Was Built for Data Pipelines

For years, enterprises focused on understanding how data moved.

A typical lineage looks like:

Source Database

↓

ETL Pipeline

↓

Data Warehouse

↓

BI Report

Organizations needed to know:

Where did data come from?
Which pipeline transformed it?
Which reports depend on it?

This became the foundation of modern data governance.

However, AI introduces a different type of data consumption.

AI does not only move data.

AI interprets data.

AI Creates a New Observability Challenge

Consider a question:

Why did sales decrease last month?

An AI assistant may:

Understand the question.
Select relevant data sources.
Generate SQL.
Execute the query.
Explain the result.

The final answer may look reasonable.

But enterprises need more visibility.

They need to understand:

User Question

↓

AI Interpretation

↓

Business Definition

↓

Selected Data Sources

↓

Relationship Path

↓

Generated SQL

↓

Final Answer

This is different from traditional data lineage.

It is AI reasoning lineage.

Why AI Query Results Need Explainability

Traditional BI systems use predefined reports.

Users usually know:

which metric is used;
which data source is connected;
how calculations are performed.

AI-powered analytics changes this.

The query path can be generated dynamically.

The selected tables may change.

The SQL may be different for different questions.

Therefore, AI systems need to explain not only the answer, but also the reasoning behind the answer.

The Four Layers of AI Data Observability

AI data observability extends traditional data governance with new capabilities.

1. Query Lineage

Traditional lineage asks:

Where did this data come from?

AI query lineage asks:

How did AI produce this answer?

It should capture:

user question;
generated SQL;
selected tables;
filters;
returned datasets.

2. Semantic Lineage

Enterprise concepts often have multiple meanings.

For example:

Revenue may mean:

sales revenue;
recognized revenue;
invoice amount;
payment amount.

AI observability needs to record:

which business definition was used;
which metric mapping was applied;
which semantic rules affected the answer.

3. Relationship Lineage

Enterprise queries often depend on relationships:

Customer

↓

Order

↓

Invoice

↓

Payment

AI needs to know:

which relationship path was selected;
why this path was chosen;
whether the relationship is trusted.

A valid SQL join does not always represent a valid business relationship.

4. Answer Explainability

Enterprise users need more than a result.

They need to understand:

where the data came from;
what logic was applied;
what assumptions were made.

This is critical in industries where decisions depend on reliable data.

From Data Governance to AI Governance

Traditional data governance focused on:

data quality;
metadata;
lineage;
ownership.

AI introduces new requirements:

AI query traceability;
semantic consistency;
relationship validation;
answer transparency.

The question is changing from:

"Can we manage enterprise data?"

to:

"Can we trust how AI uses enterprise data?"

Final Thoughts

AI can generate answers quickly.

But enterprises need more than speed.

They need confidence.

Traditional data lineage helped organizations understand how data moved.

AI observability will help organizations understand how AI reasons over data.

The future of enterprise AI will belong to systems that are not only intelligent, but also transparent and trustworthy.

Building Data Intelligence for Reliable Enterprise AI

ArisynData — Mon, 03 Aug 2026 11:41:00 +0000

Large language models are becoming more capable every year.

They can generate SQL, write code, analyze documents, and interact with enterprise systems through tools.

However, when organizations move AI applications from prototypes to production, a common problem appears:

AI can access enterprise data.

But access does not mean understanding.

Many enterprise AI failures are not caused by weak models.

They are caused by the fact that enterprise data was designed for applications and humans, not for AI systems.

Data Access Is Not Data Intelligence

Modern enterprises already have mature data platforms:

relational databases;
data warehouses;
data lakes;
BI systems;
metadata management platforms.

These systems solve an important problem:

How do we store and access data?

AI introduces a different challenge:

How does AI understand and use this data correctly?

A database can tell an AI system:

this is a customer table;
this is a revenue column;
this field contains an identifier.

But it usually cannot explain:

which customer definition is trusted;
what revenue means in this business context;
how data from different systems should be connected.

The difference is the gap between data access and data intelligence.

Why Traditional Metadata Is Not Enough

Many enterprise AI systems start by providing metadata to LLMs.

For example:

Table: customer

Columns:
customer_id
customer_name
create_time

This information helps AI discover available data.

But it does not provide enough business understanding.

Consider a simple question:

Who are our most valuable customers?

An enterprise may have:

crm_customer

erp_customer

billing_customer

All three tables may contain customer information.

The challenge is not finding these tables.

The challenge is understanding:

Are they the same business entity?
Which one is authoritative?
How should they be related?

Without this knowledge, AI has to guess.

And guessing creates unreliable answers.

Data Intelligence Starts With Business Meaning

Enterprise data contains many ambiguous concepts.

A common example is:

Revenue

It may represent:

sales revenue;
recognized revenue;
invoice amount;
payment amount;
contract value.

From a database perspective, all of these are valid fields.

From a business perspective, they answer different questions.

Humans resolve this ambiguity through experience.

AI needs this business meaning to be explicitly available.

This is why semantic understanding is a fundamental requirement for enterprise AI.

Relationships Are the Hidden Challenge

Enterprise data is not a collection of isolated tables.

Business processes connect data together.

A simple example:

Customer

↓

Order

↓

Invoice

↓

Payment

However, real enterprise environments are much more complex.

Organizations often have:

multiple systems;
duplicated entities;
inconsistent identifiers;
historical data models.

A database may allow many possible joins.

But only some relationships represent real business logic.

A technically valid join does not always mean a business-valid relationship.

For AI applications, trusted relationships matter.

Why Bigger LLMs Are Not Enough

A common assumption is:

"If the model becomes smarter, it will solve enterprise data problems."

Better models definitely improve reasoning capabilities.

But models cannot automatically know enterprise-specific knowledge.

They cannot magically determine:

which data source the company trusts;
which metric definition is correct;
which relationship represents the real business process.

The problem is not only intelligence.

The problem is the intelligence available around the data.

Building an AI-Ready Data Intelligence Layer

Reliable enterprise AI requires more than connecting an LLM to a database.

It requires data intelligence capabilities.

Business Entity Understanding

AI needs to understand important business objects:

Customer;
Product;
Supplier;
Contract;
Project.

Not just as database tables.

As business entities.

Data Semantic Understanding

AI needs to understand:

business terms;
metric definitions;
calculation logic;
organizational language.

The same field name can represent different meanings across organizations.

Trusted Data Relationships

AI needs to understand:

which entities are connected;
which relationships are reliable;
which data paths should be used.

This enables AI to generate answers based on trusted enterprise knowledge.

The Future of Enterprise AI Data Architecture

Traditional data platforms answered:

Where is the data?

AI requires a deeper answer:

What does this data mean?

How is it connected?

Can AI trust it?

The next generation of enterprise data architecture will not replace databases, warehouses, or lakes.

Instead, it will add intelligence that makes enterprise data understandable and usable for AI systems.

Final Thoughts

Enterprise AI is not only a model problem.

It is also a data intelligence problem.

Large language models provide reasoning capabilities.

Data intelligence provides the foundation that allows those capabilities to work reliably.

The future of enterprise AI will not only depend on smarter models.

It will depend on smarter data.