DEV Community: Arisyn

Building Answer Lineage for Enterprise Data Agents

Arisyn — Tue, 15 Sep 2026 23:45:00 +0000

If a data agent returns €18.6M, production observability should tell you exactly how that answer was constructed.

Most Text-to-SQL pipelines log prompts, generated SQL, latency, tokens, and execution status. Those signals matter, but when a business user says, “This number is wrong,” they are not enough.

The failure may have happened before SQL generation:

"Revenue"      → wrong metric
"Germany"      → wrong dimension
Customer data  → wrong relationship path
"Last quarter" → wrong time interpretation

The SQL can be valid while encoding the wrong business meaning.

For enterprise data agents, observability should cover the entire path from question to answer. That is Answer Lineage.

Data Lineage vs. Answer Lineage

Traditional data lineage answers:

Where did this dataset or column come from?

ERP → ETL → Warehouse → Semantic Model → Dashboard

Answer lineage asks:

How did this natural-language question become this business answer?

Question
   ↓
Resolved Intent
   ↓
Semantic Objects
   ↓
Selected Data
   ↓
Relationship Path
   ↓
Query Plan
   ↓
SQL
   ↓
Execution Result
   ↓
Answer

Data lineage follows data. Answer lineage follows a decision request.

Start With an AnswerLineage Object

Create the lineage object when the request begins, not after the answer is generated.

{
  "lineage_id": "al_78291",
  "question": "What was revenue in Germany last quarter?",
  "status": "running",
  "created_at": "2026-09-11T10:24:18Z"
}

Enrich it as the pipeline executes.

A completed object might contain:

{
  "lineage_id": "al_78291",

  "intent": {
    "metric": "revenue",
    "dimension": "country",
    "time_range": "last_quarter"
  },

  "semantic_resolution": {
    "metric": {
      "name": "Recognized Revenue",
      "version": "v4",
      "owner": "Finance"
    },
    "dimension": {
      "name": "Customer Country"
    }
  },

  "filters": {
    "country": "Germany",
    "time_range": "2026-Q2"
  },

  "fields": [
    "finance_revenue.recognized_amount",
    "customer.country_code"
  ],

  "relationship_path": [
    "customer -> account",
    "account -> sales_order",
    "sales_order -> finance_revenue"
  ],

  "query_id": "q_91821",

  "answer": {
    "value": 18600000,
    "currency": "EUR"
  }
}

Now €18.6M is not an isolated output. It has provenance.

Capture Lineage as Events

An append-only event stream works well for multi-stage agents:

question_received
intent_resolved
semantic_object_selected
relationship_path_selected
query_plan_created
sql_generated
query_executed
answer_generated

Example:

{
  "lineage_id": "al_78291",
  "event": "semantic_object_selected",
  "payload": {
    "type": "metric",
    "name": "Recognized Revenue",
    "version": "v4"
  }
}

Later:

{
  "lineage_id": "al_78291",
  "event": "relationship_path_selected",
  "payload": {
    "path": [
      "customer -> account",
      "account -> sales_order",
      "sales_order -> finance_revenue"
    ]
  }
}

This provides auditability, ordering, replay, debugging, and version history.

Capture Intent Evidence

The first important artifact is not SQL. It is the system's interpretation of the question.

{
  "metric": "revenue",
  "dimension": "country",
  "filter_value": "Germany",
  "time_expression": "last quarter"
}

If ambiguity exists, preserve it:

{
  "metric": {
    "status": "ambiguous",
    "candidates": [
      "recognized_revenue",
      "invoice_amount"
    ]
  }
}

Otherwise engineers may blame SQL for what was actually an intent-resolution failure.

Capture Semantic Evidence

Record how business language mapped to governed concepts:

{
  "input_term": "revenue",
  "resolved_metric": {
    "id": "metric.recognized_revenue",
    "name": "Recognized Revenue",
    "version": "v4",
    "status": "certified",
    "owner": "Finance"
  }
}

The lineage should preserve:

User Language
      ↓
Business Concept
      ↓
Physical Mapping

not merely the final column name.

Capture Relationship Evidence

For multi-table questions, store the path selected by the planner:

{
  "path_id": "rp_4821",
  "nodes": [
    "customer",
    "account",
    "sales_order",
    "finance_revenue"
  ],
  "edges": [
    {
      "from": "customer.customer_id",
      "to": "account.customer_id",
      "status": "trusted"
    },
    {
      "from": "account.account_id",
      "to": "sales_order.account_id",
      "status": "trusted"
    }
  ]
}

Why separate this from SQL?

Because the relationship decision is a reasoning artifact. SQL is only its execution representation.

If the answer is inflated by fanout, this is often the first artifact to inspect.

Preserve a Semantic Query Plan

Before generating SQL, create a structured plan:

{
  "metric": {
    "field": "finance_revenue.recognized_amount",
    "aggregation": "SUM"
  },

  "dimension": {
    "field": "customer.country_code"
  },

  "filters": [
    {
      "field": "customer.country_code",
      "operator": "=",
      "value": "DE"
    }
  ],

  "time_filter": {
    "field": "finance_revenue.recognition_date",
    "start": "2026-04-01",
    "end": "2026-06-30"
  },

  "relationship_path_id": "rp_4821"
}

This separates the business query plan from the SQL dialect and makes semantic, relationship, grain, and policy validation easier.

Store SQL as Evidence, Not the Whole Explanation

Generated SQL belongs in lineage:

{
  "query_id": "q_91821",
  "dialect": "postgresql",
  "sql_hash": "sha256:...",
  "execution_status": "success",
  "row_count": 1
}

But SQL tells you what executed, not why those business choices were made.

You still need semantic and relationship evidence.

Link Results to Answers

Execution output should remain traceable:

{
  "query_id": "q_91821",
  "result": {
    "columns": ["recognized_revenue"],
    "rows": [[18600000]]
  }
}

Then link the user-facing answer:

{
  "answer": "Revenue in Germany last quarter was €18.6M.",
  "evidence": {
    "query_id": "q_91821",
    "lineage_id": "al_78291"
  }
}

The statement is now connected to the actual execution artifact.

Make Historical Evidence Reproducible

If Revenue v4 becomes Revenue v5 tomorrow, yesterday's answer should still point to v4.

Persist versions for:

Metric definitions
Semantic mappings
Relationship definitions
Policies

Reproducibility depends on historical identity, not only current configuration.

Useful provenance fields include:

Source
Version
Owner
Status
Timestamp
Validation State

This turns lineage from a debugging trace into governance evidence.

Evidence Is Not Chain-of-Thought

An enterprise evidence layer should expose verifiable artifacts, not hidden model reasoning.

Useful evidence includes:

Resolved Metric
Metric Version
Selected Fields
Relationship Path
Filters
Time Range
SQL
Query Result

A generated explanation describes.

Evidence artifacts verify.

Design an Evidence API

A simple API might be:

GET /answers/{answer_id}/evidence

with a response such as:

{
  "answer_id": "ans_1192",
  "lineage_id": "al_78291",

  "summary": {
    "metric": "Recognized Revenue",
    "dimension": "Customer Country",
    "filter": "Germany",
    "time_range": "2026 Q2"
  },

  "semantic_evidence": "...",
  "relationship_evidence": "...",
  "query_evidence": "...",
  "governance_evidence": "..."
}

The UI can then progressively disclose detail.

A business user sees:

€18.6M
Metric: Recognized Revenue
Period: Q2 2026
Country: Germany
[View Evidence]

A technical view can expose fields, relationship paths, SQL, query ID, and execution metadata.

Build an Evidence Graph

For complex systems, lineage can be modeled as a graph:

Question
   │
   ├── Intent
   ├── Metric ──→ Metric Version
   ├── Dimension
   ├── Relationship Path
   │        ├── Edge 1
   │        ├── Edge 2
   │        └── Edge 3
   ├── Query Plan
   ├── SQL
   ├── Result
   └── Answer

This enables questions such as:

Which answers used Revenue v4?
Which answers used this relationship edge?
Which answers depended on this field?
Which answers were generated before a metric was deprecated?

Answer lineage starts becoming an impact-analysis system.

Debugging Becomes Faster

Suppose:

Expected: €16.9M
Returned: €18.6M

Without lineage:

Reproduce
Inspect prompt
Inspect SQL
Guess

With lineage:

Intent        ✓
Metric        ✓
Dimension     ✓
Time Range    ✓
Relationship  ✕

Now you know where to investigate.

That makes answer lineage an observability capability, not merely a compliance feature.

Add Validation Status

Each stage can carry validation state:

{
  "semantic_resolution": {
    "status": "validated"
  },
  "relationship_path": {
    "status": "trusted"
  },
  "query_plan": {
    "status": "validated"
  },
  "execution": {
    "status": "success"
  }
}

A user-facing trust summary can then show:

Semantic Context       ✓
Relationship Context   ✓
Query Validation       ✓
Execution              ✓

This is more meaningful than an arbitrary Confidence: 94%.

The Evidence Test

A simple production test for any data agent:

Ask:

What was revenue by region last quarter?

Let it answer.

Then ask:

Prove it.

The system should be able to retrieve:

Business Definition
Metric
Fields
Relationship Path
Filters
Time Range
SQL
Result

If it cannot, the answer may still be correct—but it is difficult to independently verify, audit, or debug.

Reference Architecture

                  User Question
                        ↓
                 Intent Resolver
                        │
                        ├── evidence event
                        ↓
                Semantic Resolver
                        │
                        ├── evidence event
                        ↓
               Relationship Resolver
                        │
                        ├── evidence event
                        ↓
                  Query Planner
                        │
                        ├── evidence event
                        ↓
                  SQL Generator
                        │
                        ├── query artifact
                        ↓
                    Executor
                        │
                        ├── result artifact
                        ↓
                Answer Generator
                        │
                        └── answer artifact

All artifacts
      ↓
Answer Lineage Store
      ↓
Evidence API / Audit / Debugging / UI

The architecture is model- and database-agnostic.

The requirement is simply to preserve the decisions that matter.

What to Measure

Once lineage exists, useful operational metrics become possible:

% of answers with complete lineage
% using certified metrics
% using trusted relationship paths
% with validated query plans
Average time to diagnose incorrect answers
Answers impacted by deprecated semantic objects

These describe production reliability more meaningfully than token count alone.

Final Thoughts

A production data agent should not produce an answer and then forget how it got there.

The answer should remain linked to:

Intent
Semantics
Data
Relationships
Query Plan
SQL
Result
Governance

That is Answer Lineage.

It turns:

"Revenue was €18.6M."

from an isolated generated statement into a verifiable analytical artifact.

Don't just log the answer. Preserve the evidence chain that produced it.

Production AI analytics needs more than observability of models and SQL.

It needs observability of business meaning.

Why RBAC Alone Isn't Enough for Enterprise Data Agents

Arisyn — Thu, 10 Sep 2026 15:22:00 +0000

A user can be blocked from a sensitive column and still receive sensitive information derived from data they are allowed to access.

That changes the authorization problem for enterprise data agents.

Traditional access control asks:

Can this user read this database object?

An AI analytics system also needs to ask:

Is this user allowed to receive what the system can infer from those objects?

Consider a simple example.

A user cannot access:

employee.salary

But the same user can access:

department.total_cost
department.employee_count

A capable data agent can derive:

estimated_average_salary
=
department.total_cost
/
department.employee_count

No forbidden salary column was queried.

The database permission model may have worked perfectly.

The answer may still disclose information the policy intended to protect.

This is why:

Table access ≠ Answer access.

RBAC Still Matters

This is not an argument against role-based access control.

RBAC remains a critical foundation.

A typical model might define:

Role: Sales Manager

ALLOW:
  customer
  sales_order
  product
  regional_revenue

DENY:
  employee.salary
  payroll
  compensation_detail

At the database layer, those controls should continue to be enforced.

The problem is that an AI agent introduces several stages above the database:

Natural Language
      ↓
Intent Resolution
      ↓
Semantic Resolution
      ↓
Context Retrieval
      ↓
Relationship Planning
      ↓
SQL Generation
      ↓
Execution
      ↓
Answer Generation

Authorization therefore has more surfaces than a traditional application issuing predefined SQL.

The Inference Gap

Let's formalize the salary example.

Suppose policy says:

{
  "resource": "employee.salary",
  "action": "read",
  "effect": "deny"
}

But:

{
  "resource": "department.total_cost",
  "action": "read",
  "effect": "allow"
}

and:

{
  "resource": "department.employee_count",
  "action": "read",
  "effect": "allow"
}

The agent creates:

f(total_cost, employee_count)
→ estimated_average_salary

Every input is authorized.

The derived concept may not be.

Call this the inference gap:

Authorized Inputs
      ↓
Reasoning / Aggregation
      ↓
Restricted Information

Traditional object-level authorization may not express that boundary.

Add Semantic Authorization

Users ask questions in business concepts.

So policy should increasingly understand business concepts too.

Instead of governing only:

employee.salary

define a semantic concept:

{
  "concept": "employee_compensation",
  "direct_access": "deny",
  "derived_access": "deny"
}

Now a request such as:

What is the average salary of the engineering team?

can be resolved first:

Intent
→ Employee Compensation

Then evaluated:

Employee Compensation
→ DENY

before SQL generation begins.

This is semantic authorization.

It lets policy operate at the same abstraction level as the user's question.

Authorization Should Start Before SQL Generation

A common architecture is:

Question
   ↓
Retrieve Schema
   ↓
Generate SQL
   ↓
Database Permission Check
   ↓
Execute

The problem is that the model may already have received context it should not use.

A stronger pipeline is:

Question
      ↓
Identity
      ↓
Intent Resolution
      ↓
Semantic Policy
      ↓
Authorized Context
      ↓
Authorized Relationship Graph
      ↓
Query Planning
      ↓
SQL Generation
      ↓
Database Enforcement
      ↓
Answer Policy

The important change is:

Authorization constrains reasoning context before it constrains execution.

Build an Authorized Context Resolver

Imagine the enterprise semantic layer contains:

Revenue
Gross Margin
Customer Risk
Payroll Cost
Employee Compensation
Product Profitability

A generic context retriever might return all concepts semantically related to the question.

That is risky.

Instead:

Candidate Context
      ↓
Identity + Policy
      ↓
Authorized Context
      ↓
LLM

Pseudocode:

def resolve_authorized_context(question, user):
    intent = resolve_intent(question)

    candidates = retrieve_semantic_context(intent)

    allowed = [
        item for item in candidates
        if policy.can_use(user, item)
    ]

    return allowed

The real implementation will need stronger policy semantics, but the architectural boundary matters.

Do not give the model unauthorized context and hope the final SQL check fixes everything.

Relationships Need Authorization Too

Suppose relationship discovery finds:

Employee
   ↓
Department
   ↓
Cost Center
   ↓
Financial Cost

The path is structurally valid.

But a Sales user may not be allowed to traverse it.

So distinguish:

Trusted Relationship

from:

Authorized Relationship

A relationship object could carry policy metadata:

{
  "source": "department",
  "target": "cost_center",
  "status": "trusted",

  "policy": {
    "allowed_roles": [
      "finance",
      "hr"
    ]
  }
}

Then query planning uses a user-specific graph:

def authorized_graph(graph, user):
    return graph.filter(
        lambda edge: policy.can_traverse(user, edge)
    )

This gives us another useful rule:

Valid relationship ≠ Authorized relationship.

Query Planning Should Operate on the Authorized Graph

Assume the full relationship graph contains:

Customer ─ Order ─ Payment
Employee ─ Department ─ Cost Center
Supplier ─ Contract ─ Pricing

For a Sales user:

Customer ─ Order ─ Payment

may be available.

But:

Employee ─ Department ─ Cost Center

may be removed from the planning graph.

The SQL generator never sees that path.

That is safer than generating the query first and rejecting it later.

Direct Access and Derived Access Are Different Policies

Some concepts need two policy dimensions.

Example:

{
  "concept": "customer_credit_risk",

  "direct_access": {
    "roles": ["risk", "finance"]
  },

  "derived_access": {
    "roles": ["risk", "finance"]
  }
}

Why distinguish them?

Because an organization might allow:

Department Cost

but restrict:

Individual Compensation

or permit individual operational metrics while restricting a derived risk score.

The derived concept may have different sensitivity from its inputs.

Answer-Level Policy Is the Final Boundary

Even with pre-query authorization, a final result check is useful.

The pipeline may produce:

SQL Valid               ✓
Database Access         ✓
Relationship Valid      ✓
Execution               ✓
Answer Policy           ✕

The system should not return the result.

Conceptually:

result = execute(sql)

answer_concepts = classify_result_semantics(
    question=question,
    plan=query_plan,
    result=result
)

for concept in answer_concepts:
    if not policy.can_receive(user, concept):
        raise PolicyDenied(concept)

This is not simply SQL validation.

It is answer validation.

Why Result Classification Is Hard

A result rarely arrives labeled:

"This is sensitive compensation information."

The system needs evidence from:

Original Intent
Resolved Semantic Concepts
Selected Metrics
Query Plan
Aggregations
Relationship Path
Output Columns

That means answer governance should not be implemented as a disconnected moderation step.

It should preserve semantic provenance throughout query execution.

Preserve a Semantic Query Plan

Instead of storing only SQL:

SELECT ...

store a structured plan:

{
  "intent": "average employee compensation",

  "concepts": [
    "employee_compensation"
  ],

  "metrics": [
    "department_total_cost",
    "employee_count"
  ],

  "derived_metric": {
    "name": "estimated_average_salary",
    "expression": "department_total_cost / employee_count"
  },

  "relationships": [
    "employee -> department"
  ]
}

Now authorization has something meaningful to evaluate.

This is another reason production Text-to-SQL should not be treated as:

Question → SQL

The intermediate query plan matters.

Add Policy to the Query Plan

A policy-aware plan might look like:

{
  "concept": "employee_compensation",

  "authorization": {
    "semantic_access": false,
    "data_access": true,
    "relationship_access": true,
    "answer_access": false
  },

  "decision": "deny"
}

The system can stop before execution.

For a different user:

{
  "role": "HR Partner",
  "decision": "allow"
}

The same natural-language question can therefore produce different authorized query plans.

Policy-Aware Clarification

Authorization can also affect clarification.

Suppose a user asks:

Show employee cost.

The system resolves two candidates:

Department Operating Cost
Employee Compensation

The user is authorized for the first but not the second.

A naive clarification UI might reveal both options.

That itself may leak sensitive semantic structure.

Instead, candidate generation should be policy-filtered:

Candidate Concepts
      ↓
Policy Filter
      ↓
Allowed Clarification Options

Authorization therefore affects not only execution but also what the system is allowed to discuss.

Explain Denials in Business Terms

A natural-language interface should not return:

SQLSTATE 42501
permission denied

when the real issue is semantic.

A better response might be:

This question would reveal restricted employee compensation information. You can query department-level operating cost, but not derived salary information.

This improves both security and user experience.

The system can explain:

What category is restricted
What level is allowed
What alternative question is permitted

without exposing sensitive details.

Don't Try to Solve Every Possible Inference

There is an important practical limit.

If two harmless numbers can theoretically be combined into sensitive information, trying to enumerate every possible derivation can become impossible.

So focus governance on high-impact semantic concepts.

Examples:

Compensation
Protected Personal Information
Credit Risk
Confidential Pricing
Sensitive Forecasts
Health Information

Then model known derivation patterns and business policies around those concepts.

The goal is not mathematical prevention of all inference.

It is business-risk-aware governance.

A Practical Policy Model

One possible abstraction:

concept: employee_compensation

sensitivity: restricted

direct_access:
  allow:
    - hr
    - executive

derived_access:
  allow:
    - hr
    - executive

related_metrics:
  - salary
  - bonus
  - estimated_average_salary

restricted_derivations:
  - department_cost / employee_count

Another:

relationship:
  source: employee
  target: cost_center

trusted: true

traverse:
  allow:
    - finance
    - hr

This makes policy part of semantic and relationship metadata rather than an afterthought.

Audit the Reasoning Path

When a query is allowed or denied, log why.

Example:

{
  "user": "sales_manager",
  "question": "What is the average salary in engineering?",

  "resolved_intent": "employee_compensation",

  "policy_decision": "deny",

  "reason": "derived_access_not_allowed"
}

For allowed queries, record:

Semantic concepts used
Relationship path
Metrics selected
Policy decisions
Generated SQL

This creates an audit trail that is much more useful than logging SQL alone.

Test Authorization With Adversarial Questions

Enterprise data-agent security testing should include inference cases.

For example:

Direct request

Show individual employee salaries.

Expected:

DENY

Derived request

Divide engineering payroll cost by headcount.

Expected:

DENY

Allowed aggregate

Show total engineering operating cost.

Expected:

ALLOW

Relationship traversal

Join employee records with cost-center financials.

Expected:

ROLE DEPENDENT

These tests reveal whether the system governs meaning or only columns.

What to Measure

Useful authorization metrics could include:

Direct Sensitive Query Block Rate
Derived Sensitive Query Block Rate
False Denial Rate
Unauthorized Relationship Block Rate
Policy Explanation Accuracy

A secure system that denies every complex query is not useful.

The goal is:

Maximum useful access
within authorized semantic boundaries

A Reference Architecture

Putting everything together:

                 Natural Language
                        ↓
                     Identity
                        ↓
                Intent Resolution
                        ↓
                Semantic Policy
                        ↓
               Authorized Context
                        ↓
          Authorized Relationship Graph
                        ↓
                 Query Planning
                        ↓
                Policy-Aware Plan
                        ↓
                 SQL Generation
                        ↓
              Database Enforcement
                        ↓
                    Execution
                        ↓
              Answer Policy Check
                        ↓
             Return / Explain / Deny

RBAC remains underneath this architecture.

The new layers do not replace database security.

They extend governance into the reasoning process.

Final Thoughts

Enterprise data agents make databases easier to use because users no longer need to know schemas or SQL.

That abstraction is powerful.

It also means users can ask for information without knowing which fields, tables, joins, or calculations the agent will use.

So authorization has to follow the same abstraction upward.

From:

Who can access this table?

to:

Who can use this business concept?

and finally:

Who can receive this derived answer?

That is why RBAC alone is not the whole solution for AI-powered analytics.

Keep RBAC.

Keep row- and column-level controls.

But add governance around:

Intent
Semantics
Relationships
Derivations
Answers

Because:

Table access ≠ Answer access.

And:

Valid relationship ≠ Authorized relationship.

A production data agent should not only know how to find an answer.

It should know whether it is allowed to reveal it.

The Semantic Cold Start Problem in Enterprise Data Agents

Arisyn — Mon, 07 Sep 2026 15:50:00 +0000

Connecting an AI agent to a database is easy. Bootstrapping enough enterprise context for it to use that database correctly is the real engineering problem.

A new data agent can inspect a schema almost immediately:

tables
columns
data types
primary keys
sample values

Then a user asks:

What was revenue by product code last quarter?

The schema alone does not tell the agent:

What does "Revenue" mean here?
Which "product code" does the business use?
Which fields are authoritative?
How should the required tables be connected?
Which relationship candidates are trusted?

This gap between data access and business understanding is what I call the semantic cold start problem.

For production data agents, reducing that cold-start cost may matter as much as improving the underlying LLM.

1. Model the Problem as Knowledge Bootstrapping

A useful bootstrap pipeline looks like:

Connect Data
     ↓
Schema Discovery
     ↓
Relationship Discovery
     ↓
Semantic Candidate Generation
     ↓
Existing Knowledge Import
     ↓
Human Confirmation
     ↓
Runtime Clarification
     ↓
Knowledge Persistence
     ↓
Reuse

The important point is that no single step is expected to magically understand the enterprise.

Instead, the system progressively converts different forms of evidence into reusable context.

2. Start With a Machine-Readable Inventory

The first layer is deterministic.

For each data source, collect technical metadata such as:

{
  "table": "finance_revenue",
  "columns": [
    {
      "name": "recognized_amount",
      "type": "decimal"
    },
    {
      "name": "recognition_date",
      "type": "date"
    },
    {
      "name": "order_id",
      "type": "bigint"
    }
  ]
}

This makes the agent schema-aware.

But schema awareness is only Level 1.

It tells the system what exists, not what the business means.

3. Discover Relationship Candidates

The next problem is connectivity.

Enterprise databases often have incomplete foreign-key definitions, so relationship discovery may need to combine several signals:

Declared constraints
Column-name similarity
Compatible data types
Uniqueness
Value overlap
Value inclusion
Existing metadata

For example:

account.customer_id
customer.customer_id

may be a strong relationship candidate even if no FK constraint exists.

A candidate could be represented as:

{
  "source": "account.customer_id",
  "target": "customer.customer_id",
  "evidence": {
    "type_compatible": true,
    "name_similarity": 0.91,
    "inclusion": 0.98,
    "target_uniqueness": 0.99
  },
  "confidence": 0.96,
  "status": "candidate"
}

The important distinction is:

discovered relationship
≠
trusted relationship

Discovery reduces manual work.

Validation turns a candidate into enterprise knowledge.

4. Generate Semantic Candidates, Not Semantic Truth

Now consider the business term:

Product Code

The schema may contain:

product_master.material_id
inventory.item_code
sales_detail.sku_no
product_dim.prod_master_id

An LLM or semantic retrieval system can rank candidates:

{
  "term": "Product Code",
  "candidates": [
    {
      "field": "product_master.material_id",
      "score": 0.88
    },
    {
      "field": "inventory.item_code",
      "score": 0.84
    },
    {
      "field": "sales_detail.sku_no",
      "score": 0.79
    }
  ]
}

This is useful.

But it should not automatically become:

Product Code = material_id

A semantic candidate is evidence.

A business mapping is a governed decision.

That separation prevents model confidence from being mistaken for organizational truth.

5. Import What the Enterprise Already Knows

Cold start should not mean starting from zero.

Useful knowledge may already exist in:

Business glossaries
Metric definitions
Data catalogs
BI models
Dashboard logic
Documentation
Existing SQL

For example, Finance may already define:

metric:
  name: Revenue

definition:
  Recognized revenue for financial reporting

source:
  table: finance_revenue
  field: recognized_amount

time_field:
  recognition_date

If this definition already exists, the agent should reuse it instead of asking users to redefine Revenue.

A good bootstrap system therefore needs both:

Discovery

and:

Import

6. Use Humans for Confirmation, Not Exhaustive Modeling

Some mappings cannot be inferred safely.

Instead of asking a data team to document everything manually, let the system narrow the decision first.

Example:

"Product Code" has 3 likely mappings:

1. product_master.material_id
2. inventory.item_code
3. sales_detail.sku_no

A domain expert selects:

product_master.material_id

Persist the decision:

{
  "knowledge_type": "semantic_mapping",
  "term": "Product Code",
  "target": "product_master.material_id",
  "source": "human_confirmation",
  "status": "trusted",
  "confidence": 1.0
}

This changes the human role from:

Author everything

to:

Review high-value uncertainty

That is much more scalable.

7. Runtime Clarification Is Also a Learning Channel

Some ambiguity only appears when users ask real questions.

Suppose:

Show revenue by product code.

The semantic resolver returns:

{
  "term": "Product Code",
  "status": "ambiguous",
  "candidates": [
    "Master Product Code",
    "SKU Code",
    "Internal Product ID"
  ]
}

The agent asks:

Which product code do you mean?

The user chooses:

Master Product Code

A weak implementation uses the answer only for the current query.

A stronger implementation creates a knowledge event:

{
  "event": "semantic_confirmation",
  "user_phrase": "product code",
  "resolved_concept": "Master Product Code",
  "physical_field": "product_master.material_id",
  "source": "runtime_clarification"
}

Now the interaction improves future queries.

Every Correction Should Make the Next Question Cheaper

This is the core learning property.

Without persistence:

User A asks
   ↓
System clarifies
   ↓
Answer

User B asks same concept
   ↓
System clarifies again

With reusable knowledge:

User A asks
   ↓
System clarifies
   ↓
Knowledge stored
   ↓
User B asks
   ↓
Known mapping
   ↓
Answer

The second query requires less reasoning and less human effort.

That is what enterprise learning should look like.

8. Separate Runtime State From Persistent Knowledge

Not every user choice should become global enterprise truth.

Suppose a user says:

Use invoice amount for this analysis.

That may be a query-specific preference, not a new definition of Revenue.

So distinguish:

Session State
User Preference
Workspace Default
Governed Enterprise Knowledge

A useful persistence policy might look like:

def persist_resolution(resolution):
    if resolution.scope == "query":
        return save_session_state(resolution)

    if resolution.source == "user_selection":
        return propose_workspace_knowledge(resolution)

    if resolution.approved_by_domain_owner:
        return save_governed_knowledge(resolution)

The exact workflow varies, but the principle matters:

Learning requires scope and governance.

Otherwise one user's temporary choice can corrupt the shared semantic layer.

9. Track Provenance

Every knowledge object should answer:

Where did this come from?
Who confirmed it?
When was it created?
How confident are we?
Is it still active?

Example:

{
  "term": "Revenue",
  "target": "finance_revenue.recognized_amount",
  "source": "finance_metric_catalog",
  "owner": "finance",
  "status": "active",
  "confidence": 1.0,
  "version": 4
}

For discovered relationships:

{
  "relationship": "account.customer_id -> customer.customer_id",
  "source": "relationship_discovery",
  "confirmed_by": "data_team",
  "status": "trusted"
}

Without provenance, accumulated knowledge becomes difficult to trust.

10. Build a Context Resolver Above the Raw Knowledge

The query runtime should not dump every known object into the prompt.

Instead:

User Question
      ↓
Intent Extraction
      ↓
Context Resolver
      ↓
Relevant Business Terms
Relevant Metrics
Relevant Fields
Relevant Relationships
      ↓
Compact Query Context
      ↓
SQL Generation

For:

Revenue by product code last quarter

the context resolver might return:

{
  "metric": {
    "name": "Revenue",
    "field": "finance_revenue.recognized_amount"
  },

  "dimension": {
    "name": "Product Code",
    "field": "product_master.material_id"
  },

  "relationship_path": [
    "product_master -> sales_order",
    "sales_order -> finance_revenue"
  ],

  "time_range": "last_quarter"
}

The model reasons over a smaller, better-defined space.

11. Treat Knowledge Acquisition as an Event Stream

A useful architecture is to treat semantic learning as events:

schema_discovered
relationship_candidate_found
relationship_confirmed
metric_imported
semantic_mapping_confirmed
clarification_resolved
mapping_deprecated

For example:

{
  "event_type": "semantic_mapping_confirmed",
  "timestamp": "2026-09-07T10:30:00Z",
  "payload": {
    "term": "Product Code",
    "field": "product_master.material_id"
  }
}

This gives you:

Auditability
Versioning
Rollback
Analytics
Learning metrics

It also avoids hiding important enterprise knowledge inside opaque prompt history.

12. Measure Time to Intelligence

Connection latency is easy to measure:

Database connected in 4 minutes.

But a better enterprise metric is:

Time to Intelligence

Possible operational measurements include:

Time until first trusted answer

% of common business terms resolved

% of critical metrics governed

% of required relationships trusted

Clarification rate

Repeated clarification rate

Manual confirmations per 100 queries

A healthy system should improve over time.

For example:

Week 1
Semantic coverage: 42%
Repeated clarification: 31%

Month 1
Semantic coverage: 68%
Repeated clarification: 17%

Month 3
Semantic coverage: 84%
Repeated clarification: 6%

The exact targets depend on the enterprise.

The direction is what matters.

13. Measure Enterprise Learning Rate

Another useful concept is:

Enterprise Learning Rate

How quickly does real usage produce reusable trusted knowledge?

You could track:

New trusted mappings / week
New trusted relationships / week
Clarifications converted to reusable knowledge
Repeated ambiguity reduction
Human confirmation workload

The goal is not maximum knowledge creation.

It is reducing repeated uncertainty.

A useful signal might be:

Repeated Clarification Rate ↓

If users keep clarifying the same concepts month after month, the system is not learning effectively.

14. A Maturity Model

A data agent can be viewed as progressing through several levels.

Level 0 — Connected

Can access data

Level 1 — Schema-Aware

Understands tables and fields

Level 2 — Relationship-Aware

Understands how relevant data connects

Level 3 — Semantic-Aware

Understands business terms,
metrics, and dimensions

Level 4 — Business-Aware

Uses governed definitions,
trusted mappings, and business context

Level 5 — Learning

Turns corrections,
confirmations, and clarifications
into reusable knowledge

Most enterprise AI demos prove Levels 0 and 1.

Production value increasingly appears in Levels 3 through 5.

15. The Goal Is Not Fully Automatic Semantics

There is a tempting but unrealistic objective:

Connect database
↓
AI automatically understands everything

Some business meaning cannot be inferred from data.

If a company decides that Revenue means:

Recognized Revenue

rather than:

Invoice Amount

that is an organizational decision.

The system should not pretend otherwise.

A more realistic architecture is:

Machine discovers
      ↓
AI proposes
      ↓
Humans confirm high-impact meaning
      ↓
System remembers

The goal is not zero human input.

It is:

Minimum repetitive human work with preserved human authority over business meaning.

16. Keep Enterprise Knowledge Model-Agnostic

The underlying LLM will change.

Enterprise knowledge should survive those changes.

Avoid storing important semantics only inside:

Prompt templates
Few-shot examples
Conversation history
Model-specific instructions

Instead, keep durable knowledge as structured objects:

Business Terms
Metrics
Dimensions
Mappings
Relationships
Rules
Provenance

Then different models can consume the same enterprise context.

Models are replaceable.

Enterprise knowledge compounds.

17. A Practical Bootstrap Architecture
Putting the pieces together:

              Enterprise Data
                     ↓
             Metadata Discovery
                     ↓
          Relationship Discovery
                     ↓
      Existing Knowledge Import
                     ↓
       Semantic Candidate Layer
                     ↓
            Human Confirmation
                     ↓
          Trusted Knowledge Store
                     ↑
                     │
User Question → Runtime Resolver
                     │
                     ↓
              Clarification
                     │
                     ↓
              Knowledge Event
                     │
                     └──────────→ Trusted Knowledge Store

Trusted Context
      ↓
Query Planning
      ↓
SQL Generation
      ↓
Validation
      ↓
Execution

The system becomes better not because the model retrains after every question, but because the enterprise context around the model improves.

Final Thoughts

The semantic cold start problem is easy to underestimate.

A database connection gives an agent access to:

Data

Production analytics requires:

Data
+
Meaning
+
Relationships
+
Authority
+
Learning

So the engineering goal should not simply be:

Connect the database faster.

It should be:

Reduce the amount of work required for the system to become business-aware.

That means discovering what can be discovered, importing what already exists, asking humans only where judgment matters, and turning useful corrections into durable knowledge.

The most important property is simple:

Every correction should make the next question cheaper.

That is how a generic data agent starts becoming an enterprise data agent.

How I Would Benchmark a Text-to-SQL System for Production

Arisyn — Wed, 02 Sep 2026 14:30:00 +0000

A clean question against a clean schema is a demo. A production benchmark should deliberately test ambiguity, missing relationships, competing metrics, and SQL that executes successfully but answers the wrong business question.

Most Text-to-SQL evaluations start with questions like:

What was revenue last quarter?
Show sales by region.
List the top 10 customers.

These tests are useful for checking whether the basic pipeline works.

They are not enough to tell you whether the system is ready for enterprise production.

In production, the difficult cases are rarely caused by SQL syntax alone. They come from business language, incomplete metadata, ambiguous metrics, undocumented relationships, aggregation grain, and assumptions that look reasonable but are wrong.

So if I were benchmarking a Text-to-SQL system, I would build the benchmark around failure paths, not just happy paths.

## Why Execution Accuracy Is Not Enough

A typical evaluation might measure:

Question
   ↓
Generated SQL
   ↓
Expected SQL

or:

Generated SQL
   ↓
Execute
   ↓
Compare Result

Execution-based evaluation is better than exact SQL matching because multiple SQL statements can produce the same correct answer.

But enterprise reliability requires additional questions:

Did the system understand the business term correctly?

Did it choose the authoritative metric?

Did it use the right relationship path?

Did it preserve the correct aggregation grain?

Did it recognize ambiguity?

Did it know when not to query?

A query can execute successfully and still fail every one of these tests.

# Build a Failure-Path Benchmark

I would create at least seven test categories.

A. Ambiguous Intent
B. Business Term Mapping
C. Competing Metrics
D. Missing Relationships
E. Executable-but-Wrong SQL
F. Insufficient Intent
G. Environment Change

Each category tests a different part of the system.

## Test A: Ambiguous Intent

Question:

Show me our best customers.

Possible interpretations:

Revenue
Profit
Growth
Retention
Lifetime Value

The benchmark should not define best in the question.

The correct behavior depends on enterprise context.

If a governed definition exists:

term: best_customer
metric: customer_lifetime_value
status: active

the system can resolve it.

If multiple valid definitions remain, the expected behavior should be:

CLARIFY

not:

GENERATE_SQL

### Example benchmark case

{
  "id": "ambiguity_001",
  "question": "Show me our best customers.",
  "valid_metrics": [
    "revenue",
    "profit",
    "growth",
    "retention"
  ],
  "governed_default": null,
  "expected_action": "clarify"
}

### What to score

Ambiguity detected?
Correct candidates identified?
Clarification triggered?
Clarification options useful?

This gives you a Clarification Accuracy metric rather than only SQL accuracy.

## Test B: Business Term Mapping

Question:

What was revenue by product code this quarter?

Now make sure the schema does not contain a field called:

product_code

Instead:

product_master.material_id
sales_detail.sku_no
inventory.item_code
product_dim.prod_master_id

The benchmark should contain a governed semantic mapping:

business_term:
  name: Product Code

maps_to:
  table: product_master
  field: material_id

### Failure mode

A schema-similarity system may select:

inventory.item_code

because the name looks closest.

But the benchmark expects:

product_master.material_id

because that is the governed enterprise definition.

### What to score

Business term resolved correctly?
Correct physical field selected?
Governed mapping preferred over lexical similarity?

Call this:

# **Semantic Mapping Accuracy**

## Test C: Competing Metrics

Create several plausible financial fields:

sales_order.total_amount
invoice.invoice_amount
finance_revenue.recognized_amount
payment.received_amount

Question:

What was revenue last quarter?

All four fields are relevant to money.

Only one represents the governed Revenue metric.

Benchmark definition:

metric:
  name: Revenue

source:
  table: finance_revenue
  field: recognized_amount

time_field:
  finance_revenue.recognition_date

### What to inspect

Do not only inspect the final number.

Inspect the query plan.

Expected:

Metric
→ Revenue

Field
→ finance_revenue.recognized_amount

Time
→ finance_revenue.recognition_date

### What to score

Metric resolution accuracy
Source selection accuracy
Time-field accuracy

This catches systems that accidentally return the right number from the wrong business definition.

## Test D: Missing Relationships

Production databases often lack perfect foreign keys.

Build:

customer
account
sales_order
finance_revenue

Expected relationship path:

Customer
   ↓
Account
   ↓
Order
   ↓
Revenue

Then deliberately remove some database constraints.

The benchmark should still contain enough data evidence for a relationship system to infer or retrieve valid relationships.

For example:

account.customer_id
sales_order.account_id
finance_revenue.order_id

### Useful relationship evidence

A relationship engine might consider:

Column names
Data types
Uniqueness
Value overlap
Inclusion
Known metadata
Validated relationships

One useful inclusion signal is:

Inclusion(A → B)
=
|distinct(A) ∩ distinct(B)|
---------------------------
|distinct(A)|

where A is a candidate foreign-key-like column and B is a candidate referenced column.

### What to score

Correct tables selected?
Correct relationship path found?
Unsupported direct joins avoided?

Call this:

# **Relationship Path Accuracy**

## Test E: Executable-but-Wrong SQL

This is the test I would care about most.

Suppose the schema allows:

Customer → Order

and:

Customer → Account → Order

Both can generate executable SQL.

But for consolidated customers, only:

Customer → Account → Order

preserves the correct business grain.

Create two candidate queries.

### Query A

SELECT
    c.customer_name,
    SUM(o.amount)
FROM customer c
JOIN sales_order o
    ON c.customer_id = o.customer_id
GROUP BY c.customer_name;

### Query B

SELECT
    c.customer_name,
    SUM(o.amount)
FROM customer c
JOIN account a
    ON c.customer_id = a.customer_id
JOIN sales_order o
    ON a.account_id = o.account_id
GROUP BY c.customer_name;

Both may execute.

Only Query B is business-valid for the benchmark scenario.

### What to score

SQL executable?
Business relationship valid?
Aggregation grain valid?
Result semantically correct?

This creates an important distinction:

Execution Accuracy
≠
Business Accuracy

## Test F: Insufficient Intent

Question:

Show our best-performing products recently.

The benchmark intentionally leaves unresolved:

best-performing
recently

Possible metric candidates:

Revenue
Profit
Units Sold
Growth

Possible time candidates:

7 Days
30 Days
Current Month
Current Quarter

If no governed defaults exist, the expected result is not SQL.

It is:

{
  "action": "clarify",
  "unresolved": [
    "metric",
    "time_range"
  ]
}

### What to score

Did the system detect missing intent?
Did it avoid premature SQL generation?
Did it ask the minimum useful clarification?

This measures:

# **Safe Failure Accuracy**

A production system should not be rewarded for answering every question.

## Test G: Environment Change

Static benchmarks miss one of the largest production costs: maintenance.

Start with a working environment.

Then introduce changes:

Add a table
Rename a field
Add a new metric
Deprecate an old metric
Add a new relationship
Change a semantic mapping

For example:

Before:
Product Code → product_master.material_id

After:
Product Code → product_dim.product_code

Then rerun the benchmark.

Measure:

How many cases break?

How much manual configuration is required?

How quickly does the system recover?

Which knowledge objects need updating?

This produces a metric that benchmark leaderboards rarely show:

# **Semantic Maintenance Cost**

# Score More Than SQL

A useful benchmark report could look like:

Metric	Score
Semantic Mapping Accuracy	94%
Metric Resolution Accuracy	96%
Relationship Path Accuracy	91%
Clarification Accuracy	93%
SQL Execution Accuracy	98%
Business Answer Accuracy	89%
Safe Failure Accuracy	95%
Maintenance Effort	3.2 min/change

The exact metrics depend on the product.

The important point is that:

SQL Accuracy

should be one row, not the entire benchmark.

# Use a Multi-Stage Evaluation Harness

Instead of treating Text-to-SQL as one black box:

Question
→ SQL

capture intermediate artifacts.

For example:

{
  "question": "Revenue by customer last quarter",

  "intent": {
    "metric": "recognized_revenue",
    "dimension": "customer",
    "time": "last_quarter"
  },

  "semantic_objects": [
    "metric.revenue",
    "dimension.customer"
  ],

  "tables": [
    "customer",
    "account",
    "sales_order",
    "finance_revenue"
  ],

  "relationship_path": [
    "customer.customer_id -> account.customer_id",
    "account.account_id -> sales_order.account_id",
    "sales_order.order_id -> finance_revenue.order_id"
  ],

  "generated_sql": "...",

  "action": "execute"
}

Now failures become diagnosable.

If the final answer is wrong, you can determine whether the failure came from:

Intent
Semantic Mapping
Table Selection
Relationship Resolution
Metric Selection
SQL Generation
Execution

That is much more useful than:

FAILED

# Build Adversarial Test Pairs

A strong benchmark should include pairs of questions that look similar but require different interpretations.

Example:

Q1

What was our revenue last quarter?

Expected metric:

recognized_revenue

Q2

What was our invoiced amount last quarter?

Expected metric:

invoice_amount

Another pair:

Q1

Sales by customer region.

Expected dimension:

customer_region

Q2

Sales by billing region.

Expected dimension:

billing_region

These pairs test whether the system is actually resolving semantics or simply reusing the most common mapping.

# Include Negative Tests

Traditional benchmarks mostly contain questions that should be answerable.

Production benchmarks should include questions that should not.

For example:

Question:
"Show profitability by happiness score."

No governed metric:
happiness_score

Expected:
Cannot resolve metric / clarification required

Or:

Question:
"Join employee salary with public customer data."

Expected:
Rejected by policy

The exact safety rules depend on the deployment.

The principle is:

A system should be tested on its ability to refuse invalid query plans, not only generate valid ones.

# Test With Messy Schemas

Do not benchmark only:

customer
orders
products

Use schemas that look more like enterprise reality:

t_cust_m
cust_master_old
acct_rel
f_ord_h
ord_detail_v2
fin_rev_rec
inv_hdr
inv_line

Add:

Deprecated tables
Duplicate concepts
Missing descriptions
Inconsistent naming
Cross-system IDs
Partial foreign keys

If the product claims to work with enterprise data, the benchmark should look like enterprise data.

# Measure Human Setup Cost

Before the first benchmark run, track setup effort.

For example:

Schema descriptions:       4 hours
Metric definitions:        3 hours
Relationship configuration: 6 hours
Example SQL:                5 hours
Prompt tuning:              2 hours

Total:

20 hours

Another system may achieve slightly lower raw accuracy but require:

4 hours

of setup.

That difference matters.

So benchmark:

Accuracy

and:

Cost to achieve that accuracy

together.

# A More Useful Production Score

I would think about production value approximately as:

                  Accuracy × Trust × Coverage
Value  ≈  ─────────────────────────────────────
                Setup + Maintenance Cost

Again, this is not a universal mathematical formula.

It forces the benchmark to consider what enterprises actually pay for.

A system that scores 97% on 50 carefully prepared questions may be less valuable than one scoring 93% across thousands of messy tables with far less manual configuration.

# What I Would Put in a POC

If I were designing a vendor POC, I would include:

10 clear questions
10 ambiguous questions
10 semantic-alias questions
10 competing-metric questions
10 missing-relationship questions
10 multi-path join questions
10 executable-but-wrong traps
10 questions that require clarification
10 unanswerable questions
10 post-schema-change regressions

That gives:

100 cases

with intentionally different failure modes.

Then evaluate both:

Can it answer correctly?

and:

Can it recognize when it should not answer yet?

# Final Thoughts

A production Text-to-SQL benchmark should not primarily test whether an LLM knows SQL.

Modern models are already good at SQL generation under clean conditions.

The harder questions are:

Did it understand the business term?

Did it select the right metric?

Did it find the right data?

Did it use the right relationship?

Did it preserve the right grain?

Did it detect ambiguity?

Did it fail safely?

How much human work was required?

Those are system-level questions.

So when evaluating a Text-to-SQL platform:

Don't benchmark the easiest route from question to SQL.

Deliberately break the assumptions.

Remove the foreign key.

Add three plausible revenue fields.

Use business terminology that does not match the schema.

Create two executable join paths where only one is correct.

Ask a question that should trigger clarification.

Then change the data model and run everything again.

That is where you start learning whether you have a good demo—or a production system.

Building a Trustworthy Semantic Layer for Natural Language Analytics

Arisyn — Mon, 31 Aug 2026 15:52:00 +0000

Natural language analytics looks simple from the outside:

User asks a question.

The system generates SQL.

The database returns an answer.

In real enterprise environments, this breaks quickly.

The reason is not just SQL generation. The reason is semantic ambiguity.

When a user asks:

Show sales by product code for last month.

The system needs to resolve several questions before it can safely generate a query:

What does "sales" mean?
Is it gross sales, net sales, recognized revenue, bookings, or paid order amount?
What does "product code" mean?
Is it internal product code, SKU, marketplace product code, barcode, or ERP material number?
Which date field should represent "last month"?
Which tables contain the right facts and dimensions?
Which join path is trusted?
Is the current user allowed to access the required data?

This is why a semantic layer is becoming a critical part of natural language analytics.

But not all semantic layers are designed for the same job.

Databricks is building semantic capabilities natively inside its lakehouse platform. Semora is building an independent semantic layer for complex, cross-platform enterprise data. From an engineering perspective, the interesting question is not "Which product has a better chat interface?"

The better question is:

What system architecture makes natural language analytics both fluent and accurate?

## The Core Problem: Language Models Should Not Guess the Data Model

A large language model can usually understand that these phrases are related:

product code
product ID
item number
SKU
material number

But in a real company, those phrases may not be interchangeable.

One table may contain:

product_code
product_id
sku_id
barcode
platform_item_id
material_number

If the system maps "product ID" to product_code just because the words sound close, the answer may be fluent but wrong.

On the other hand, if the system asks the user to clarify every similar term, the experience becomes painful.

The engineering challenge is to build a runtime that can decide:

When can we answer directly?
When should we show the assumption?
When should we ask a lightweight clarification?
When must we require explicit confirmation?
How do we save the user's answer so we do not ask again?

This is where Semora can differentiate.

## Databricks: Platform-Native Semantic Context

Databricks has been expanding its semantic layer and natural language analytics stack through Unity Catalog business semantics, metric views, Genie, and Genie Agents.

The Databricks approach is powerful because semantic assets live close to the governed data platform.

A Databricks-style flow can look like this:

User question
  -> Genie Agent scope
  -> Unity Catalog metadata
  -> Metric views
  -> Table and column descriptions
  -> Trusted queries and examples
  -> SQL generation
  -> Query inspection
  -> Governed answer

This works especially well when:

The company already uses Databricks as the main analytical platform.
Tables, permissions, lineage, and metrics are managed in Unity Catalog.
Business domains are curated into focused Genie Agents.
Data teams maintain metric views, examples, and trusted assets.

The key engineering lesson is that Databricks does not treat the language model as the only source of intelligence. It surrounds the model with governed context.

That context reduces the search space.

If a user is asking inside a Sales Genie Agent, "revenue" can be interpreted using sales-domain semantics. If they ask inside a Finance context, the default meaning may be different.

This domain scoping is one of the most important reasons a natural language analytics experience feels smooth.

## Semora: Independent Semantic Runtime

Semora has a different opportunity.

Many enterprises do not have one clean analytical platform. They have multiple warehouses, operational databases, business intelligence tools, spreadsheets, and legacy systems. Field names are inconsistent. Foreign keys are missing. Business terms vary by department.

In this environment, an independent semantic layer is valuable because it can sit above the data platforms instead of being tied to only one.

A Semora-style flow can look like this:

User question
  -> Intent classification
  -> Role and permission scope
  -> Business domain routing
  -> Synonym and semantic retrieval
  -> Business concept mapping
  -> Certified field selection
  -> Relationship discovery
  -> Trusted question template matching
  -> Query generation
  -> Query validation
  -> Answer with lineage
  -> Clarification memory update

The important word here is runtime.

It is not enough to have pages for synonyms, business semantics, data definitions, question templates, and knowledge bases. Those assets must be actively retrieved, ranked, and applied while the user is asking a question.

Otherwise, the product can have many semantic assets and still feel rough.

## Runtime Component 1: Intent Classification

The first step should not be SQL generation.

The first step should be intent classification.

For example, the user may ask:

What questions can you answer for me?

This is not a data query. It is a capability discovery question.

A poor system routes it into the query pipeline, fails to find a metric, and asks for clarification.

A better system recognizes the intent and answers based on the user's role, permissions, business domains, and available question templates.

Useful intent classes include:

DATA_QUERY
CAPABILITY_DISCOVERY
METRIC_EXPLANATION
FIELD_EXPLANATION
DOMAIN_EXPLORATION
TEMPLATE_RECOMMENDATION
PERMISSION_SCOPE_EXPLANATION
PRODUCT_HELP

For CAPABILITY_DISCOVERY, the response should be generated from available semantic assets:

Accessible business domains
Certified metrics
Question templates
Frequently used questions
Available dimensions
Permission boundaries

Example response:

Based on your current access, I can answer questions about sales, products, inventory, and channels. You can ask about sales trends, product rankings, inventory shortages, channel comparisons, and customer repeat purchase behavior.

This small routing layer can dramatically improve first-time user experience.

## Runtime Component 2: Business Domain Routing

Even if Semora keeps one universal question box, the backend should not treat all data as one global space.

The system should route each question into one or more business domains.

For example:

"sales by product code"
  -> Sales domain
  -> Product domain

"inventory risk by SKU"
  -> Inventory domain
  -> Product domain

"recognized revenue by customer"
  -> Finance domain
  -> Customer domain

Domain routing should use:

User role
Permission scope
Question text
Matched metrics
Matched templates
Historical user behavior
Business glossary terms

The user can still see one clean interface. Internally, the system should load a scoped semantic context.

This reduces ambiguity before the language model starts planning the query.

## Runtime Component 3: Multi-Path Semantic Retrieval

A common failure mode is exact or weak lexical matching.

If the physical field is called product_code, and the user says "product number," the system may fail unless that synonym was manually configured.

This does not scale.

Semora should use multi-path retrieval:

User phrase
  -> Exact match
  -> Token match
  -> Synonym match
  -> Embedding similarity
  -> Field description match
  -> Sample value match
  -> Historical clarification match
  -> Question template match
  -> Domain default match

For the phrase "product code," candidates might include:

Business concept: Product Code
Fields:
  - dim_product.product_code
  - order_items.product_code
  - inventory_sku.sku_code
  - marketplace_items.platform_item_id
  - product_barcodes.barcode

The retrieval layer should not immediately decide. It should return candidates with evidence.

Example evidence:

{
  "candidate": "dim_product.product_code",
  "business_concept": "Product Code",
  "evidence": {
    "synonym_match": ["product code", "item code"],
    "domain": "Product",
    "field_description": "Internal product code used across sales and inventory",
    "historical_confirmations": 17,
    "certification_status": "certified"
  },
  "confidence": 0.93
}

This gives the language model structured context instead of asking it to guess from raw schema names.

## Runtime Component 4: Confidence-Based Clarification

Clarification should be a decision, not a default.

A useful policy could be:

confidence >= 0.85
  -> answer directly and show the assumption

0.60 <= confidence < 0.85
  -> answer with a lightweight change option, or ask a short clarification

confidence < 0.60
  -> require clarification

high-risk metric or high-risk join
  -> require confirmation even if confidence is moderate

For a high-confidence case:

I interpreted "product code" as the certified Product Code concept and used the internal product code field.

For an ambiguous case:

Which product identifier do you mean?

Internal product code

SKU number

Marketplace product code

Barcode

ERP material number

This is better than showing raw column names to a business user.

Technical users can still inspect the underlying fields, joins, and SQL.

## Runtime Component 5: Trusted Question Templates

Question templates should not be simple prompt examples.

They should be trusted analytical paths.

For example, a template for:

Top-selling products in a time period

Should include:

template_id: top_selling_products
intent: ranking
metric:
  name: Sales Quantity
  certification: certified
dimensions:
  - Product Code
  - Product Name
time:
  default_field: Paid Time
filters:
  - Paid Orders Only
relationships:
  - order_items -> dim_product by product_code
parameters:
  - time_range
  - channel
  - category
validation:
  - sales_quantity >= 0
  - product_code is not null
owner: Sales Analytics Team
status: certified

When the user asks:

Show last month's product code sales ranking.

The system should match the template first, extract parameters, and reuse the certified query path.

The language model should not rebuild the business logic from scratch.

## Runtime Component 6: Relationship Discovery and Join Safety

Natural language analytics systems often fail silently when joins are wrong.

A query can run successfully and still be incorrect because:

It joined at the wrong grain.
It created duplicate rows.
It used a weak relationship.
It joined through the wrong entity.
It mixed internal identifiers with external identifiers.

For Semora, relationship discovery should be a major differentiator.

A relationship engine should evaluate:

Field name similarity
Data type compatibility
Value overlap
Inclusion ratio
Cardinality
Null rate
Uniqueness
Historical usage
Human approval
Domain relevance

The output should be explainable:

{
  "join_path": [
    "order_items.product_code",
    "dim_product.product_code"
  ],
  "confidence": 0.91,
  "evidence": {
    "value_overlap": "high",
    "right_side_uniqueness": "strong",
    "approved_by": "data_owner",
    "fanout_risk": "low"
  }
}

This is where independent semantic layers can create real value. Platform-native semantic layers work well when relationships are already modeled. Independent semantic layers can help discover and govern relationships where metadata is incomplete.

## Runtime Component 7: Query Validation

SQL generation is not the end of the pipeline.

The system should validate the generated query before presenting the answer.

Validation checks may include:

Does the user have permission to access every field?
Is the selected metric certified?
Is the date field appropriate for the metric?
Are filter values valid?
Is the join path approved?
Is there fanout risk?
Is the aggregation grain correct?
Is the result within a reasonable historical range?
Did the query return no rows because of a value mapping issue?

This validation layer is one of the most important differences between a demo and a production-ready product.

## Runtime Component 8: Clarification Memory

Every clarification should become a reusable asset.

If a user confirms:

By "product code," I mean the internal product code.

The system should save:

user_phrase: product code
standard_concept: Product Code
field_mapping: dim_product.product_code
business_domain:
  - Product
  - Sales
confirmed_by: user_or_owner
scope: user | team | organization
status: draft | reviewed | certified
usage_count: 1

Over time, repeated confirmations should create candidate semantic rules. Data owners can review and promote them.

This makes the system smoother with use.

The goal is not to ask better clarification questions forever. The goal is to ask fewer clarification questions over time because the semantic layer is learning.

## A Practical Architecture

Here is a practical architecture for a natural language analytics runtime:

                         +----------------------+
                         |   User Question      |
                         +----------+-----------+
                                    |
                                    v
                         +----------------------+
                         | Intent Classifier    |
                         +----------+-----------+
                                    |
              +---------------------+---------------------+
              |                                           |
              v                                           v
   +------------------------+                 +------------------------+
   | Non-query Intents      |                 | Data Query Intent      |
   | Help, scope, examples  |                 | Metrics, filters, SQL  |
   +------------------------+                 +-----------+------------+
                                                          |
                                                          v
                                           +---------------------------+
                                           | Domain Router             |
                                           +------------+--------------+
                                                        |
                                                        v
                                           +---------------------------+
                                           | Semantic Retrieval        |
                                           +------------+--------------+
                                                        |
                                                        v
                                           +---------------------------+
                                           | Template Matching         |
                                           +------------+--------------+
                                                        |
                                                        v
                                           +---------------------------+
                                           | Confidence Decision       |
                                           +------------+--------------+
                                                        |
                             +--------------------------+--------------------------+
                             |                                                     |
                             v                                                     v
                 +------------------------+                           +------------------------+
                 | Clarification          |                           | Query Planning         |
                 +-----------+------------+                           +-----------+------------+
                             |                                                    |
                             v                                                    v
                 +------------------------+                           +------------------------+
                 | Semantic Memory        |                           | SQL Generation         |
                 +------------------------+                           +-----------+------------+
                                                                                  |
                                                                                  v
                                                                      +------------------------+
                                                                      | Query Validation       |
                                                                      +-----------+------------+
                                                                                  |
                                                                                  v
                                                                      +------------------------+
                                                                      | Answer + Lineage       |
                                                                      +------------------------+

The language model appears in multiple places:

Intent classification
Query planning
Clarification generation
Parameter extraction
Result explanation
Follow-up question handling

But the model is not operating alone. It is grounded by semantic assets, trusted templates, permissions, relationship evidence, and validation rules.

That is the difference between natural language analytics as a demo and natural language analytics as enterprise infrastructure.

## Where Semora Can Stand Apart

Databricks has a strong platform-native story. If a company has standardized its data and governance inside Databricks, that integration is hard to beat.

Semora should not try to be a smaller version of Databricks.

It should focus on the problems that appear when the enterprise data world is messy:

Business terms vary across teams.
Field names do not match user language.
Relationships are not fully documented.
Metrics conflict across systems.
Users need answers across multiple platforms.
Clarification should become governed semantic memory.
Natural language queries need explainable lineage.

This is a valuable and distinct engineering problem.

The best semantic layer is not the one that lets the model guess more freely. It is the one that gives the model enough context, evidence, and constraints to answer correctly.

## Conclusion

Natural language analytics is not solved by connecting a language model to a database.

It requires a semantic runtime.

That runtime must classify intent, route by business domain, retrieve semantic assets, match trusted templates, evaluate confidence, clarify only when necessary, generate governed queries, validate results, and save human feedback as reusable semantic knowledge.

Databricks approaches this through a platform-native lakehouse stack.

Semora's opportunity is to build an independent semantic intelligence layer for enterprises whose data, terminology, and relationships span many systems.

For developers and data engineers, the lesson is clear:

Do not make the model guess the enterprise data model. Build the semantic system that lets it reason with evidence.

When Should a Data Agent Ask a Clarifying Question?

Arisyn — Wed, 26 Aug 2026 14:17:00 +0000

A production Text-to-SQL agent should not generate SQL simply because it can interpret a question. It should first decide whether the business intent is clear enough to query safely.

Consider:

Show me our best customers.

A language model can parse this easily.

It can identify customers, search the schema, choose a metric, generate SQL, and return a ranked list.

The problem is not language understanding.

The problem is that best has no unique analytical meaning.

It could mean:

Revenue
Profit
Growth
Retention
Lifetime Value

If the agent silently chooses Revenue, the generated SQL may be syntactically correct and the returned data may be completely real.

But the system has made a business decision the user never made.

This creates an important engineering requirement for production data agents:

Before generating SQL, decide whether the intent is sufficiently resolved.

## 1. Add an Intent Gate Before SQL Generation

A basic Text-to-SQL pipeline often looks like:

Question
   ↓
Schema Retrieval
   ↓
Prompt Construction
   ↓
LLM
   ↓
SQL

For enterprise use, add an intent-resolution gate:

Question
   ↓
Intent Extraction
   ↓
Semantic Candidate Resolution
   ↓
Ambiguity Detection
   ↓
Intent Complete?
   │
   ├── YES → Data Context → SQL
   │
   └── NO  → Clarification
                  ↓
             Update Intent
                  ↓
             Re-evaluate

Clarification is therefore not a conversational feature added after the agent fails.

It is part of query planning.

## 2. Represent Intent Explicitly

Do not let the entire interpretation live only inside the LLM prompt.

Represent the query intent as structured state.

For example:

{
  "entity": "customer",
  "metric": null,
  "metric_candidates": [
    "revenue",
    "profit",
    "growth",
    "retention"
  ],
  "dimensions": [],
  "time_range": "last_quarter",
  "scope": null,
  "requires_clarification": true
}

After the user selects Profit:

{
  "entity": "customer",
  "metric": "profit",
  "dimensions": [],
  "time_range": "last_quarter",
  "scope": null,
  "requires_clarification": false
}

This gives the system an inspectable state transition:

Ambiguous Intent
      ↓
User Clarification
      ↓
Resolved Intent

Only the resolved form should move into SQL generation.

## 3. Detect Different Types of Ambiguity

Not all ambiguity is the same.

A useful implementation should detect at least four categories.

### Metric Ambiguity

Question:

Who are our best customers?

Candidates:

{
  "term": "best",
  "type": "metric",
  "candidates": [
    "revenue",
    "profit",
    "growth",
    "retention"
  ]
}

### Dimension Ambiguity

Question:

Show sales by region.

Possible dimensions:

customer_region
sales_region
billing_region
delivery_region

### Time Ambiguity

Question:

Show recent revenue.

Possible interpretations:

last_7_days
last_30_days
current_month
current_quarter

### Scope Ambiguity

Question:

How many active customers do we have?

Possible business definitions:

purchased_in_last_30_days
logged_in_last_30_days
active_contract
non_churned

These ambiguities are dangerous because each candidate may produce perfectly executable SQL.

## 4. Candidate Count Is Not Enough

A naive ambiguity detector might say:

if len(candidates) > 1:
    clarify()

That creates too many questions.

Suppose Revenue maps to three technical fields, but one is the governed enterprise metric and the others are deprecated or non-authoritative.

There may be multiple candidates, but no meaningful ambiguity.

A better decision considers:

Candidate Similarity
Governed Definition
Authority
Business Impact
Default Availability
User / Workspace Context

For example:

def resolve_concept(term, context):
    candidates = semantic_search(term)

    governed = [
        c for c in candidates
        if c.is_governed and c.is_active
    ]

    if len(governed) == 1:
        return Resolution(
            value=governed[0],
            requires_clarification=False
        )

    return evaluate_ambiguity(
        candidates=candidates,
        context=context
    )

The goal is not to ask whenever the model sees alternatives.

The goal is to ask when multiple materially different interpretations remain valid.

## 5. Score Clarification Need

A useful conceptual model is:

Clarification Need
=
Ambiguity
×
Business Impact

You can extend this for implementation:

Clarification Score
=
Ambiguity Score
× Business Impact
× (1 - Default Confidence)
× (1 - Governance Confidence)

For example:

def clarification_score(concept):
    return (
        concept.ambiguity_score
        * concept.business_impact
        * (1 - concept.default_confidence)
        * (1 - concept.governance_confidence)
    )

Then:

if clarification_score(concept) > THRESHOLD:
    request_clarification(concept)

The exact formula is domain-specific.

The important architectural point is:

Clarification should be a policy decision, not an LLM reflex.

## 6. Estimate Business Impact

Ambiguity only matters if different interpretations can materially change the result.

Consider:

Top 10 customers.

Whether ties return 10 or 11 rows is usually low impact.

Now consider:

Most profitable customers.

If the enterprise has:

Gross Profit
Contribution Profit
Operating Profit

the choice can completely change the ranking.

A practical impact model might use:

LOW
Formatting, display, minor ranking behavior

MEDIUM
Time defaults, optional filters, non-critical dimensions

HIGH
Metric definition, financial scope, entity identity,
relationship path, aggregation grain

High-impact ambiguity should have a lower clarification threshold.

## 7. Resolve What the Enterprise Already Knows

A good clarification engine should ask fewer questions as enterprise knowledge improves.

Suppose the semantic layer already contains:

term: revenue

resolved_metric:
  id: recognized_revenue

status: active
owner: finance

Then:

Show revenue by region last quarter.

should not trigger:

What do you mean by revenue?

The enterprise has already answered that question.

The resolution order should look like:

User Input
   ↓
Session Context
   ↓
Governed Business Definition
   ↓
Workspace Default
   ↓
Candidate Meanings
   ↓
Clarify Only If Still Unresolved

This is critical for usability.

## 8. Generate Clarifications From Known Candidates

Avoid generic questions such as:

Could you clarify what you mean?

The agent already has candidate meanings.

Use them.

Instead:

How should “best customers” be ranked?

[ Revenue ] [ Profit ] [ Growth ] [ Retention ]

Or:

Which region should I use?

[ Customer Region ]
[ Sales Region ]
[ Billing Region ]
[ Delivery Region ]

This transforms clarification from an open-ended conversation into a constrained resolution step.

## 9. Ask the Highest-Impact Question First

A single user question can contain multiple ambiguities.

Example:

Show our best customers in Europe recently.

Potential ambiguity:

best      → Revenue / Profit / Growth
Europe    → Customer / Sales / Billing Region
recently  → 7 / 30 / 90 days

Do not immediately ask three questions.

Rank unresolved concepts:

ambiguities = [
    Ambiguity("best", impact=0.95),
    Ambiguity("Europe", impact=0.35),
    Ambiguity("recently", impact=0.40)
]

ambiguities.sort(
    key=lambda x: x.clarification_score,
    reverse=True
)

Ask:

How should “best” be measured?

After resolving it, re-evaluate the remaining intent.

The other ambiguities may already be resolvable through workspace defaults or governed semantics.

## 10. Use a Clarification Budget

Every clarification has a UX cost.

A useful runtime configuration could be:

clarification:
  max_rounds: 2
  max_questions_per_round: 1
  minimum_impact: 0.5

This forces the system to prioritize.

If the intent remains unresolved after the budget is exhausted, the system can:

1. Show the assumption explicitly
2. Ask the user to choose whether to proceed
3. Refuse high-risk execution

For example:

I can proceed using Revenue as the ranking metric and the last 30 days as “recent.” Continue?

This is better than silently guessing.

## 11. Define Intent Completeness

The agent needs a deterministic stop condition.

For a typical analytical query, required slots might include:

required_intent:
  metric: true
  dimensions: optional
  time_range: true
  filters: optional
  entity_scope: true

Then compute:

def intent_complete(intent, requirements):
    for slot, required in requirements.items():
        if required and intent.get(slot) is None:
            return False

    return True

A more advanced version can include confidence:

Metric       Revenue          0.98
Dimension    Customer         0.99
Time         Last Quarter     0.96
Scope        Enterprise       0.93

SQL generation begins only after the required intent reaches the accepted state.

## 12. Separate Semantic Confidence From Model Confidence

This distinction is important.

An LLM may be:

95% confident

that “best customers” probably means highest revenue.

That does not mean the enterprise has defined it that way.

So distinguish:

Model Confidence

from:

Semantic Resolution Confidence

Semantic confidence should come from evidence such as:

Governed Metric Match
Business Glossary Match
Workspace Configuration
Prior Clarification
User Selection

Do not treat model certainty as proof of business intent.

## 13. Preserve Clarification State Across Turns

Clarification should not restart the entire reasoning process.

Maintain state:

{
  "query_id": "q_1042",

  "original_question":
    "Show our best customers last quarter",

  "resolved": {
    "entity": "customer",
    "time_range": "last_quarter"
  },

  "unresolved": {
    "metric": [
      "revenue",
      "profit",
      "growth"
    ]
  }
}

User:

Profit.

Update only the unresolved slot:

{
  "resolved": {
    "entity": "customer",
    "metric": "profit",
    "time_range": "last_quarter"
  },

  "unresolved": {}
}

Then continue query planning.

This keeps the interaction efficient and auditable.

## 14. Feed Resolved Intent Into Data Context Resolution

Clarification should narrow the downstream search space.

Before clarification:

best customers

may require retrieving context for:

Revenue
Profit
Growth
Retention
Customer

After:

best = Profit

the system can focus on:

Profit Metric
Customer Dimension
Required Data Sources
Required Relationships
Last Quarter

So clarification does more than improve UX.

It reduces context size and downstream reasoning complexity.

## 15. Validate SQL Against Resolved Intent

After SQL generation, compare the query against the intent state.

Suppose the user clarified:

best = Profit

but generated SQL ranks:

ORDER BY revenue DESC

That should fail validation.

Conceptually:

def validate_query(sql_plan, intent):
    violations = []

    if sql_plan.metric != intent.metric:
        violations.append({
            "type": "metric_mismatch",
            "expected": intent.metric,
            "actual": sql_plan.metric
        })

    return violations

Clarification therefore creates a stronger validation target.

## 16. Log Why the Agent Asked

For production systems, store the clarification decision:

{
  "concept": "best",
  "reason": "multiple_valid_metrics",
  "candidates": [
    "revenue",
    "profit",
    "growth",
    "retention"
  ],
  "ambiguity_score": 0.92,
  "business_impact": 0.95,
  "decision": "clarify"
}

This helps answer:

Why did the agent ask this question?
Why did it not ask another one?
Which ambiguities cause the most friction?

It also makes clarification behavior measurable.

## 17. Measure the Clarification System

Useful production metrics include:

### Clarification Rate

% of user queries requiring at least one clarification

### Average Clarification Rounds

Mean number of follow-up turns before query generation

### Resolution Rate

% of ambiguous queries successfully resolved

### Abandonment Rate

% of users leaving during clarification

### Post-Clarification Accuracy

Compare:

SQL / answer accuracy before clarification
vs.
after clarification

### Unnecessary Clarification Rate

How often users select the obvious/default interpretation.

This is important because too many questions can be as harmful as too few.

## 18. A Practical Clarification Runtime

Putting everything together:

User Question
      ↓
Intent Parser
      ↓
Semantic Resolver
      ↓
Candidate Meanings
      ↓
Ambiguity Detector
      ↓
Impact + Governance Evaluation
      ↓
Clarification Needed?
      │
      ├── NO
      │    ↓
      │  Resolved Intent
      │
      └── YES
           ↓
      Ranked Clarification
           ↓
      User Selection
           ↓
      Update Intent State
           ↓
      Re-evaluate
           ↓
      Resolved Intent
           ↓
Data / Relationship Context
           ↓
SQL Generation
           ↓
Intent Validation
           ↓
Execution

The LLM remains important.

But the decision to query is no longer left entirely to free-form generation.

## 19. The Engineering Principle

The goal is not to make the agent less autonomous.

It is to define where autonomy is appropriate.

If the enterprise has already defined the answer, use the definition.

If a safe default exists, use it.

If ambiguity is low impact, proceed.

But when multiple valid business interpretations remain and the choice materially changes the answer:

Give the decision back to the user.

That is not a weakness.

It is good query planning.

## Final Thoughts

A production data agent should not optimize for:

Always generate SQL.

It should optimize for:

Generate SQL when intent is sufficiently resolved.

That requires:

Structured Intent
Semantic Candidates
Ambiguity Detection
Business-Impact Scoring
Governed Defaults
Clarification Budget
Intent Completeness
Post-Generation Validation

The most dangerous enterprise data errors often happen when the model makes a reasonable assumption that nobody explicitly approved.

So before improving the SQL generator again, add one question to the pipeline:

Do we actually know what the user means?

If not, the correct next step is not SQL.

It is clarification.

What Makes Enterprise Data AI-Readable?

Arisyn — Mon, 24 Aug 2026 15:56:00 +0000

A schema tells an AI what exists. It does not tell the AI what the data means, how it should be connected, or when it should not be used.

For decades, enterprise databases have been designed primarily for developers, data engineers, and analysts.

That design worked because humans supplied the missing context.

An experienced analyst knows that invoice_amount is not the same as recognized revenue.

A data engineer knows that two tables should not be joined directly even though their IDs look compatible.

A finance team knows that created_at is an operational timestamp while financial reporting should use settlement_date.

None of this knowledge has to exist in the physical schema for humans to work effectively.

AI agents change that assumption.

If an agent is expected to query enterprise data directly, the data model must communicate much more than:

table
column
type

The real engineering question becomes:

What information does an AI system need before enterprise data becomes reliably usable?

I think the answer is larger than schema retrieval and embeddings.

A useful model is:

AI-Readable Data
=
Meaning
+
Structure
+
Relationships
+
Usage Rules
+
Trust

## 1. Machine-Readable Is Not AI-Readable

Consider this schema:

CREATE TABLE sales_order (
    order_id      BIGINT,
    customer_id   BIGINT,
    total_amount  DECIMAL(18,2),
    created_at    TIMESTAMP,
    status        VARCHAR(20)
);

A model can parse it.

It knows that total_amount is numeric and created_at is temporal.

But it still does not know:

Does total_amount include tax?

Are cancelled orders included?

Is total_amount order value or recognized revenue?

Should created_at be used for financial reporting?

How does customer_id map across CRM and ERP?

Can this table be joined directly to invoice?

The schema describes structure.

Enterprise analytics requires usage context.

That gap is where many AI data failures begin.

## 2. Descriptions Help, but They Do Not Solve the Problem

A common improvement is to add table and column descriptions:

table: sales_order

columns:
  total_amount:
    type: decimal
    description: Total amount of the sales order.

  created_at:
    type: timestamp
    description: Time when the order was created.

This is useful metadata.

But now ask:

What was revenue last quarter?

The description still does not tell the model whether total_amount should be used as Revenue.

A technically accurate description can still be insufficient for analytical reasoning.

The missing information is not:

What is this column?

It is:

When is this column valid for a business question?

## 3. Meaning Should Be Explicit

Suppose the warehouse contains:

sales_order.total_amount
invoice.invoice_amount
finance_revenue.recognized_amount
payment.received_amount

All four fields may be retrieved for the word:

Revenue

Embedding similarity is doing exactly what it should.

The fields are semantically related.

But enterprise AI needs a stronger object:

metric:
  id: revenue
  name: Revenue
  version: 2.1

  definition:
    business_term: Recognized Revenue
    aggregation: SUM

  source:
    table: finance_revenue
    column: recognized_amount

  time:
    field: recognition_date

This changes the problem from:

Which revenue-like field should the model choose?

to:

Retrieve the governed Revenue definition.

That is a much more reliable operation.

## 4. Relationships Need to Be First-Class Context

Even perfect semantic definitions are not enough for multi-table analytics.

Suppose a user asks:

Which customers have unpaid invoices?

The system may need:

Customer
   ↓
Order
   ↓
Invoice
   ↓
Payment

But real enterprise schemas often do not contain complete foreign keys.

A relationship may exist because:

order.customer_id

and:

customer.customer_id

share values.

One useful relationship signal is inclusion:

Inclusion(A → B)
=
|distinct(A) ∩ distinct(B)|
---------------------------
|distinct(A)|

where:

A = order.customer_id
B = customer.customer_id

If the inclusion ratio is high and B has high uniqueness, the system has evidence for a possible reference relationship.

Relationship discovery can combine:

Database Constraints
Column Names
Value Inclusion
Uniqueness
Business Validation

Then relationships can move through states:

Discovered
    ↓
Candidate
    ↓
Validated
    ↓
Trusted

At query time, the AI should prefer trusted relationships instead of reconstructing joins from scratch.

## 5. Positive Knowledge Is Only Half the Model

Most semantic systems are good at representing positive knowledge:

Revenue uses recognized_amount.

But production systems also need negative knowledge:

Do NOT use invoice_amount as Revenue.

That distinction is more important than it looks.

Imagine:

field:
  table: sales_order
  name: total_amount

usage:
  valid_for:
    - order_value
    - sales_volume

  invalid_for:
    - recognized_revenue
    - cash_collection

Now the system knows both:

What this field can answer.

and:

What this field must not answer.

Humans use this kind of knowledge constantly.

We say:

Don't use that table for finance.

Don't join those tables directly.

Don't use created_at for reporting.

That field looks like customer_id, but it is actually account_id.

These constraints often exist only as tribal knowledge.

For AI systems, they should become structured context.

## 6. Why Negative Knowledge Matters for Retrieval

Suppose a vector search for:

customer revenue

returns:

sales_order.total_amount       score 0.91
invoice.invoice_amount         score 0.89
finance_revenue.recognized_amount score 0.87

A naive retrieval pipeline might choose the highest score.

But relevance is not authority.

If the governed context says:

sales_order.total_amount
invalid_for: recognized_revenue

the retrieval system can exclude or penalize that candidate.

Conceptually:

def rank_candidate(candidate, query_context):
    score = candidate.semantic_similarity

    if candidate.is_authoritative_for(query_context.metric):
        score += AUTHORITY_BONUS

    if candidate.is_invalid_for(query_context.metric):
        score -= INVALID_USAGE_PENALTY

    return score

The important shift is that retrieval is no longer based only on similarity.

It becomes:

Similarity
+
Business Validity
+
Trust
+
Usage Constraints

## 7. Usage Rules Should Be Machine-Consumable

A free-text description such as:

This table is generally used for order reporting and should normally not be used for financial revenue reporting.

is useful to humans.

For AI systems, structured rules are easier to enforce:

table: sales_order

valid_for:
  - order_analysis
  - sales_pipeline

invalid_for:
  - recognized_revenue
  - financial_close

preferred_time_field:
  order_analysis: created_at

constraints:
  - exclude_cancelled_orders

This does not mean every business rule must become rigid YAML.

Some rules remain contextual.

But high-value, frequently reused constraints should be represented in a form tools and agents can consume deterministically.

## 8. Trust Signals Resolve Competing Context

Enterprise systems frequently contain multiple definitions.

For example:

Revenue v1.8
Revenue v2.0
Revenue v2.1

Or several candidate relationships:

Customer → Order
Customer → Account → Order
Customer → Contract → Order

An AI-readable context needs signals that help select among them.

For example:

metric:
  id: revenue
  version: 2.1
  status: active
  owner: finance
  effective_from: 2026-01-01
  validated: true

A relationship might include:

relationship:
  id: rel_customer_account
  status: trusted
  confidence: 0.97
  validated_by: data_governance

Now retrieval can distinguish:

Relevant

from:

Relevant + Authoritative

That difference matters in production.

## 9. AI-Readable Context Can Be Built as a Data Object

A richer representation for a field might look like:

field:
  table: sales_order
  name: total_amount
  type: decimal

meaning:
  business_term: Order Value
  description: Confirmed sales order value

structure:
  nullable: false
  distinct_ratio: 0.84

relationships:
  - target: customer.customer_id
    via: sales_order.customer_id
    status: trusted

usage:
  valid_for:
    - order_analysis
    - sales_volume

  invalid_for:
    - recognized_revenue
    - cash_received

trust:
  owner: sales_operations
  status: active
  validated: true

This object contains far more actionable context than:

total_amount DECIMAL

or even:

Total amount of the sales order.

## 10. Build Query Context From AI-Readable Objects

Once these objects exist, query-time context construction becomes more reliable.

Suppose the question is:

Show recognized revenue by customer for last quarter.

A context builder could resolve:

Business Metric
→ Revenue v2.1

Dimension
→ Customer

Relevant Sources
→ finance_revenue
→ customer

Trusted Relationship
→ finance_revenue.customer_id
→ customer.customer_id

Time Field
→ recognition_date

Excluded Candidates
→ sales_order.total_amount
→ invoice.invoice_amount

The resulting model context can be compact:

metric:
  name: Revenue
  source: finance_revenue.recognized_amount
  aggregation: SUM

dimension:
  name: Customer
  source: customer.customer_name

relationship:
  finance_revenue.customer_id -> customer.customer_id

time_field:
  finance_revenue.recognition_date

do_not_use:
  - sales_order.total_amount
  - invoice.invoice_amount

The LLM no longer needs to infer the enterprise model from raw schema.

It receives an instruction-ready representation.

## 11. This Is More Than RAG

A standard RAG pipeline looks like:

Question
   ↓
Embedding
   ↓
Retrieve Relevant Context
   ↓
LLM

An AI-readable data layer introduces additional steps:

Question
   ↓
Resolve Business Concepts
   ↓
Retrieve Relevant Data Objects
   ↓
Apply Usage Constraints
   ↓
Prefer Trusted Definitions
   ↓
Resolve Relationships
   ↓
Build Query Context
   ↓
LLM

The goal is not merely:

Find context.

It is:

Find context that is valid for this analytical task.

## 12. AI-Readable Data Should Be Model-Independent

One architectural mistake is putting all of this context inside a model-specific system prompt.

For example:

If user asks Revenue, use table X.
Never join A directly to B.
For finance reporting use settlement_date.
...

That works initially.

But the knowledge becomes coupled to one agent implementation.

A better architecture keeps enterprise data knowledge outside the model:

AI Agent
    ↓
Context Resolver
    ↓
AI-Readable Data Layer
    ↓
Enterprise Data

Then different agents can consume the same knowledge:

Sales Agent
Finance Agent
Analytics Agent
Operations Agent

The model can change without rebuilding the enterprise's data understanding.

## 13. AI-Readable Data Needs a Lifecycle

This context cannot be static.

Schemas change.

Metrics change.

Relationships change.

Usage rules change.

So the system needs:

Discover
   ↓
Validate
   ↓
Publish
   ↓
Monitor
   ↓
Detect Change
   ↓
Update

For example, if:

Revenue v2.1

becomes:

Revenue v2.2

the old version should not silently disappear.

Historical queries may still need to be reproduced.

The same applies to relationships and usage constraints.

## 14. A Practical Architecture

One possible design:

                Enterprise Data
                      │
                      ▼
          ┌─────────────────────┐
          │ Metadata Extraction │
          └──────────┬──────────┘
                     │
          ┌──────────▼──────────┐
          │ Relationship        │
          │ Discovery           │
          └──────────┬──────────┘
                     │
          ┌──────────▼──────────┐
          │ Semantic / Metric   │
          │ Governance          │
          └──────────┬──────────┘
                     │
          ┌──────────▼──────────┐
          │ Usage Rules &       │
          │ Negative Knowledge  │
          └──────────┬──────────┘
                     │
                     ▼
          AI-Readable Data Objects
                     │
                     ▼
              Context Resolver
                     │
                     ▼
                 AI Agent

The point is not that every implementation needs these exact services.

The important idea is the separation between:

Raw Enterprise Data

and:

Context AI Can Reliably Use

## 15. What Should Be Measured?

If AI-readability becomes a real engineering objective, teams can measure it.

Possible metrics include:

Semantic Coverage

% of important tables / fields mapped to business concepts

Relationship Coverage

% of frequently queried multi-table paths represented as validated relationships

Usage Rule Coverage

% of high-value data objects with explicit valid / invalid usage

Trusted Context Resolution Rate

% of queries where metric + sources + relationships
are resolved without free-form LLM inference

Negative Constraint Hit Rate

How often invalid candidates are removed
because of explicit usage constraints

These metrics move the conversation from:

Our documentation is better.

to:

Our data is increasingly usable by AI systems.

## 16. The New Definition of Data Readiness

Traditional data readiness asks:

Is the data available?
Is it clean?
Is it integrated?
Is it documented?

AI adds another requirement:

Can an AI determine:
- what the data means,
- how it connects,
- when it is valid,
- when it is invalid,
- and why it should be trusted?

That is a much stronger definition.

## Final Thoughts

Enterprise AI is exposing a problem that humans have hidden for years.

Our data models often depend on undocumented organizational knowledge.

Experienced people know:

which field to use,
which table not to use,
which relationship is valid,
which timestamp matters,
which definition is authoritative.

AI does not automatically know any of that.

So the goal should not be merely to give AI access to more schemas.

The goal should be to make enterprise data AI-readable.

That means representing:

Meaning
Structure
Relationships
Usage Rules
Trust

And especially the knowledge most systems still ignore:

What the AI should not do with the data.

Sometimes the difference between a demo and a production data agent is not another model upgrade.

It is one critical piece of enterprise knowledge:

Don't use this data that way.

Beyond SQL Accuracy: Building Evidence Chains for AI Data Agents

Arisyn — Thu, 20 Aug 2026 15:10:00 +0000

AI data agents are getting good at producing executable SQL.

That is useful, but executable SQL is not the same thing as a correct business answer.

A query can compile, run successfully, return real rows, and still answer the wrong question because the agent selected the wrong metric definition, source table, relationship, grain, time field, or filter.

For production systems, this creates a different engineering requirement:

An AI data agent should not only generate an answer. It should preserve the evidence that produced it.

This article explores how to treat that evidence as a first-class artifact rather than an explanation generated after the fact.

## 1. Why SQL Accuracy Is an Incomplete Target

Consider this question:

What was net revenue by region last quarter?

The agent generates:

SELECT
    c.region,
    SUM(i.invoice_amount) AS net_revenue
FROM invoices i
JOIN customers c
    ON i.customer_id = c.customer_id
WHERE i.invoice_date >= '2026-04-01'
  AND i.invoice_date < '2026-07-01'
GROUP BY c.region;

The query is valid.

But suppose the enterprise definition is:

Net Revenue
=
Recognized Revenue
-
Refunds
-
Credits

and regional attribution is based on the billing account rather than the customer master.

The SQL engine cannot detect this error.

From the database's perspective, the query is correct.

From the business's perspective, it is wrong.

That gives us two different validation layers:

SQL Validity
=
Can this query execute?

Business Validity
=
Does this query represent the intended business question?

Production AI analytics needs both.

## 2. The Failure Surface Is Larger Than SQL Generation

A useful way to model an AI data query is:

Question
   ↓
Semantic Resolution
   ↓
Source Selection
   ↓
Relationship Selection
   ↓
Filter / Time Resolution
   ↓
SQL Generation
   ↓
Execution
   ↓
Answer

An error can occur at any stage.

For example:

Semantic Resolution
Revenue → Invoice Amount       ❌

Source Selection
invoices                       ✓

Relationship Selection
invoice.customer_id → customer ✓

Time Resolution
invoice_date                   ❌

SQL Generation
Valid SQL                      ✓

Execution
Success                        ✓

If we evaluate only the final SQL syntax or execution status, the system appears healthy.

The real failure happened much earlier.

This is why AI data agents need evidence chains, not just query logs.

## 3. What Is an Evidence Chain?

An evidence chain records the artifacts and decisions that connect a user question to the final answer.

Conceptually:

User Question
      │
      ▼
Business Definition
      │
      ▼
Source Data
      │
      ▼
Relationships
      │
      ▼
Filters / Time Rules
      │
      ▼
Generated SQL
      │
      ▼
Execution Result
      │
      ▼
Final Answer

The goal is not to create another verbose chain-of-thought log.

The goal is to preserve verifiable system artifacts.

That distinction matters.

Internal model reasoning is probabilistic and may not be suitable as an audit artifact.

A metric identifier, relationship identifier, SQL query, data-source identifier, and execution result are inspectable.

## 4. Evidence Should Be Structured

Instead of storing only:

question
sql
answer

store a structured evidence object.

For example:

{
  "question": "What was net revenue by region last quarter?",

  "semantic_resolution": {
    "metric_id": "net_revenue",
    "metric_version": "3.2",
    "dimensions": ["billing_region"]
  },

  "sources": [
    "orders",
    "refunds",
    "billing_accounts"
  ],

  "relationships": [
    {
      "relationship_id": "rel_orders_billing_account",
      "from": "orders.billing_account_id",
      "to": "billing_accounts.account_id",
      "status": "trusted"
    }
  ],

  "time": {
    "field": "settlement_date",
    "period": "last_quarter"
  },

  "sql": {
    "query_id": "q_8271"
  },

  "execution": {
    "result_id": "r_4412",
    "status": "success"
  },

  "validation": {
    "status": "passed"
  }
}

Now the answer has provenance.

## 5. Capture Evidence During Execution, Not Afterward

A common anti-pattern is:

1. Agent generates answer
2. User asks "Why?"
3. LLM generates an explanation

That produces a narrative, not necessarily provenance.

The model may describe what it believes happened.

A stronger implementation captures evidence as each stage executes.

For example:

evidence = Evidence(question=question)

metric = resolve_metric(question)
evidence.metric_id = metric.id
evidence.metric_version = metric.version

dimension = resolve_dimension(question)
evidence.dimensions.append(dimension.id)

path = relationship_service.get_trusted_path(
    metric=metric,
    dimensions=[dimension]
)
evidence.relationship_ids.extend(path.relationship_ids)

sql = generate_sql(
    question=question,
    metric=metric,
    dimensions=[dimension],
    relationship_path=path
)
evidence.sql = sql

result = execute(sql)
evidence.execution_status = result.status
evidence.result_id = result.id

The evidence object is built from actual system events.

That makes it much more useful for debugging and audit.

6. Semantic Evidence

The first layer is semantic resolution.

If the user asks:

What was revenue last quarter?

the system should be able to show:

User Term:
Revenue

Resolved Metric:
Net Revenue

Version:
3.2

Definition:
Recognized Revenue - Refunds - Credits

This matters because business terms are often ambiguous.

The SQL may be technically perfect while using the wrong definition.

A metric version is particularly important.

If Revenue v3.1 and v3.2 differ, the evidence chain should preserve which version generated the answer.

## 7. Source Evidence

The next question is:

Which data actually contributed to the answer?

For example:

orders
refunds
billing_accounts

A production evidence record may also include fields:

{
  "source": "orders",
  "fields": [
    "billing_account_id",
    "recognized_amount",
    "settlement_date"
  ]
}

This makes it possible to inspect whether the agent used an authoritative source rather than merely a semantically similar table.

## 8. Relationship Evidence

Multi-table queries require another layer of provenance.

Suppose the query joins:

orders
   ↓
billing_accounts
   ↓
region

The evidence should preserve why that path was selected.

At minimum:

{
  "relationship_id": "rel_orders_billing_account",
  "from": "orders.billing_account_id",
  "to": "billing_accounts.account_id",
  "status": "trusted"
}

More advanced systems could include relationship evidence such as:

Database Constraint
Naming Similarity
Value Inclusion
Uniqueness
Business Validation

The important point is that the agent is not silently inventing a join.

The relationship is inspectable.

## 9. Filter and Time Evidence

Filters are easy to overlook because they often appear as simple SQL predicates.

But they can completely change the business answer.

For example:

"last quarter"

might resolve to:

2026-04-01 → 2026-06-30

and use:

settlement_date

rather than:

invoice_date

The evidence record should preserve both.

For example:

{
  "time_resolution": {
    "business_period": "last_quarter",
    "start": "2026-04-01",
    "end": "2026-06-30",
    "field": "settlement_date"
  }
}

This makes temporal interpretation auditable.

## 10. Query Evidence

The SQL itself remains critical evidence.

But do not treat SQL as the entire explanation.

SQL is the executable consequence of upstream decisions.

A useful query record might include:

{
  "query_id": "q_8271",
  "sql_hash": "sha256:...",
  "generated_at": "2026-08-20T09:15:00Z",
  "metric_version": "3.2",
  "relationship_ids": [
    "rel_orders_billing_account"
  ]
}

This makes the query reproducible and connects it to the semantic and relationship state used during generation.

## 11. Execution Evidence

Execution success is useful evidence, just not sufficient evidence.

Capture:

Execution Status
Duration
Rows Returned
Data Source
Query Timestamp
Result Identifier

Potentially:

{
  "status": "success",
  "duration_ms": 842,
  "rows_returned": 8,
  "result_id": "r_4412"
}

This lets the system distinguish:

Bad SQL

from:

Valid SQL + Wrong Business Interpretation

## 12. Validation Should Check the Chain

Instead of asking only:

Did the SQL execute?

validation can inspect several layers.

### Semantic Validation

Was an approved metric definition used?

### Source Validation

Were authorized and current data sources used?

### Relationship Validation

Were joins based on trusted relationships?

### Filter Validation

Were required business filters applied?

### Query Validation

Did the SQL pass syntax and safety checks?

This produces a richer validation object:

{
  "semantic": "passed",
  "sources": "passed",
  "relationships": "passed",
  "filters": "passed",
  "sql": "passed"
}

## 13. Why LLM-as-a-Judge Cannot Be the Only Validator

A second model can review generated SQL.

That is useful.

But consider:

Agent:
Revenue = invoice_amount

Judge:
The SQL correctly sums invoice_amount.

Both models agree.

Both can still be wrong relative to the enterprise definition.

Language models are good at evaluating logical consistency inside the context they receive.

They cannot recover enterprise truth that is missing from that context.

So:

You cannot verify a data answer with language alone.

Verification needs grounded evidence from semantic definitions, metadata, relationships, executed SQL, and query results.

## 14. Evidence Chains Improve Debugging

Suppose an answer is reported as incorrect.

Without structured evidence, the debugging workflow may be:

Reproduce Prompt
↓
Inspect Agent Logs
↓
Inspect SQL
↓
Guess What Went Wrong

With evidence:

Semantic Resolution      PASS
Source Selection         PASS
Relationship Selection   FAIL
Filter Resolution        PASS
SQL Validation           PASS
Execution                PASS

The engineering team immediately knows where to investigate.

This is especially valuable because many production NL2SQL failures happen before SQL generation.

## 15. Evidence Chains Improve Evaluation

Evidence also makes offline and online evaluation more granular.

Instead of only measuring:

Exact SQL Match
Execution Accuracy
Final Answer Accuracy

teams can measure:

Metric Resolution Accuracy
Source Selection Accuracy
Relationship Selection Accuracy
Filter Resolution Accuracy
Trusted Relationship Usage
SQL Validation Pass Rate

This turns a monolithic accuracy score into a diagnostic system.

If final-answer accuracy drops, the team can determine which layer caused the regression.

## 16. Evidence as an API Object

Evidence should not exist only in the UI.

It can be part of the agent API.

For example:

{
  "answer": {
    "metric": "Net Revenue",
    "value": 27300000,
    "currency": "USD"
  },

  "evidence": {
    "metric": {
      "id": "net_revenue",
      "version": "3.2"
    },

    "sources": [
      "orders",
      "refunds",
      "billing_accounts"
    ],

    "relationships": [
      "rel_orders_billing_account"
    ],

    "query_id": "q_8271",

    "validation": "passed"
  }
}

Now downstream systems can choose how much evidence to expose.

## 17. Progressive Disclosure in the UI

Most business users do not want to read SQL after every question.

So the default UI can remain simple:

Net Revenue
$27.3M

✓ Evidence available

Then:

Why this answer?

can reveal:

Metric Definition
Source Data
Relationships
Filters
SQL
Validation

Different users can inspect different levels.

An executive may stop at the answer.

An analyst may inspect the metric and filters.

A data engineer may inspect joins and SQL.

## 18. Evidence Is Stronger Than Explanation

There is an important difference between:

Explainable

and:

Inspectable

An LLM can generate a convincing explanation.

An inspectable system exposes the artifacts that actually produced the answer.

For enterprise data, inspectability is often the stronger trust mechanism.

The user does not have to believe the explanation.

They can inspect the evidence.

## 19. A Practical Evidence Pipeline

A simplified architecture could look like:

User Question
      │
      ▼
Semantic Resolver
      │
      ├── metric_id
      └── dimension_ids
      │
      ▼
Relationship Resolver
      │
      └── relationship_ids
      │
      ▼
Context Builder
      │
      ▼
SQL Generator
      │
      └── query_id
      │
      ▼
SQL Validator
      │
      ▼
Query Executor
      │
      └── result_id
      │
      ▼
Answer Generator
      │
      ▼
Evidence Object

Every stage contributes structured evidence.

The evidence object becomes the trace that connects the original question to the final answer.

## 20. What Should Be Stored?

At minimum:

Question ID
User Question
Resolved Metric + Version
Resolved Dimensions
Source Tables / Fields
Relationship IDs
Filters
Time Interpretation
Generated SQL / Query ID
Execution Result ID
Validation Status
Timestamp

Depending on governance requirements, also consider:

Semantic Model Version
Relationship Model Version
Data Snapshot / Query Timestamp
Authorization Context
Agent Version
Model Version

This makes answers reproducible even as the system evolves.

## 21. A Better Definition of Production Readiness

For a prototype, this may be enough:

Question → SQL → Answer

For production enterprise AI, a stronger standard is:

Question
→ Governed Interpretation
→ Trusted Data Path
→ Executable Query
→ Traceable Result
→ Evidence-Backed Answer

The difference is not cosmetic.

It changes how the system can be trusted, evaluated, debugged, audited, and improved.

## Final Thoughts

AI data agents are rapidly improving at generating SQL.

But SQL generation is only one stage in a much larger correctness problem.

A production system should be able to answer two questions:

What is the answer?

and:

What evidence produced that answer?

That evidence should include the business definition, source data, relationships, filters, SQL, execution result, and validation state.

Not because every user wants to inspect every detail.

But because enterprise trust should be based on something stronger than model confidence.

A trustworthy data agent does not just return a number. It preserves the chain of evidence that makes the number verifiable.

The Reasoning Tax: Why AI Data Agents Waste Tokens Relearning Your Schema

Arisyn — Tue, 18 Aug 2026 02:49:01 +0000

If your data agent has to rediscover metric definitions, table relationships, and trusted query paths on every request, you are spending LLM reasoning on knowledge your system should already have.

AI data agents are becoming increasingly capable.

A modern agent can:

retrieve schemas;
interpret business terms;
identify candidate tables;
infer joins;
generate SQL;
validate queries;
execute them;
explain the result.

That looks like progress.

But from an engineering perspective, there is an uncomfortable question:

How much of this work is genuinely new reasoning, and how much is the agent repeatedly rediscovering facts the enterprise already knows?

That repeated work creates a hidden cost: the reasoning tax.

## A Typical Data Agent Does Too Much at Query Time

Consider this question:

What was revenue by customer last quarter?

A typical agent pipeline may look like:

User Question
      ↓
Intent Detection
      ↓
Schema Retrieval
      ↓
Business Term Resolution
      ↓
Candidate Table Selection
      ↓
Relationship Discovery
      ↓
Join Path Selection
      ↓
Metric Construction
      ↓
SQL Generation
      ↓
SQL Validation
      ↓
Execution

Now the user asks:

Show customer revenue for Q2.

The analytical intent is almost identical.

But many implementations repeat most of the pipeline.

The agent may again retrieve schemas, resolve Revenue, identify Customer, compare join paths, and choose the reporting date.

This is wasteful because much of that information is not query-specific.

*## Separate Query-Time Reasoning From Reusable Knowledge
*
A useful engineering distinction is:

Reusable Enterprise Knowledge
vs.
Query-Specific Reasoning

### Reusable Enterprise Knowledge

Examples:

Revenue = Recognized Revenue

Customer = canonical customer entity

Revenue Date = recognition_date

Customer → Order = trusted relationship

Order → Revenue = validated query path

These should not be probabilistically reconstructed on every request.

### Query-Specific Reasoning

Examples:

What is the user asking?

Does "last quarter" mean fiscal or calendar quarter?

Should results be grouped by customer or customer segment?

Is the user asking for a comparison?

What should be investigated next?

These genuinely depend on the current request.

The design principle is:

Retrieve the known. Reason about the unknown.

## Why the Reasoning Tax Matters

There are three immediate engineering consequences.

### 1. Token Usage

Suppose schema retrieval returns 30 tables, each with:

table descriptions;
columns;
data types;
comments;
sample values.

The agent may receive thousands of tokens before reasoning even starts.

If every query repeatedly includes the same structural information, token consumption scales with query volume.

A rough model is:

Total Token Cost
≈
Requests
×
(Context Tokens + Reasoning Tokens + Output Tokens)

Reducing repeated context has a direct effect on cost.

### 2. Latency

Agent workflows often involve multiple model or tool calls:

Retrieve
→ Classify
→ Resolve
→ Plan
→ Generate
→ Validate

Even if each step takes only a small amount of time, the total latency accumulates.

If a trusted metric definition can be returned from a deterministic service in milliseconds, there is little value in asking an LLM to infer it again.

### 3. Inconsistency

Repeated reasoning also introduces variability.

One request may produce:

Revenue → invoice_amount

Another:

Revenue → recognized_revenue

Another:

Revenue → order_amount

The model may be behaving reasonably in all three cases.

The architecture is simply asking it to repeatedly solve an ambiguous problem.

Govern the definition once and retrieve it consistently.

## Schema Retrieval Is Necessary, but It Is Not Enough

A common NL2SQL architecture is:

Question
   ↓
Embedding Search
   ↓
Relevant Tables
   ↓
LLM
   ↓
SQL

This is much better than sending the entire warehouse schema.

But table retrieval still leaves unresolved questions:

Which metric is authoritative?

Which entity is canonical?

Which relationship is trusted?

Which date field should be used?

Which join path is safe?

Retrieval that returns only schema is still forcing the LLM to reconstruct business knowledge.

A richer query context should return something closer to:

question: "revenue by customer last quarter"

metric:
  name: revenue
  definition: recognized_revenue
  aggregation: SUM

dimensions:
  - customer

time:
  field: recognition_date

tables:
  - customer
  - sales_order
  - finance_revenue

trusted_path:
  - customer
  - sales_order
  - finance_revenue

Now the LLM is not discovering the data model.

It is using it.

## Precompute Relationship Knowledge

Relationship inference is a major source of unnecessary query-time reasoning.

Suppose the agent needs to connect:

Customer

to:

Payment

It may discover several candidate paths:

Customer → Order → Invoice → Payment

Customer → Account → Payment

Customer → Contract → Invoice → Payment

If the organization has already validated the first path for this analytical scenario, there is no reason to compare all three paths again.

Relationship knowledge can be discovered ahead of time using evidence such as:

Primary / Foreign Keys
Column Naming
Value Overlap
Uniqueness
Inclusion Ratio
Historical Query Patterns
Business Validation

For example:

A = order.customer_id
B = customer.customer_id

A useful signal is:

Inclusion(A → B)
=
|distinct(A) ∩ distinct(B)|
---------------------------
|distinct(A)|

Relationship candidates can then move through:

Discovered
    ↓
Candidate
    ↓
Validated
    ↓
Trusted

Query-time agents should preferentially retrieve the trusted result.

## Move Metric Resolution Out of the Prompt

Metric definitions are another common source of repeated reasoning.

Instead of embedding this in every system prompt:

When the user says revenue, use recognized_amount
from finance_revenue unless...

maintain a governed metric object:

{
  "metric": "revenue",
  "version": "2.1",
  "aggregation": "SUM",
  "table": "finance_revenue",
  "column": "recognized_amount",
  "time_field": "recognition_date",
  "status": "active"
}

The agent calls:

get_metric_definition("revenue")

and receives the current definition.

This has several advantages:

one definition across agents;
easier versioning;
easier auditing;
less prompt complexity;
fewer tokens;
less ambiguity.

## Build Query Context Before Calling the LLM

A useful architecture is:

                User Question
                     │
                     ▼
          ┌────────────────────┐
          │ Context Resolver   │
          ├────────────────────┤
          │ Business Terms     │
          │ Metrics            │
          │ Metadata           │
          │ Relationships      │
          │ Trusted Paths      │
          └─────────┬──────────┘
                    │
                    ▼
             Trusted Context
                    │
                    ▼
                  LLM
                    │
             Reason / Generate
                    │
                    ▼
                   SQL

The LLM receives only what is relevant.

This changes the role of the model.

Before:

LLM = Data Discovery + Business Interpretation + Reasoning + SQL

After:

Data Layer = Known Enterprise Facts

LLM = Intent + Reasoning + SQL / Analysis

## A Simple Context Resolver

Conceptually, the resolver could work like this:

def build_query_context(question):
    concepts = resolve_business_terms(question)

    metrics = get_metric_definitions(concepts)
    entities = get_business_entities(concepts)

    metadata = get_relevant_metadata(
        metrics=metrics,
        entities=entities
    )

    relationships = get_trusted_relationships(
        metadata.tables
    )

    return {
        "metrics": metrics,
        "entities": entities,
        "metadata": metadata,
        "relationships": relationships
    }

Then:

context = build_query_context(question)

sql = llm.generate_sql(
    question=question,
    context=context
)

The exact implementation will vary.

The architectural point is that the model does not have to infer every layer of enterprise knowledge itself.

## Cache Stable Knowledge at the Right Level

Not all knowledge changes at the same frequency.

For example:

Table Schema
→ changes occasionally

Metric Definition
→ changes occasionally

Trusted Relationship
→ changes occasionally

User Question
→ changes every request

This suggests different caching and refresh strategies.

A system might maintain:

Metadata Cache
Semantic Cache
Relationship Cache
Query Context Cache

with invalidation triggered by:

Schema Change
Metric Version Change
Relationship Update
Governance Update

This is more efficient than treating every query as a completely new reasoning problem.

## MCP Can Expose the Known Layer

For agent-based architectures, MCP can provide a clean interface to reusable enterprise knowledge.

For example:

get_metric_definition

get_business_entity

get_table_metadata

get_trusted_relationships

get_query_context

The agent workflow becomes:

Question
   ↓
Agent
   ↓
MCP Tools
   ↓
Enterprise Data Intelligence
   ↓
Trusted Context
   ↓
LLM Reasoning

MCP does not remove the need for semantic or relationship intelligence.

It gives agents a standardized way to access it.

## Do Not Confuse Precomputation With Hard-Coding

Moving knowledge out of query-time reasoning does not mean freezing the data model.

Enterprise knowledge changes.

Schemas evolve.

Metrics change.

Relationships change.

The reusable layer therefore needs:

Discover
   ↓
Detect Change
   ↓
Evaluate Impact
   ↓
Validate
   ↓
Version
   ↓
Publish
   ↓
Invalidate Cache

The objective is not:

Never reason about the data model again.

It is:

Do not reason about the same established data knowledge on every request.

## Measure the Reasoning Tax

Teams can make this problem observable.

Useful metrics might include:

### Context Tokens per Query

Average tokens sent before generation

*### Reasoning Calls per Query
*

Average LLM calls required before SQL execution

*### Schema Candidates per Query
*

How many tables / columns must the model evaluate?

### Relationship Resolution Rate

% of queries using prevalidated relationships

### Metric Resolution Rate

% of business metrics resolved without LLM inference

### Time to First SQL

Question received
→
Executable SQL generated

These metrics can reveal whether the agent is spending most of its time solving analytical problems or reconstructing the enterprise data model.

## The Optimization Target Changes

A common optimization question is:

Which model gives the best SQL accuracy?

That still matters.

But production systems should also ask:

How much unnecessary reasoning are we forcing the model to perform?

A more capable model may hide poor architecture by successfully reasoning through large amounts of noisy context.

That does not make the architecture efficient.

The better system may be the one that gives the model less to figure out.

## Final Thoughts

AI agents are good at reasoning.

Reasoning is also probabilistic, expensive, and slower than deterministic lookup.

So use it where it adds value.

If Revenue already has a governed definition:

retrieve it.

If Customer already has a canonical entity:

retrieve it.

If a join path is already trusted:

retrieve it.

Then let the model solve the genuinely new problem:

what the user is asking and how to analyze it.

The next generation of enterprise data agents may not win by thinking harder.

They may win by knowing what they no longer need to think about.

Why Every AI Agent Shouldn't Have to Rediscover Your Data Model

Arisyn — Fri, 14 Aug 2026 01:38:04 +0000

Enterprise AI is moving toward a multi-agent architecture.

Instead of one general-purpose assistant, organizations are starting to build specialized agents for:

sales;
finance;
operations;
customer support;
analytics;
management.

That creates an engineering problem that is easy to miss.

If every agent independently receives database schemas, retrieves metadata, learns metric definitions, discovers join paths, and builds its own interpretation of the business, then every new agent becomes another data-modeling project.

The result is not only duplicated engineering work.

It is duplicated enterprise understanding.

A better architecture is to separate agent reasoning from shared data intelligence.

## The Problem With Agent-Local Data Knowledge

A simple data agent often looks something like:

User
  ↓
Agent
  ├── System Prompt
  ├── Schema Retrieval
  ├── Metric Definitions
  ├── Join Instructions
  ├── SQL Tool
  └── LLM
        ↓
    Database

For a single proof of concept, this is reasonable.

Now imagine five teams building five agents:

Sales Agent
Finance Agent
Supply Chain Agent
Support Agent
Analytics Agent

Each team has to solve the same problems:

Which tables are relevant?

What does "Customer" mean?

Which definition of Revenue is authoritative?

How should Customer connect to Order?

Which data source should be trusted?

The implementation may be different, but the underlying enterprise knowledge is largely the same.

If that knowledge lives inside each agent, duplication begins immediately.

## Example: Five Agents, Three Definitions of Revenue

Suppose an enterprise contains:

sales_order.total_amount
finance_invoice.invoice_amount
finance_revenue.recognized_amount
payment.received_amount

A sales agent might map:

Revenue → sales_order.total_amount

A finance agent might use:

Revenue → finance_revenue.recognized_amount

An analytics agent might retrieve:

Revenue → finance_invoice.invoice_amount

All three agents may generate syntactically correct SQL.

All three queries may execute successfully.

But the organization now has three AI systems answering the same business question differently.

The problem is not LLM reasoning.

The problem is that business meaning was implemented locally inside each agent.

## Business Knowledge Should Be an Enterprise Dependency

Agent-specific knowledge and enterprise-wide knowledge should be separated.

An agent may legitimately own:

task instructions;
workflow logic;
persona;
tool selection;
planning strategy.

But definitions such as these should not be duplicated:

Customer
Revenue
Active Customer
Inventory Balance
Gross Margin

The same applies to data relationships:

Customer
   ↓
Order
   ↓
Invoice
   ↓
Payment

These are enterprise data assets.

They should be reusable dependencies.

## Thick Agents vs. Thin Agents

A thick agent contains a large amount of enterprise-specific intelligence:

Agent
├── LLM
├── Business Semantics
├── Metrics
├── Schema Knowledge
├── Relationship Knowledge
├── Business Rules
├── Query Logic
└── Tools

This creates coupling.

If the Revenue definition changes, multiple agents may need to change.

If a trusted join path changes, multiple prompts or tools may need to be updated.

If the organization switches models, important business knowledge may be buried inside model-specific implementation.

A thinner design looks different:

Agent
├── LLM
├── Task Logic
├── Planning
└── Tools
      ↓
Shared Data Intelligence
├── Business Semantics
├── Metrics & Dimensions
├── Metadata
├── Trusted Relationships
└── Query Context
      ↓
Enterprise Data

Now the agent does not need to rediscover the enterprise every time it answers a question.

## What Should the Shared Data Layer Expose?

The shared layer does not have to be one monolithic service.

It can expose several reusable capabilities.

### 1. Metadata Retrieval

For example:

{
  "table": "sales_order",
  "columns": [
    "order_id",
    "customer_id",
    "order_date",
    "total_amount"
  ]
}

But raw metadata alone is not enough.

### 2. Business Semantics

The agent should be able to resolve:

"recognized revenue"

into something like:

metric: revenue
definition: recognized revenue
aggregation: SUM
source:
  table: finance_revenue
  column: recognized_amount

This definition can then be reused by every authorized agent.

### 3. Relationship Discovery

Suppose the agent needs Customer and Payment data.

It should not have to guess the join path from raw schemas.

A relationship service could return:

Customer
   ↓ customer_id
Order
   ↓ order_id
Invoice
   ↓ invoice_id
Payment

along with evidence or confidence information.

For example:

{
  "path": [
    "customer",
    "sales_order",
    "invoice",
    "payment"
  ],
  "status": "trusted"
}

### 4. Query Context

Instead of sending an entire warehouse schema into the prompt, the agent can receive only the relevant context:

Question
   ↓
Resolve Business Concepts
   ↓
Retrieve Relevant Metadata
   ↓
Retrieve Trusted Relationships
   ↓
Generate Query Context
   ↓
LLM

This reduces unnecessary context and gives the model more targeted information.

## Why Schema Retrieval Alone Is Not Enough

A common pattern today is:

Question
   ↓
Embedding Search
   ↓
Relevant Tables
   ↓
LLM
   ↓
SQL

This is useful.

It solves the problem of sending thousands of tables to the model.

But it does not answer:

Which metric definition is authoritative?

Which relationship is trusted?

Which business entity does this table represent?

Retrieval answers:

What looks relevant?

A shared data intelligence layer must also answer:

What is valid for this business question?

That distinction matters more as the number of agents grows.

## Relationship Knowledge Should Be Evidence-Based

Enterprise relationships are often not fully represented by foreign keys.

A relationship engine can use several signals:

Database constraints
Naming similarity
Value overlap
Uniqueness
Inclusion relationships
Validated business mappings

For example, if:

A = order.customer_id
B = customer.customer_id

we can calculate:

Inclusion(A → B)
=
|distinct(A) ∩ distinct(B)|
---------------------------
|distinct(A)|

A high inclusion ratio can provide evidence that A references B.

But discovered relationships should not automatically become business truth.

A safer lifecycle is:

Discovered
    ↓
Candidate
    ↓
Validated
    ↓
Trusted

Agents should preferentially consume trusted relationships rather than independently inventing joins.

## MCP Fits Here — But It Is Not the Data Model

MCP gives agents a standardized way to discover and call tools.

That makes it a natural interface for shared data capabilities.

For example, an enterprise data MCP server could expose tools such as:

get_metric_definition()

get_table_metadata()

discover_relationships()

get_query_context()

An agent could then call:

Agent
  ↓
MCP
  ↓
Shared Data Intelligence Services

But MCP itself does not determine:

What Revenue means

Which customer definition is authoritative

Which join path is trusted

Those decisions come from the data intelligence behind the interface.

A useful distinction is:

MCP standardizes access. Shared data intelligence standardizes understanding.

## Centralization Does Not Mean Static Knowledge

Moving enterprise knowledge into a shared layer solves duplication, but it creates another responsibility:

the shared knowledge must stay current.

Enterprise systems continuously change:

New tables
New columns
New metrics
New business rules
New relationships

So the shared layer needs its own lifecycle:

Discover
   ↓
Detect Change
   ↓
Evaluate Impact
   ↓
Validate
   ↓
Version
   ↓
Publish

Otherwise the organization simply replaces many stale agent configurations with one stale central configuration.

## What Happens When a Metric Changes?

Suppose:

Revenue v1
=
SUM(invoice_amount)

changes to:

Revenue v2
=
SUM(recognized_amount)

With agent-local knowledge:

Sales Agent → update
Finance Agent → update
Analytics Agent → update
Management Agent → update
Support Agent → maybe update

Someone has to discover every dependency.

With shared semantics:

Revenue Definition
       ↓
Version Update
       ↓
Shared Data Intelligence
       ↓
All Authorized Agents

The enterprise changes the definition once.

Agents consume the updated version.

That is a much cleaner dependency model.

## Shared Data Intelligence Also Reduces Agent Coupling

There is another benefit.

Agent frameworks are changing quickly.

Organizations may switch between:

models;
orchestration frameworks;
agent runtimes;
application interfaces.

Business knowledge should survive those changes.

If metric definitions, data relationships, and business mappings are independent of the agent implementation, agents become easier to replace.

That suggests an architectural principle:

Agents = Replaceable Compute / Reasoning

Enterprise Data Intelligence = Durable Knowledge

The enterprise should own its understanding of itself.

Not the current agent framework.

## A Practical Multi-Agent Data Architecture

A simplified architecture might look like:

               Sales Agent
                    │
              Finance Agent
                    │
             Analytics Agent
                    │
             Operations Agent
                    │
                    ▼
        ┌──────────────────────┐
        │ Shared Data          │
        │ Intelligence Layer   │
        ├──────────────────────┤
        │ Business Semantics   │
        │ Metrics & Dimensions │
        │ Metadata             │
        │ Relationships        │
        │ Trusted Query Paths  │
        └──────────────────────┘
                    │
                    ▼
             Enterprise Data

The shared layer does not replace agents.

It makes them thinner.

It does not replace the LLM.

It gives the LLM a consistent representation of the enterprise.

## Final Thoughts

The first generation of enterprise agents focused on making individual agents more capable.

The multi-agent era creates a different engineering problem:

How do many agents share one consistent understanding of enterprise data?

If every agent independently discovers schemas, defines metrics, and guesses relationships, organizations will create duplicated logic and inconsistent answers.

The better pattern is separation of concerns:

Agents reason.

Shared data intelligence defines what the enterprise means.

Models will change.

Agents will change.

Frameworks will change.

Your enterprise should not have to rediscover its own data model every time they do.

Semantic Drift: The Hidden Failure Mode of Enterprise AI Analytics

Arisyn — Wed, 12 Aug 2026 15:20:00 +0000

Enterprise AI systems rarely fail only because the model is weak.

A more subtle failure happens when the model is working correctly, the SQL executes successfully, and the result looks reasonable — but the system is reasoning over an outdated representation of the business.

This is semantic drift.

As more teams put semantic layers between LLMs and enterprise data, maintaining those semantics becomes a production engineering problem rather than a one-time modeling task.

The database changes.

Business definitions change.

Relationships change.

If the AI's understanding does not change with them, accuracy degrades quietly.

A Semantic Layer Is Runtime Infrastructure

A common AI analytics architecture looks roughly like this:

User Question
     ↓
LLM / Query Agent
     ↓
Semantic Layer
     ↓
Enterprise Data

The semantic layer may contain:

business terms;
metric definitions;
dimensions;
mappings to physical tables and columns;
relationships used to generate queries.

This helps prevent an LLM from guessing directly from raw schemas.

But there is an important consequence:

Once AI depends on the semantic layer at query time, stale semantics become a runtime failure.

The problem is that semantic models are often maintained as if they were documentation.

Define them once.

Review them.

Publish them.

Then assume they remain correct.

Enterprise data does not behave that way.

Failure Mode 1: Schema Drift

Suppose a metric is mapped to:

customer.customer_id

After a warehouse migration, the organization introduces:

account_customer.customer_key

The old table may remain available for compatibility.

That creates a dangerous situation.

Nothing necessarily breaks.

The old query can still execute.

The semantic mapping is simply pointing to a representation that is no longer authoritative.

Traditional schema monitoring may tell you that a column was added.

AI analytics needs to answer a harder question:

Does this schema change invalidate any business meaning used by the AI?

That requires connecting physical metadata changes to semantic dependencies.

Failure Mode 2: Semantic Drift

Semantic drift happens even when the schema does not change.

Consider:

Active Customer

Version 1:

customer.status = 'ACTIVE'

Later, the business changes the definition:

customer with >= 1 completed order in the last 90 days

The database can remain exactly the same.

But the meaning has changed.

An AI system using the old definition may continue returning perfectly valid SQL and perfectly wrong business answers.

This is why execution success is a weak validation signal for enterprise AI.

SQL succeeded != business meaning is correct

Failure Mode 3: Relationship Drift

This problem becomes more interesting when queries span multiple tables.

Imagine the original analytical path is:

Customer
   ↓ customer_id
Order

After an ERP redesign:

Customer
   ↓
Account
   ↓
Order

The old join may still work because legacy identifiers remain populated.

But the organization has changed the business relationship.

An LLM that sees both paths now has multiple executable options.

Which one is correct?

This cannot be solved reliably with schema retrieval alone.

The system needs maintained relationship knowledge.

Why Embeddings Do Not Solve Drift

A common approach is to embed schema metadata and retrieve relevant tables for each question.

For example:

User Question
     ↓
Embedding Search
     ↓
Relevant Tables / Columns
     ↓
LLM
     ↓
SQL

This is useful for reducing schema size.

But semantic similarity does not tell us whether a definition is current.

Embedding search may find:

invoice_amount
recognized_revenue
payment_amount

for the term:

revenue

All three are semantically related.

Only business governance can determine which one currently represents the metric.

Likewise, embeddings may identify two similar identifiers, but similarity does not prove that they form the trusted join path.

Retrieval solves relevance.

It does not solve validity.

Treat Semantic Assets Like Versioned Code

If semantic definitions affect AI-generated answers, they should be treated more like production code than documentation.

A metric definition should have:

metric: revenue
version: 2.1
status: active
definition: recognized revenue
source:
  table: finance_revenue
  column: recognized_amount
valid_from: 2026-07-01

A relationship should also carry explicit evidence:

relationship:
  from: customer.customer_id
  to: account.customer_id
type: business_validated
confidence: 0.97
status: active

The exact format is less important than the engineering principle:

Meaning needs identity, state, history, and validation.

Without versioning, it becomes difficult to answer basic production questions:

Which definition generated this answer?
When did that definition change?
Which queries are affected?
Can we roll back?

Detect Changes Before They Become Wrong Answers

A living semantic system needs change detection.

At the physical layer, monitor:

New table
Column added
Column removed
Type changed
Constraint changed

At the relationship layer:

New candidate relationship
Join coverage changed
Identifier uniqueness changed
Relationship confidence changed

At the semantic layer:

Metric definition changed
Mapping became ambiguous
Business term changed
New semantic version published

The important part is not generating more alerts.

It is calculating impact.

For example:

finance_invoice.invoice_amount changed
                 ↓
revenue metric mapping affected
                 ↓
12 query templates affected
                 ↓
AI queries using Revenue require validation

Now schema monitoring becomes useful to AI governance.

Relationship Discovery Should Produce Candidates, Not Truth

Automatically discovering relationships can help keep a data model current.

Signals may include:

declared primary/foreign keys;
naming similarity;
value overlap;
uniqueness;
inclusion ratio;
historical joins.

For two columns A and B, a simple inclusion signal could be:

inclusion(A → B)
=
|distinct(A) ∩ distinct(B)|
---------------------------
|distinct(A)|

A high value can indicate a possible relationship.

But it should not automatically become trusted business logic.

A better lifecycle is:

Relationship Detected
        ↓
Candidate
        ↓
Evidence / Confidence
        ↓
Validation
        ↓
Trusted Relationship

This distinction matters.

Automated discovery improves coverage.

Governance establishes trust.

Human-in-the-Loop Is a Feature, Not a Failure

Enterprise semantics often cannot be inferred safely from technical metadata alone.

If the system finds:

Revenue

mapped plausibly to both:

sales_order.total_amount

and:

finance_revenue.recognized_amount

the correct behavior may be to ask for clarification.

A useful workflow is:

Ambiguity Detected
       ↓
Candidate Definitions
       ↓
Human Review
       ↓
Validate With Query
       ↓
Publish New Version

The goal of automation is not to eliminate domain experts.

It is to stop asking them to manually rediscover every schema and relationship change.

A Practical Living Data Model Loop

Putting the pieces together, the lifecycle looks like:

1. Discover metadata
        ↓
2. Detect schema and relationship changes
        ↓
3. Identify impacted semantic assets
        ↓
4. Generate candidate updates
        ↓
5. Validate ambiguous business meaning
        ↓
6. Version and publish
        ↓
7. Use validated semantics for AI queries
        ↓
8. Repeat

This is fundamentally different from:

Build semantic layer → Done

The semantic model becomes an operational system.

What Should Be Monitored?

A few useful signals include:

Semantic Coverage

How much of the active analytical surface has governed meaning?

governed metrics / queried metrics

Relationship Coverage

How many required multi-table query paths are backed by trusted relationships?

Ambiguity Rate

How often does a business term map to multiple plausible definitions?

Stale Mapping Rate

How many semantic mappings reference changed or deprecated physical assets?

Validation Failure Rate

How often do proposed semantic or relationship updates fail business validation?

These metrics tell you more about production readiness than simply measuring whether SQL execution succeeds.

The Key Engineering Shift

The first generation of LLM analytics focused heavily on query generation.

The next engineering challenge is maintaining the data knowledge used to generate those queries.

A stronger model cannot compensate for a stale business definition.

A larger context window cannot determine whether an old join path is still authoritative.

And better embeddings cannot decide when the organization changed the meaning of revenue.

The system needs a maintained layer of enterprise data knowledge.

Not static metadata.

Not a one-time semantic project.

A living model.

Final Thoughts

If your AI analytics system depends on business semantics, those semantics are production infrastructure.

Treat them accordingly.

Monitor changes.

Track relationships.

Version definitions.

Detect ambiguity.

Validate business meaning.

Because the most dangerous enterprise AI failure is not always a broken query.

Sometimes the query works perfectly — against yesterday's understanding of the business.

Building AI Analysts: Why Enterprise Analytics Needs More Than BI Dashboards

Arisyn — Mon, 10 Aug 2026 01:25:31 +0000

For decades, enterprises have relied on dashboards as the primary way to consume business data.

A typical workflow looks like:

Business Question
↓
Find the Right Dashboard
↓
Select Filters
↓
Review Metrics
↓
Human Analysis

This model works well for predefined questions.

But modern businesses increasingly ask questions that were never designed into dashboards:

Why did sales decline in East China last month?

Answering this requires combining:

sales data;
customer data;
product data;
regional information;
order history.

A dashboard can show what happened.

But understanding why it happened requires deeper analysis.

This is where AI Analysts become valuable.

From BI Dashboards to AI Analysts

AI changes the interaction model.

Instead of searching for reports, users can directly express business questions.

Business Question

↓

AI Understanding

↓

Data Discovery

↓

Multi-dimensional Analysis

↓

Business Insight

The goal is not simply generating charts.

The goal is helping users understand:

What happened?
Why did it happen?
What factors contributed?
What should be investigated next?

Why Building AI Analysts Is Difficult

Connecting an LLM to a database is not enough.

Enterprise analytics requires understanding the business behind the data.

An AI Analyst needs to understand:

business concepts;
data relationships;
analytical logic.

Understanding Business Concepts

Business users do not think in tables.

They ask:

Show me high-value customers.

But enterprises may have:

customer_table

account_table

crm_customer

billing_customer

The AI needs to understand:

Which customer definition is correct?
Which attributes define value?
Which data source should be trusted?

A business concept is not always a table.

It is a combination of entities, metrics, rules, and relationships.

Understanding Data Relationships

Enterprise analysis rarely depends on one table.

A typical path may involve:

Customer
↓
Order
↓
Contract
↓
Payment

However, enterprises often contain:

multiple applications;
duplicated entities;
different identifiers;
historical data models.

A database may allow many joins.

But only some relationships represent business logic.

The important question is not:

Can these tables be joined?

It is:

Is this the correct business relationship?

Moving From Answers to Analysis

Traditional systems answer:

What is the number?

AI Analysts should answer:

Why is this number changing?

For example:

Sales decreased last month. Why?

The system should analyze:

regional changes;
customer changes;
product changes;
historical trends.

This requires dynamic analysis instead of fixed dashboards.

The Architecture Behind AI-Native Analytics

A future analytics system is not simply:

User
↓
LLM
↓
Database

A more reliable architecture requires:

User Question

↓

AI Understanding

↓

Semantic Layer

↓

Relationship Intelligence

↓

Query Generation

↓

Analysis Result

AI needs to understand:

Business Semantics

What does the data mean?

Examples:

revenue definition;
active customer definition;
inventory calculation.

Data Relationships

How is enterprise data connected?

Examples:

customer to order;
order to invoice;
invoice to payment.

Analytical Context

What analysis should happen next?

Examples:

comparison;
trend analysis;
root cause exploration.

The Future of Enterprise Analytics

The next generation of analytics will combine:

large language models;
semantic understanding;
relationship intelligence;
enterprise data governance.

The future is not about creating more dashboards.

It is about creating AI systems that can become intelligent partners in business analysis.

Final Thoughts

Traditional BI helped enterprises understand what happened.

AI Analysts will help enterprises understand why it happened.

But this requires more than connecting an LLM to a database.

AI needs to understand business concepts, data relationships, and analytical context.

Only then can enterprise AI move from answering questions to supporting real analysis.