DEV Community

Arisyn
Arisyn

Posted on

The Reasoning Tax: Why AI Data Agents Waste Tokens Relearning Your Schema

#ai #data #llm #sql

If your data agent has to rediscover metric definitions, table relationships, and trusted query paths on every request, you are spending LLM reasoning on knowledge your system should already have.

AI data agents are becoming increasingly capable.

A modern agent can:

  • retrieve schemas;
  • interpret business terms;
  • identify candidate tables;
  • infer joins;
  • generate SQL;
  • validate queries;
  • execute them;
  • explain the result.

That looks like progress.

But from an engineering perspective, there is an uncomfortable question:

How much of this work is genuinely new reasoning, and how much is the agent repeatedly rediscovering facts the enterprise already knows?

That repeated work creates a hidden cost: the reasoning tax.

## A Typical Data Agent Does Too Much at Query Time

Consider this question:

What was revenue by customer last quarter?

A typical agent pipeline may look like:

User Question
      ↓
Intent Detection
      ↓
Schema Retrieval
      ↓
Business Term Resolution
      ↓
Candidate Table Selection
      ↓
Relationship Discovery
      ↓
Join Path Selection
      ↓
Metric Construction
      ↓
SQL Generation
      ↓
SQL Validation
      ↓
Execution
Enter fullscreen mode Exit fullscreen mode

Now the user asks:

Show customer revenue for Q2.

The analytical intent is almost identical.

But many implementations repeat most of the pipeline.

The agent may again retrieve schemas, resolve Revenue, identify Customer, compare join paths, and choose the reporting date.

This is wasteful because much of that information is not query-specific.

*## Separate Query-Time Reasoning From Reusable Knowledge
*
A useful engineering distinction is:

Reusable Enterprise Knowledge
vs.
Query-Specific Reasoning
Enter fullscreen mode Exit fullscreen mode

### Reusable Enterprise Knowledge

Examples:

Revenue = Recognized Revenue

Customer = canonical customer entity

Revenue Date = recognition_date

Customer → Order = trusted relationship

Order → Revenue = validated query path
Enter fullscreen mode Exit fullscreen mode

These should not be probabilistically reconstructed on every request.

### Query-Specific Reasoning

Examples:

What is the user asking?

Does "last quarter" mean fiscal or calendar quarter?

Should results be grouped by customer or customer segment?

Is the user asking for a comparison?

What should be investigated next?
Enter fullscreen mode Exit fullscreen mode

These genuinely depend on the current request.

The design principle is:

Retrieve the known. Reason about the unknown.

## Why the Reasoning Tax Matters

There are three immediate engineering consequences.

### 1. Token Usage

Suppose schema retrieval returns 30 tables, each with:

  • table descriptions;
  • columns;
  • data types;
  • comments;
  • sample values.

The agent may receive thousands of tokens before reasoning even starts.

If every query repeatedly includes the same structural information, token consumption scales with query volume.

A rough model is:

Total Token Cost
≈
Requests
×
(Context Tokens + Reasoning Tokens + Output Tokens)
Enter fullscreen mode Exit fullscreen mode

Reducing repeated context has a direct effect on cost.

### 2. Latency

Agent workflows often involve multiple model or tool calls:

Retrieve
→ Classify
→ Resolve
→ Plan
→ Generate
→ Validate
Enter fullscreen mode Exit fullscreen mode

Even if each step takes only a small amount of time, the total latency accumulates.

If a trusted metric definition can be returned from a deterministic service in milliseconds, there is little value in asking an LLM to infer it again.

### 3. Inconsistency

Repeated reasoning also introduces variability.

One request may produce:

Revenue → invoice_amount
Enter fullscreen mode Exit fullscreen mode

Another:

Revenue → recognized_revenue
Enter fullscreen mode Exit fullscreen mode

Another:

Revenue → order_amount
Enter fullscreen mode Exit fullscreen mode

The model may be behaving reasonably in all three cases.

The architecture is simply asking it to repeatedly solve an ambiguous problem.

Govern the definition once and retrieve it consistently.

## Schema Retrieval Is Necessary, but It Is Not Enough

A common NL2SQL architecture is:

Question
   ↓
Embedding Search
   ↓
Relevant Tables
   ↓
LLM
   ↓
SQL
Enter fullscreen mode Exit fullscreen mode

This is much better than sending the entire warehouse schema.

But table retrieval still leaves unresolved questions:

Which metric is authoritative?

Which entity is canonical?

Which relationship is trusted?

Which date field should be used?

Which join path is safe?
Enter fullscreen mode Exit fullscreen mode

Retrieval that returns only schema is still forcing the LLM to reconstruct business knowledge.

A richer query context should return something closer to:

question: "revenue by customer last quarter"

metric:
  name: revenue
  definition: recognized_revenue
  aggregation: SUM

dimensions:
  - customer

time:
  field: recognition_date

tables:
  - customer
  - sales_order
  - finance_revenue

trusted_path:
  - customer
  - sales_order
  - finance_revenue
Enter fullscreen mode Exit fullscreen mode

Now the LLM is not discovering the data model.

It is using it.

## Precompute Relationship Knowledge

Relationship inference is a major source of unnecessary query-time reasoning.

Suppose the agent needs to connect:

Customer
Enter fullscreen mode Exit fullscreen mode

to:

Payment
Enter fullscreen mode Exit fullscreen mode

It may discover several candidate paths:

Customer → Order → Invoice → Payment

Customer → Account → Payment

Customer → Contract → Invoice → Payment
Enter fullscreen mode Exit fullscreen mode

If the organization has already validated the first path for this analytical scenario, there is no reason to compare all three paths again.

Relationship knowledge can be discovered ahead of time using evidence such as:

Primary / Foreign Keys
Column Naming
Value Overlap
Uniqueness
Inclusion Ratio
Historical Query Patterns
Business Validation
Enter fullscreen mode Exit fullscreen mode

For example:

A = order.customer_id
B = customer.customer_id
Enter fullscreen mode Exit fullscreen mode

A useful signal is:

Inclusion(A → B)
=
|distinct(A) ∩ distinct(B)|
---------------------------
|distinct(A)|
Enter fullscreen mode Exit fullscreen mode

Relationship candidates can then move through:

Discovered
    ↓
Candidate
    ↓
Validated
    ↓
Trusted
Enter fullscreen mode Exit fullscreen mode

Query-time agents should preferentially retrieve the trusted result.

## Move Metric Resolution Out of the Prompt

Metric definitions are another common source of repeated reasoning.

Instead of embedding this in every system prompt:

When the user says revenue, use recognized_amount
from finance_revenue unless...
Enter fullscreen mode Exit fullscreen mode

maintain a governed metric object:

{
  "metric": "revenue",
  "version": "2.1",
  "aggregation": "SUM",
  "table": "finance_revenue",
  "column": "recognized_amount",
  "time_field": "recognition_date",
  "status": "active"
}
Enter fullscreen mode Exit fullscreen mode

The agent calls:

get_metric_definition("revenue")
Enter fullscreen mode Exit fullscreen mode

and receives the current definition.

This has several advantages:

  • one definition across agents;
  • easier versioning;
  • easier auditing;
  • less prompt complexity;
  • fewer tokens;
  • less ambiguity.

## Build Query Context Before Calling the LLM

A useful architecture is:

                User Question
                     │
                     ▼
          ┌────────────────────┐
          │ Context Resolver   │
          ├────────────────────┤
          │ Business Terms     │
          │ Metrics            │
          │ Metadata           │
          │ Relationships      │
          │ Trusted Paths      │
          └─────────┬──────────┘
                    │
                    ▼
             Trusted Context
                    │
                    ▼
                  LLM
                    │
             Reason / Generate
                    │
                    ▼
                   SQL
Enter fullscreen mode Exit fullscreen mode

The LLM receives only what is relevant.

This changes the role of the model.

Before:

LLM = Data Discovery + Business Interpretation + Reasoning + SQL
Enter fullscreen mode Exit fullscreen mode

After:

Data Layer = Known Enterprise Facts

LLM = Intent + Reasoning + SQL / Analysis
Enter fullscreen mode Exit fullscreen mode

## A Simple Context Resolver

Conceptually, the resolver could work like this:

def build_query_context(question):
    concepts = resolve_business_terms(question)

    metrics = get_metric_definitions(concepts)
    entities = get_business_entities(concepts)

    metadata = get_relevant_metadata(
        metrics=metrics,
        entities=entities
    )

    relationships = get_trusted_relationships(
        metadata.tables
    )

    return {
        "metrics": metrics,
        "entities": entities,
        "metadata": metadata,
        "relationships": relationships
    }
Enter fullscreen mode Exit fullscreen mode

Then:

context = build_query_context(question)

sql = llm.generate_sql(
    question=question,
    context=context
)
Enter fullscreen mode Exit fullscreen mode

The exact implementation will vary.

The architectural point is that the model does not have to infer every layer of enterprise knowledge itself.

## Cache Stable Knowledge at the Right Level

Not all knowledge changes at the same frequency.

For example:

Table Schema
→ changes occasionally

Metric Definition
→ changes occasionally

Trusted Relationship
→ changes occasionally

User Question
→ changes every request
Enter fullscreen mode Exit fullscreen mode

This suggests different caching and refresh strategies.

A system might maintain:

Metadata Cache
Semantic Cache
Relationship Cache
Query Context Cache
Enter fullscreen mode Exit fullscreen mode

with invalidation triggered by:

Schema Change
Metric Version Change
Relationship Update
Governance Update
Enter fullscreen mode Exit fullscreen mode

This is more efficient than treating every query as a completely new reasoning problem.

## MCP Can Expose the Known Layer

For agent-based architectures, MCP can provide a clean interface to reusable enterprise knowledge.

For example:

get_metric_definition

get_business_entity

get_table_metadata

get_trusted_relationships

get_query_context
Enter fullscreen mode Exit fullscreen mode

The agent workflow becomes:

Question
   ↓
Agent
   ↓
MCP Tools
   ↓
Enterprise Data Intelligence
   ↓
Trusted Context
   ↓
LLM Reasoning
Enter fullscreen mode Exit fullscreen mode

MCP does not remove the need for semantic or relationship intelligence.

It gives agents a standardized way to access it.

## Do Not Confuse Precomputation With Hard-Coding

Moving knowledge out of query-time reasoning does not mean freezing the data model.

Enterprise knowledge changes.

Schemas evolve.

Metrics change.

Relationships change.

The reusable layer therefore needs:

Discover
   ↓
Detect Change
   ↓
Evaluate Impact
   ↓
Validate
   ↓
Version
   ↓
Publish
   ↓
Invalidate Cache
Enter fullscreen mode Exit fullscreen mode

The objective is not:

Never reason about the data model again.

It is:

Do not reason about the same established data knowledge on every request.

## Measure the Reasoning Tax

Teams can make this problem observable.

Useful metrics might include:

### Context Tokens per Query

Average tokens sent before generation
Enter fullscreen mode Exit fullscreen mode

*### Reasoning Calls per Query
*

Average LLM calls required before SQL execution
Enter fullscreen mode Exit fullscreen mode

*### Schema Candidates per Query
*

How many tables / columns must the model evaluate?
Enter fullscreen mode Exit fullscreen mode

### Relationship Resolution Rate

% of queries using prevalidated relationships
Enter fullscreen mode Exit fullscreen mode

### Metric Resolution Rate

% of business metrics resolved without LLM inference
Enter fullscreen mode Exit fullscreen mode

### Time to First SQL

Question received
→
Executable SQL generated
Enter fullscreen mode Exit fullscreen mode

These metrics can reveal whether the agent is spending most of its time solving analytical problems or reconstructing the enterprise data model.

## The Optimization Target Changes

A common optimization question is:

Which model gives the best SQL accuracy?

That still matters.

But production systems should also ask:

How much unnecessary reasoning are we forcing the model to perform?

A more capable model may hide poor architecture by successfully reasoning through large amounts of noisy context.

That does not make the architecture efficient.

The better system may be the one that gives the model less to figure out.

## Final Thoughts

AI agents are good at reasoning.

Reasoning is also probabilistic, expensive, and slower than deterministic lookup.

So use it where it adds value.

If Revenue already has a governed definition:

retrieve it.

If Customer already has a canonical entity:

retrieve it.

If a join path is already trusted:

retrieve it.

Then let the model solve the genuinely new problem:

what the user is asking and how to analyze it.

The next generation of enterprise data agents may not win by thinking harder.

They may win by knowing what they no longer need to think about.

Top comments (0)