DEV Community

Arisyn
Arisyn

Posted on

When Should a Data Agent Ask a Clarifying Question?

#ai #data #llm #sql

A production Text-to-SQL agent should not generate SQL simply because it can interpret a question. It should first decide whether the business intent is clear enough to query safely.

Consider:

Show me our best customers.

A language model can parse this easily.

It can identify customers, search the schema, choose a metric, generate SQL, and return a ranked list.

The problem is not language understanding.

The problem is that best has no unique analytical meaning.

It could mean:

Revenue
Profit
Growth
Retention
Lifetime Value
Enter fullscreen mode Exit fullscreen mode

If the agent silently chooses Revenue, the generated SQL may be syntactically correct and the returned data may be completely real.

But the system has made a business decision the user never made.

This creates an important engineering requirement for production data agents:

Before generating SQL, decide whether the intent is sufficiently resolved.

## 1. Add an Intent Gate Before SQL Generation

A basic Text-to-SQL pipeline often looks like:

Question
   ↓
Schema Retrieval
   ↓
Prompt Construction
   ↓
LLM
   ↓
SQL
Enter fullscreen mode Exit fullscreen mode

For enterprise use, add an intent-resolution gate:

Question
   ↓
Intent Extraction
   ↓
Semantic Candidate Resolution
   ↓
Ambiguity Detection
   ↓
Intent Complete?
   │
   ├── YES → Data Context → SQL
   │
   └── NO  → Clarification
                  ↓
             Update Intent
                  ↓
             Re-evaluate
Enter fullscreen mode Exit fullscreen mode

Clarification is therefore not a conversational feature added after the agent fails.

It is part of query planning.

## 2. Represent Intent Explicitly

Do not let the entire interpretation live only inside the LLM prompt.

Represent the query intent as structured state.

For example:

{
  "entity": "customer",
  "metric": null,
  "metric_candidates": [
    "revenue",
    "profit",
    "growth",
    "retention"
  ],
  "dimensions": [],
  "time_range": "last_quarter",
  "scope": null,
  "requires_clarification": true
}
Enter fullscreen mode Exit fullscreen mode

After the user selects Profit:

{
  "entity": "customer",
  "metric": "profit",
  "dimensions": [],
  "time_range": "last_quarter",
  "scope": null,
  "requires_clarification": false
}
Enter fullscreen mode Exit fullscreen mode

This gives the system an inspectable state transition:

Ambiguous Intent
      ↓
User Clarification
      ↓
Resolved Intent
Enter fullscreen mode Exit fullscreen mode

Only the resolved form should move into SQL generation.

## 3. Detect Different Types of Ambiguity

Not all ambiguity is the same.

A useful implementation should detect at least four categories.

### Metric Ambiguity

Question:

Who are our best customers?

Candidates:

{
  "term": "best",
  "type": "metric",
  "candidates": [
    "revenue",
    "profit",
    "growth",
    "retention"
  ]
}
Enter fullscreen mode Exit fullscreen mode

### Dimension Ambiguity

Question:

Show sales by region.

Possible dimensions:

customer_region
sales_region
billing_region
delivery_region
Enter fullscreen mode Exit fullscreen mode

### Time Ambiguity

Question:

Show recent revenue.

Possible interpretations:

last_7_days
last_30_days
current_month
current_quarter
Enter fullscreen mode Exit fullscreen mode

### Scope Ambiguity

Question:

How many active customers do we have?

Possible business definitions:

purchased_in_last_30_days
logged_in_last_30_days
active_contract
non_churned
Enter fullscreen mode Exit fullscreen mode

These ambiguities are dangerous because each candidate may produce perfectly executable SQL.

## 4. Candidate Count Is Not Enough

A naive ambiguity detector might say:

if len(candidates) > 1:
    clarify()
Enter fullscreen mode Exit fullscreen mode

That creates too many questions.

Suppose Revenue maps to three technical fields, but one is the governed enterprise metric and the others are deprecated or non-authoritative.

There may be multiple candidates, but no meaningful ambiguity.

A better decision considers:

Candidate Similarity
Governed Definition
Authority
Business Impact
Default Availability
User / Workspace Context
Enter fullscreen mode Exit fullscreen mode

For example:

def resolve_concept(term, context):
    candidates = semantic_search(term)

    governed = [
        c for c in candidates
        if c.is_governed and c.is_active
    ]

    if len(governed) == 1:
        return Resolution(
            value=governed[0],
            requires_clarification=False
        )

    return evaluate_ambiguity(
        candidates=candidates,
        context=context
    )
Enter fullscreen mode Exit fullscreen mode

The goal is not to ask whenever the model sees alternatives.

The goal is to ask when multiple materially different interpretations remain valid.

## 5. Score Clarification Need

A useful conceptual model is:

Clarification Need
=
Ambiguity
×
Business Impact
Enter fullscreen mode Exit fullscreen mode

You can extend this for implementation:

Clarification Score
=
Ambiguity Score
× Business Impact
× (1 - Default Confidence)
× (1 - Governance Confidence)
Enter fullscreen mode Exit fullscreen mode

For example:

def clarification_score(concept):
    return (
        concept.ambiguity_score
        * concept.business_impact
        * (1 - concept.default_confidence)
        * (1 - concept.governance_confidence)
    )
Enter fullscreen mode Exit fullscreen mode

Then:

if clarification_score(concept) > THRESHOLD:
    request_clarification(concept)
Enter fullscreen mode Exit fullscreen mode

The exact formula is domain-specific.

The important architectural point is:

Clarification should be a policy decision, not an LLM reflex.

## 6. Estimate Business Impact

Ambiguity only matters if different interpretations can materially change the result.

Consider:

Top 10 customers.

Whether ties return 10 or 11 rows is usually low impact.

Now consider:

Most profitable customers.

If the enterprise has:

Gross Profit
Contribution Profit
Operating Profit
Enter fullscreen mode Exit fullscreen mode

the choice can completely change the ranking.

A practical impact model might use:

LOW
Formatting, display, minor ranking behavior

MEDIUM
Time defaults, optional filters, non-critical dimensions

HIGH
Metric definition, financial scope, entity identity,
relationship path, aggregation grain
Enter fullscreen mode Exit fullscreen mode

High-impact ambiguity should have a lower clarification threshold.

## 7. Resolve What the Enterprise Already Knows

A good clarification engine should ask fewer questions as enterprise knowledge improves.

Suppose the semantic layer already contains:

term: revenue

resolved_metric:
  id: recognized_revenue

status: active
owner: finance
Enter fullscreen mode Exit fullscreen mode

Then:

Show revenue by region last quarter.

should not trigger:

What do you mean by revenue?

The enterprise has already answered that question.

The resolution order should look like:

User Input
   ↓
Session Context
   ↓
Governed Business Definition
   ↓
Workspace Default
   ↓
Candidate Meanings
   ↓
Clarify Only If Still Unresolved
Enter fullscreen mode Exit fullscreen mode

This is critical for usability.

## 8. Generate Clarifications From Known Candidates

Avoid generic questions such as:

Could you clarify what you mean?

The agent already has candidate meanings.

Use them.

Instead:

How should “best customers” be ranked?

[ Revenue ] [ Profit ] [ Growth ] [ Retention ]
Enter fullscreen mode Exit fullscreen mode

Or:

Which region should I use?

[ Customer Region ]
[ Sales Region ]
[ Billing Region ]
[ Delivery Region ]
Enter fullscreen mode Exit fullscreen mode

This transforms clarification from an open-ended conversation into a constrained resolution step.

## 9. Ask the Highest-Impact Question First

A single user question can contain multiple ambiguities.

Example:

Show our best customers in Europe recently.

Potential ambiguity:

best      → Revenue / Profit / Growth
Europe    → Customer / Sales / Billing Region
recently  → 7 / 30 / 90 days
Enter fullscreen mode Exit fullscreen mode

Do not immediately ask three questions.

Rank unresolved concepts:

ambiguities = [
    Ambiguity("best", impact=0.95),
    Ambiguity("Europe", impact=0.35),
    Ambiguity("recently", impact=0.40)
]

ambiguities.sort(
    key=lambda x: x.clarification_score,
    reverse=True
)
Enter fullscreen mode Exit fullscreen mode

Ask:

How should “best” be measured?

After resolving it, re-evaluate the remaining intent.

The other ambiguities may already be resolvable through workspace defaults or governed semantics.

## 10. Use a Clarification Budget

Every clarification has a UX cost.

A useful runtime configuration could be:

clarification:
  max_rounds: 2
  max_questions_per_round: 1
  minimum_impact: 0.5
Enter fullscreen mode Exit fullscreen mode

This forces the system to prioritize.

If the intent remains unresolved after the budget is exhausted, the system can:

1. Show the assumption explicitly
2. Ask the user to choose whether to proceed
3. Refuse high-risk execution
Enter fullscreen mode Exit fullscreen mode

For example:

I can proceed using Revenue as the ranking metric and the last 30 days as “recent.” Continue?

This is better than silently guessing.

## 11. Define Intent Completeness

The agent needs a deterministic stop condition.

For a typical analytical query, required slots might include:

required_intent:
  metric: true
  dimensions: optional
  time_range: true
  filters: optional
  entity_scope: true
Enter fullscreen mode Exit fullscreen mode

Then compute:

def intent_complete(intent, requirements):
    for slot, required in requirements.items():
        if required and intent.get(slot) is None:
            return False

    return True
Enter fullscreen mode Exit fullscreen mode

A more advanced version can include confidence:

Metric       Revenue          0.98
Dimension    Customer         0.99
Time         Last Quarter     0.96
Scope        Enterprise       0.93
Enter fullscreen mode Exit fullscreen mode

SQL generation begins only after the required intent reaches the accepted state.

## 12. Separate Semantic Confidence From Model Confidence

This distinction is important.

An LLM may be:

95% confident
Enter fullscreen mode Exit fullscreen mode

that “best customers” probably means highest revenue.

That does not mean the enterprise has defined it that way.

So distinguish:

Model Confidence
Enter fullscreen mode Exit fullscreen mode

from:

Semantic Resolution Confidence
Enter fullscreen mode Exit fullscreen mode

Semantic confidence should come from evidence such as:

Governed Metric Match
Business Glossary Match
Workspace Configuration
Prior Clarification
User Selection
Enter fullscreen mode Exit fullscreen mode

Do not treat model certainty as proof of business intent.

## 13. Preserve Clarification State Across Turns

Clarification should not restart the entire reasoning process.

Maintain state:

{
  "query_id": "q_1042",

  "original_question":
    "Show our best customers last quarter",

  "resolved": {
    "entity": "customer",
    "time_range": "last_quarter"
  },

  "unresolved": {
    "metric": [
      "revenue",
      "profit",
      "growth"
    ]
  }
}
Enter fullscreen mode Exit fullscreen mode

User:

Profit.

Update only the unresolved slot:

{
  "resolved": {
    "entity": "customer",
    "metric": "profit",
    "time_range": "last_quarter"
  },

  "unresolved": {}
}
Enter fullscreen mode Exit fullscreen mode

Then continue query planning.

This keeps the interaction efficient and auditable.

## 14. Feed Resolved Intent Into Data Context Resolution

Clarification should narrow the downstream search space.

Before clarification:

best customers
Enter fullscreen mode Exit fullscreen mode

may require retrieving context for:

Revenue
Profit
Growth
Retention
Customer
Enter fullscreen mode Exit fullscreen mode

After:

best = Profit
Enter fullscreen mode Exit fullscreen mode

the system can focus on:

Profit Metric
Customer Dimension
Required Data Sources
Required Relationships
Last Quarter
Enter fullscreen mode Exit fullscreen mode

So clarification does more than improve UX.

It reduces context size and downstream reasoning complexity.

## 15. Validate SQL Against Resolved Intent

After SQL generation, compare the query against the intent state.

Suppose the user clarified:

best = Profit
Enter fullscreen mode Exit fullscreen mode

but generated SQL ranks:

ORDER BY revenue DESC
Enter fullscreen mode Exit fullscreen mode

That should fail validation.

Conceptually:

def validate_query(sql_plan, intent):
    violations = []

    if sql_plan.metric != intent.metric:
        violations.append({
            "type": "metric_mismatch",
            "expected": intent.metric,
            "actual": sql_plan.metric
        })

    return violations
Enter fullscreen mode Exit fullscreen mode

Clarification therefore creates a stronger validation target.

## 16. Log Why the Agent Asked

For production systems, store the clarification decision:

{
  "concept": "best",
  "reason": "multiple_valid_metrics",
  "candidates": [
    "revenue",
    "profit",
    "growth",
    "retention"
  ],
  "ambiguity_score": 0.92,
  "business_impact": 0.95,
  "decision": "clarify"
}
Enter fullscreen mode Exit fullscreen mode

This helps answer:

Why did the agent ask this question?
Why did it not ask another one?
Which ambiguities cause the most friction?
Enter fullscreen mode Exit fullscreen mode

It also makes clarification behavior measurable.

## 17. Measure the Clarification System

Useful production metrics include:

### Clarification Rate

% of user queries requiring at least one clarification
Enter fullscreen mode Exit fullscreen mode

### Average Clarification Rounds

Mean number of follow-up turns before query generation
Enter fullscreen mode Exit fullscreen mode

### Resolution Rate

% of ambiguous queries successfully resolved
Enter fullscreen mode Exit fullscreen mode

### Abandonment Rate

% of users leaving during clarification
Enter fullscreen mode Exit fullscreen mode

### Post-Clarification Accuracy

Compare:

SQL / answer accuracy before clarification
vs.
after clarification
Enter fullscreen mode Exit fullscreen mode

### Unnecessary Clarification Rate

How often users select the obvious/default interpretation.

This is important because too many questions can be as harmful as too few.

## 18. A Practical Clarification Runtime

Putting everything together:

User Question
      ↓
Intent Parser
      ↓
Semantic Resolver
      ↓
Candidate Meanings
      ↓
Ambiguity Detector
      ↓
Impact + Governance Evaluation
      ↓
Clarification Needed?
      │
      ├── NO
      │    ↓
      │  Resolved Intent
      │
      └── YES
           ↓
      Ranked Clarification
           ↓
      User Selection
           ↓
      Update Intent State
           ↓
      Re-evaluate
           ↓
      Resolved Intent
           ↓
Data / Relationship Context
           ↓
SQL Generation
           ↓
Intent Validation
           ↓
Execution
Enter fullscreen mode Exit fullscreen mode

The LLM remains important.

But the decision to query is no longer left entirely to free-form generation.

## 19. The Engineering Principle

The goal is not to make the agent less autonomous.

It is to define where autonomy is appropriate.

If the enterprise has already defined the answer, use the definition.

If a safe default exists, use it.

If ambiguity is low impact, proceed.

But when multiple valid business interpretations remain and the choice materially changes the answer:

Give the decision back to the user.

That is not a weakness.

It is good query planning.

## Final Thoughts

A production data agent should not optimize for:

Always generate SQL.
Enter fullscreen mode Exit fullscreen mode

It should optimize for:

Generate SQL when intent is sufficiently resolved.
Enter fullscreen mode Exit fullscreen mode

That requires:

Structured Intent
Semantic Candidates
Ambiguity Detection
Business-Impact Scoring
Governed Defaults
Clarification Budget
Intent Completeness
Post-Generation Validation
Enter fullscreen mode Exit fullscreen mode

The most dangerous enterprise data errors often happen when the model makes a reasonable assumption that nobody explicitly approved.

So before improving the SQL generator again, add one question to the pipeline:

Do we actually know what the user means?

If not, the correct next step is not SQL.

It is clarification.

Top comments (0)