Ontology-Guided Execution for Data Agents: The DABstep Method Based on DataSage

zeqi — Sun, 16 Aug 2026 06:52:24 +0000

Highlights

Semantics become governed assets, not prompt text. DataSage consolidates what the data is, what it means, and how analytical intent maps to the right operations into a unified ontology of selectable, executable, traceable, and governable semantic assets.
The ontology shapes both context and execution. The same assets that build the agent's context also reach the execution layer, feeding stable semantics into semantic SQL translation and a small set of reusable query patterns rather than leaving every step to free-form code.
Ontology-built context makes failures traceable. The ontology renders its nodes and typed relationships into context deterministically, so a wrong answer points to the exact node or edge, and the fix goes back into the ontology, not a prompt.
Ontology-guided execution holds up on a hard benchmark. On DABstep, the full configuration scores 93.0% overall and 93.6% on hard tasks; a layer-by-layer ablation isolating each mechanism is reported in the evaluation section.

Abstract

Reliable data agents need more than a stronger model, a longer prompt, or a larger tool list. On complex analytical tasks, an agent that produces runnable code can still bind a term to the wrong table, apply a business rule inconsistently, or aggregate over the wrong entity set. The hard part is keeping data meaning, business rules, execution assumptions, and answer format aligned across a multi-step run. DataSage, a data-analysis agent that answers natural-language questions over structured data, addresses this with ontology-guided execution: it treats reusable domain knowledge as runtime semantic assets rather than disposable prompt text, and lets those assets guide what the agent reads, how it executes, and how it corrects itself.

1. Why Reliable Data Agents Need Semantic Governance

Many data-agent demonstrations frame analysis as a simple loop: understand the question, generate a query or script, run it, and summarize the answer. The loop breaks down when the answer depends on business semantics and precise execution constraints.

A runnable query is only a partial success signal. A query can execute and still bind a term to the wrong table. A script can compute a number and still aggregate over the wrong entity set. A model can read a rule and still reimplement it inconsistently across tasks. The hard part is not generating code; it is preserving meaning across schema binding, cross-source joins, rule application, aggregation scope, and answer format, at every step of the run (Figure 1).

DABstep makes this failure mode concrete. A question may mention a merchant, a fee rule, a card scheme, an authorization characteristics indicator (ACI), or a merchant category code (MCC). Each surface term maps to a different output entity, candidate domain (the set of values a question ranges over), and aggregation policy. Some fields are stored directly; others are derived from multiple files. Some rule fields are explicit filters; others are wildcards. Small semantic mistakes produce cleanly executed but wrong answers.

DataSage treats this as a semantic governance problem. Reusable knowledge (what the data is, what it means, and how analytical intent maps to the right operations) should not remain trapped in prompt prose. It should become a set of semantic assets the runtime can select, execute, trace, and govern.

2. DABstep as a Representative Data-Agent Testbed

DABstep is valuable because it compresses several real data-agent pressures into one public evaluation setting: structured files, contextual documentation, payment-domain rules, cross-source relationships, temporal filters, counterfactual analysis, and exact answer checking.

Three pressures are especially important.

First, DABstep exposes semantic ambiguity. A question can ask for a fee, a fee ID, a merchant, an account type, an ACI, or a card scheme. The agent has to bind the natural-language intent to the right entity and answer format before any code matters.

Second, it exposes relationship and rule fragility. A fee question must first enrich each payment through the correct cross-table joins (which tables, on which keys, in which order) before any rule even applies; fee matching then depends on formulas, null and list wildcard semantics, monthly buckets, and derived transaction fields. A model can join on a key the schema never had, or miss one rule condition, and still return a plausible number.

Third, it exposes execution instability. Different questions are naturally answered in different ways (a single SQL aggregation for one, step-by-step procedural filtering for another, a multi-table join with rule logic for a third), and the question text rarely makes the reliable choice obvious. A generic tool list does not guarantee that the agent settles on the approach that reliably computes the correct answer.

DABstep is therefore a strong testbed for ontology-guided execution. It is not merely a payment benchmark; it is a compact environment where semantic grounding, execution correctness, and reviewable evidence all matter at once.

3. Ontology-Guided Execution: From Semantic Assets to Executable Operations

DataSage organizes the DABstep runtime around one principle: semantics should be runtime assets, not passive prompt text. The ontology stores reusable knowledge, the context-construction layer selects the relevant subset, ontology-guided operation turns that context into computation, and the trace layer records what happened.

The runtime is organized as five layers (Figure 2).

Task Interface receives the question, the available data, the supporting documentation, the applicable rules, and the required output format.
Ontology Layer holds the reusable semantic assets in three tiers: structure (schema and relationships), meaning (domain semantics and rule contracts), and operation (execution capabilities and task profiles).
Semantic Context Construction selects the task-relevant assets and renders them into compact context for the agent.
Ontology-Guided Operation generates and executes code over that context; the SQL backend translates the agent's declarative semantic SQL into executable physical SQL.
Trace and Refinement records what each run assembled, executed, and produced, so a failure can be attributed to a specific asset and fixed in the ontology.

3.1 Ontology Layer

The ontology holds six types of assets across three tiers: structure, meaning, and operation (Figure 3).

Data schema assets describe tables, fields, types, value ranges, and derived fields. Relationship assets describe the join paths and keys connecting payments to merchant metadata, merchant category codes, and acquirer-country information, so the agent grounds joins on declared keys instead of guessing them. Domain semantic assets define concepts such as ACI, MCC, account type, card scheme, fee ID, and candidate domain. Rule and contract assets encode fee formulas, wildcard and rule-matching semantics, bucket definitions, and answer formats. Execution capability assets describe real runtime boundaries such as SQL access, semantic SQL behavior, and query-pattern eligibility. Task profile assets map natural-language task families to input slots, output entities, aggregation policies, and preferred operations.

The ontology is not intended as a verbose manual; it provides the runtime with stable, reusable references for concepts that the model would otherwise rediscover, misread, or apply inconsistently.

Figure 4 shows how those abstract asset types are materialized in DataSage as an executable ontology graph. The transaction fact table anchors the view, while fee rules and dimension entities remain connected through governed join and matching relationships. Field ownership, rule-sensitive attributes, measures, and reusable metrics are visible in the same semantic surface. This is not a diagram curated for one question: it is the governed runtime structure from which task-specific context and operations are selected.

3.2 Semantic Context Construction

The agent should not receive the entire ontology for every question. A merchant-impact question, a fee-ID lookup, a static ACI comparison, and a total-fee aggregation need different semantic subsets.

Semantic context construction chooses the relevant schema, relationships, rules, task profiles, and answer contracts for the current task. This makes the prompt more intentional rather than simply longer. The assembled context gives the model the right concepts at the right time: which files exist, which fields are derived, which rule semantics are stable, what the answer format requires, and which constraints the current task family imposes.

3.3 Ontology-Guided Operation

Ontology-guided execution should change what the agent does, not only what it reads. The principle is to declare each piece of stable semantics once, in the ontology, and apply it through a translation pipeline at execution time, so the agent operates at the level of intent rather than mechanics (Figure 5).

Concretely, the agent writes minimal, declarative semantic SQL: the base joins and the filters the question actually constrains, nothing more. Even this is not trivial: which tables to join, on which keys, and in which order is exactly where an unguided model fails. The ontology's relationship assets, selected into context, supply that grounding, so the join is correct by construction rather than guessed. The query is then translated into physical SQL through an ordered set of passes that read from the ontology: resolving concepts to physical tables and columns, wrapping each match condition the agent wrote with its wildcard-match semantics (a null or empty rule field matches all values), inlining derived fields, and converting bucketed fields such as capture delay. A final semantic check covers number provenance, units, and answer format.

This division of labor is what underpins reliable execution. The agent only expresses intent on a grounded join; the mechanical parts a generic agent tends to mishandle (the wildcard guards, the bucket conversions, the exact fee formula) are produced deterministically from ontology declarations rather than re-derived as error-prone hand-written SQL or pandas loops. The same declarations apply unchanged across entities, time ranges, and question families, so each task reuses grounded, translated semantics instead of rediscovering them.

The pipeline is deliberately scoped: it is a fixed set of translation passes over ontology-declared semantics, not a general planner. When a task is not covered by the ontology, the agent falls back to writing plain SQL or pandas, and the gap is recorded so it can later be promoted into an ontology asset through trace-driven refinement.

3.4 Trace-Driven Refinement

Trace-driven refinement closes the loop, but its leverage comes from the ontology layer itself. Semantics live as discrete, addressable assets (nodes, typed relationships, rules, and contracts) rather than as scattered prompt text. The runtime can therefore record exactly what each run assembled, translated, and produced, and a wrong answer can be traced to a specific asset: a missing schema relationship, a rule or contract gap, a context-selection issue, a task-profile ambiguity, or an answer-format mismatch.

This discipline makes improvement reviewable rather than anecdotal. When a failure repeats across tasks, the fix is promoted back into the ontology (a corrected relationship, rule, or contract that every later task inherits) instead of being patched as a one-off prompt instruction. The ontology makes the system observable, traceable, and iteratively improvable, while the trace supplies the evidence that drives those revisions.

4. Case Studies

Each case takes a general data-agent failure, shows how it surfaces in DABstep, and explains how the ontology's semantic assets and ontology-guided execution change the outcome.

Case 1: Total Fees as Multi-Hop Grounding and Rule Aggregation

A total-fee question appears to be a simple aggregation, yet it requires multi-hop grounding, rule applicability, wildcard semantics, bucketed monthly statistics, and a precise fee formula. The first hurdle is the join itself: a payment row carries no account type, capture delay, or MCC, so the agent must enrich it through merchant_data and then match it against fees across several dimensions. An unguided agent often joins on a key the schema does not have, picks the wrong table, or skips the enrichment step, producing a runnable query that is already wrong before any rule is applied.

For this task family, the ontology supplies the join paths and keys (payments → merchant_data → fees), the wildcard semantics, the fee formula, and the task profile. With that grounding in context, the agent can write:

-- the agent writes minimal semantic SQL; the translator rewrites it before execution
SELECT SUM(f.fixed_amount + f.rate * p.eur_amount / 10000) AS total_fee
FROM payments p
JOIN merchant_data m ON p.merchant = m.merchant          -- enrichment: account_type, capture_delay, MCC
JOIN fees f
  ON f.card_scheme = p.card_scheme
  AND list_contains(f.account_type, m.account_type)
  AND f.capture_delay = m.capture_delay                  -- further match dimensions omitted for brevity
WHERE p.merchant = '<merchant>' AND p.year = 2023

The agent declares the enrichment join, the match dimensions in plain form, and the fee formula. These are correct because the ontology supplied the tables, keys, and which dimensions matter, not because joining is easy. Before execution, the translator refines the parts that plain equality cannot express on its own: it applies wildcard-match semantics to the rule fields, converts bucketed fields such as capture delay, and inlines derived fields. Each of these rewrites draws on information the ontology maintains: which fields carry wildcard semantics, how raw values map to buckets, and how derived fields are defined. The agent expresses intent while the ontology-backed translation supplies the exact physical form.

The reusable assets are the typed join paths, the fee formula, and the wildcard-match semantics in the ontology; they apply unchanged across merchants and time ranges, so every later question reuses the same grounded, translated semantics instead of re-deriving them.

Case 2: Trace-Driven Refinement for Underspecified Dimensions

A recurring failure in rule-based analysis is over-constraint. When a task specifies only some of the dimensions that define a rule, a model tends to treat the unmentioned ones as filters rather than leaving them open, silently dropping rules that should still apply.

In DABstep this surfaces in a static ACI comparison: the task fixes card scheme, credit status, amount, and the candidate ACI, but says nothing about the remaining fee-rule dimensions. A failing trace showed SQL that added IS NULL constraints on those unmentioned dimensions, matched too few rules, and selected the wrong ACI.

With matching semantics declared in the ontology rather than scattered across prompts, the failure had a precise address: the rule that governs how transactions match fee rules. The fix was to correct that rule once, constraining only the dimensions a task specifies, leaving unmentioned dimensions open, and letting wildcard rules apply to every group they cover. The task profiles that rely on this rule reference the shared correction instead of each carrying their own.

The correction lives in one shared asset rather than in per-task prompt text, so every task that matches fee rules over this relationship inherits it at once, from static ACI comparison to account-type and merchant-category fee questions. This correction is described as a mechanism rather than measured as an isolated gain; a matched controlled rerun for a pure quantitative attribution remains future work.

Trace evidence turned a wrong answer into a durable ontology asset, completing the core loop: trace → failure attribution → semantic asset revision → validation.

5. Evaluation

We want to know not just whether ontology-guided execution helps, but which parts of it do the work, and whether the gains concentrate where the task is hardest. To separate these contributions, we evaluate DataSage on DABstep under three configurations that add the ontology's involvement one layer at a time, measuring the accuracy change introduced by each layer.

The three configurations isolate the layers of the runtime. E0 removes the ontology entirely: a capable generic agent with no semantic assets. E1 turns the ontology on and constructs task-relevant context, but leaves execution unassisted. E2 is the full system, where the runtime also guides execution and translates the agent's declarative semantic SQL into physical SQL.

Configuration	Overall	Easy	Hard
E0 — No ontology	53.8%	75.0%	49.6%
E1 — + Semantic context construction	82.1%	88.9%	80.5%
E2 — + Ontology-guided execution	93.0%	90.3%	93.6%

E1 was evaluated over three full runs; the table reports the median. E2 is reported from a representative full run (also used for leaderboard submission). E0 is a single run.

The layers compound. Constructing task-relevant context (E0 to E1) is the single largest gain — +28.3 points overall and +30.9 on hard tasks — because grounding the agent in the right schema, join paths, rules, and answer contracts stops the runnable-but-wrong analysis that dominates the baseline. Operating on those semantics at execution time (E1 to E2) adds a further +10.9 points overall and +13.1 on hard tasks, which is where DABstep's difficulty concentrates.

One clarification keeps the reading honest: the easy split is effectively saturated and noisy (72 questions). E1 and E2 easy accuracies overlap across runs (87.5–91.7% vs. 88.9–91.7%), so the method's real effect appears on hard tasks. E2 is also far more stable: hard accuracy varied by only 1.1 points across three runs (93.6–94.7%), compared with 10.6 points for E1 (70.5–81.1%), indicating that ontology-guided execution substantially reduces run-to-run variance.

6. Conclusion

Complex data-agent reliability is not only a modeling problem. It is a semantic operations problem. A reliable agent must know what the data means, which rules apply, how those semantics should be executed, and how the final answer should be checked.

DABstep provides a dense testbed for this challenge. DataSage uses ontology-guided execution to organize reusable semantics as assets, assemble task-relevant context, translate stable semantics into executable operations, and use traces to refine the system based on evidence.

The broader lesson is simple: data-agent systems should not leave semantics as incidental prompt text. They should make semantics selectable, executable, traceable, and governable.

DEV Community: zeqi