DEV Community: Yohei Onishi

Beyond the Model: 9 Practical Engineering Hurdles in Real-Robot Imitation Learning (SO-101 + ACT)

Yohei Onishi — Wed, 04 Nov 2026 14:12:32 +0000

Beyond the Model: 9 Practical Engineering Hurdles in Real-Robot Imitation Learning (SO-101 + ACT)

Introduction

I built this project as the final assignment for Physical AI Applied 1 at the UTokyo Matsuo-Iwasawa Lab. The course has finished and the assignment is submitted. The task was to use imitation learning on a small desktop robot arm to pick a sponge from a specified position with the gripper (the end effector that grabs objects), carry it onto a target sheet of paper, and set it down still. The deliverable was a working demo on the real robot.

I did not stop at simulation. I recorded datasets by teleoperating the real arm, trained a policy, and evaluated it on the real robot. This article records the problems I hit between bringing up the hardware and getting autonomous motion, and how I fixed them. The same write-up is also available in Japanese on Zenn.

My first live evaluation on the real robot scored 0/16 (0%). The arm trembled and stopped short of the target, or the gripper closed on nothing, again and again. It was a complete failure. The fixes were not only digital ones such as stricter simulation criteria and better model selection. I also changed the physical setup: a masking-tape frame on the desk, and interval cooling for the servo motors. After that, the final evaluation reached 3/3 successful grasp-and-place runs on the L2 evaluation set (verification set L2). With n=3, that is a reference point, not a benchmark. Qualitatively, the tremor was gone, and the arm approached the sponge smoothly and placed it still on the paper.

Task and prerequisites

The task was pick-and-place: lift a sponge from the desk with the gripper, carry it onto a target sheet of paper, and place it gently. Getting stable autonomous motion on a physical device meant working with an ecosystem where hardware, machine learning, and a physics simulator all interact.

Stack

SO-ARM101: a low-cost open-source 6-DoF desktop arm built from 3D-printed parts and serial bus servos.
Leader/Follower pair: a person moves the leader arm by hand to demonstrate the motion, and the follower arm copies it in real time (see the LeRobot teleoperation docs).
Imitation learning: supervised learning of an autonomous policy from time series of camera images and joint angles recorded during teleoperation.
LeRobot: the open-source framework from Hugging Face for training, inference, and dataset management in robotics.
ACT (Action Chunking with Transformers): a Transformer policy that predicts several future steps (an action chunk) at once from past observations, which gives smooth trajectories.
MuJoCo: a fast physics simulator with accurate contact computation, widely used in robotics and reinforcement learning.
Cameras: a Logicool C920 as the top camera, and the follower's gripper camera as the wrist camera.
Training: a MacBook Pro (M4) with PyTorch MPS, 25,000 steps, about 3.5 to 4 hours. No cloud GPU.

Public resources

All scripts and docs are public:

Source code: yohei1126/lerobot-so101-pick-and-place
Docs: project documentation
Demo video: YouTube

Full path to a working demo

Reaching a working demo took a loop of operations from hardware wiring to evaluation and improvement, not only model training.

To speed up a robotics development cycle, the RobOps improvement flywheel in the figure has to keep turning: physical setup of the environment and jig, then data collection and quality control, then local training and model selection, then real-robot evaluation and problem finding. I ran that loop and removed one source of friction at a time.

I went through nine stages:

Hardware wiring and power: connect the leader and follower serial links to the PC, and supply 12V power safely to avoid under-powered servos and port mix-ups.
Desk layout standardization: measure the arm's range, camera framing, and object positions, and lay masking tape on the work mat to unify the physical coordinate system.
Preflight (ports + cameras) and framing checks: run a script that verifies serial ports, camera devices, and framing before teleoperation.
Teleoperation recording: record grasp-and-place demonstrations with the leader arm, with a cooling routine to limit servo heat and tracking lag.
Data QC: inspect plots of joint angles and camera video, and remove outlier trajectories and unneeded idle time before training.
Policy training (Mac MPS / ACT): train ACT on an Apple Silicon Mac without renting a cloud GPU.
Simulation evaluation: check the trained policy in MuJoCo with contact-based judging to narrow the gap to the real robot.
Checkpoint selection: choose weights using real-robot behavior and reach, not only validation loss.
Real-robot autonomous evaluation: run autonomous tests in the standardized setup, find the causes of tremor and missed grasps, and feed them back into the next loop.

Problems and fixes by stage

For each stage, I describe what happened, what I changed, and the result.

1. Hardware wiring and power

I swapped the leader and follower serial ports, and I mixed up the 5V and 12V supplies. The control board LED lit up on USB bus power even when the 12V DC jack was not connected, so it looked powered when it was not. That false cue made it easy to believe the power connection was complete.

I plugged in one USB serial adapter at a time to fix the port names, and I labeled the physical cables. I turned the power-up sequence into a checklist and confirmed the 12V DC plug by eye and by pointing. The preflight script no longer guesses ports. It only checks whether the designated fixed ports connect.

That prevented startup communication errors from swapped ports and removed the servo errors caused by missing power.

2. Desk layout standardization

At the end of a motion, an arm joint touched the leg of the stand as the follower approached the target. Early in the project, I also changed the sponge and paper target dimensions several times, so the datasets no longer shared a coordinate system.

I redesigned the desk layout to keep clear of the arm's range of motion. I fixed one spec as the single source of truth: 140 mm from the arm base to the sponge center, and 70 mm from the arm base to the paper center. I laid masking tape on the work mat to mark a frame, which acted as a physical jig. The final layout series was layout39.

With the work area fixed to one set of dimensions, the contact between the arm and the stand stopped, and I could record reproducible data in a single coordinate system.

3. Preflight (ports + cameras) and framing checks

The device indices of the top camera (Logicool C920) and the wrist camera (on the follower's gripper) swapped often after OS reboots and USB replugs. On a Mac, Continuity Camera with an iPhone could also show up as a preferred device unintentionally and shift the camera indices.

I disabled the automatic phone camera handoff in the OS settings. I fixed the top camera 16 cm from the right front edge of the desk so the framing stayed constant. I also wrote a preflight script that checks the connection state of each USB serial port and the consistency of the camera indices. If an unexpected camera appears or an index does not match, it returns FAIL immediately and stops.

I run the preflight script before every session and do not start teleoperation or recording until it prints RESULT: PASS. That stopped me from throwing away data collected with a mismatched camera view. Voltage stayed on a visual pre-work checklist, because the script does not check it.

4. Teleoperation recording

After about 20 episodes in a row, the follower's shoulder_lift joint lagged its command by 10 to 12° instead of about 1.5°. I suspected that servo heat reduced the torque. The waiting time before each episode and extra motion after it also varied from episode to episode, which made the trajectories uneven.

I paused about every 10 episodes to let the servos cool down. I also standardized the procedure from the start of the leader arm motion to the end of grasp-and-place, and I fixed the wait times before and after each episode to one protocol.

Regular cooling kept the joint tracking lag from growing, and the demonstrations stayed consistent across the whole session.

5. Data QC

Recording sessions that I ran with nohup to keep them in the background ended abnormally. A relative path in the working directory also once made me append new episodes to an old dataset by mistake.

I moved recording to the foreground in an interactive terminal and stopped using nohup. Scripts now always use absolute paths for the dataset, and they stop if the target directory is missing or the metadata does not match.

I also built a QC tool that plots joint angles and camera video of recorded episodes to find outliers. Only data that passed QC went into training.

6. Policy training (Mac MPS / ACT)

My development machine was a MacBook Pro (M4), and I had no machine with an NVIDIA GPU. To avoid renting cloud GPU instances and uploading large video datasets each time, I kept recording, training, and checking on one Mac. I trained ACT with the PyTorch MPS (Metal Performance Shaders) backend.

The 25,000-step run took about 3.5 to 4 hours. macOS ships Bash 3.2 as the default shell, and its syntax limits broke parts of my scripts, such as associative arrays. I adjusted the scripts so they ran on the stock environment.

As a result, I could run the whole training loop on the local Mac without depending on the cloud.

7. Simulation evaluation

In MuJoCo, a box-snap shortcut (a simple rule that treats the object as grasped once it comes within a certain distance, like a magnet) counted trajectories as successes even when the real robot would not have grasped the object. That produced false-positive successes. The simulation also judged success by position within 25 mm and angle within 15°, while my real test was visual: grasp the sponge and set it down in the target area without dropping it. The two criteria did not match.

I removed box-snap from the simulation and judged success only by physical contact. I also revisited Temporal Ensembling (TE), which takes a weighted average of overlapping predicted action chunks to suppress sudden changes. With strong TE smoothing, plus the real arm's inertia and communication delay, the arm stalled short of the target. I adjusted the setting to suit the dynamics of the real arm.

The stricter simulation criteria narrowed the gap in behavior I saw when moving to the real robot.

8. Checkpoint selection

I loaded the checkpoint with the lowest validation loss (val_l1) onto the real arm. It trembled and did not extend far enough to reach the sponge, so the grasp never started. The model with the lowest L1 loss on validation data was not necessarily the best policy in real closed-loop control.

I stopped relying on val_l1 alone. I picked candidates between 10,000 and 25,000 steps that had a decent success rate in offline simulation, ran each on the real robot, and narrowed them down step by step by how smoothly they reached the target coordinates. In the scoring comparison, best_model (the lowest val_l1) ranked 6th among all candidates.

That gave me a selection flow that takes real-robot stability into account instead of relying on the loss value.

9. Real-robot autonomous evaluation

My first live evaluation failed 16 times in a row, a result of 0/16. The main causes were arm tremor, short approaches, and a gripper that closed on nothing.

I re-ran the evaluation with the standardized final layout and the selected layout39 checkpoints. On the L2 evaluation set, the arm completed grasp-and-place 3 out of 3 times, and it was the first successful autonomous carry. With so few trials, the number is a reference value, not a benchmark. Still, the result moved clearly from complete failure to autonomous success.

If you try the same thing

These are the lessons for anyone starting imitation learning on a desktop arm, in stage order:

Do not teleoperate or record while preflight shows FAIL. Data collected with a shifted camera index or a wrong port is wasted.
Turn off automatic phone camera handoff in the OS settings, for example Continuity Camera between a Mac and an iPhone. This prevents unintended camera priority changes.
Rest the servos about every 10 episodes. Short cooling cycles were necessary to avoid joint tracking lag from motor heat.
Avoid nohup for recording, and use absolute paths. This avoids abnormal exits from lost input and writes to the wrong directory.
Do not move hardware to match old drawings or memory. Use the taped jig, based on the measured spec, as the only reference.
Dry-run before powering up, and keep the 12V plug within reach so you can cut power at once if the arm collides or a servo runs away.
Do not discard an episode only because it has a still period at the start or end. These periods can help stabilize the policy, so if the episode follows the protocol, there is no need to delete it in a hurry.
Do not pick checkpoints by val_l1 alone. In real closed-loop control, the lowest-loss model often failed to reach the target. Check behavior on the real robot or in a physics simulator too.

Closing

On a small robot arm, standardizing the physical environment (power, ports, jig layout, heat management) mattered about as much as choosing the model.

Everything is public. Clone the repository, read the bring-up steps in the docs, try a dry-run on your desk, and watch the demo video. The hardware is described in the LeRobot SO-101 docs. The Japanese version is on Zenn.

Wrong but Real: Evaluating Intent-First Cypher Generation at 2,000 Properties (Ontographia Part 3)

Yohei Onishi — Sun, 01 Nov 2026 11:10:35 +0000

Wrong but Real: Evaluating Intent-First Cypher Generation at 2,000 Properties (Ontographia Part 3)

Series overview: Ontographia

Teams that query a graph database such as Neo4j in natural language usually choose between letting an AI write the query and locking the system to handwritten templates. This series examines the common compromise (agent skills plus template selection) and proposes another split: the AI extracts a structured Intent, and an ontology-checked, deterministic pipeline compiles the Cypher (Ontographia).

Part 1: The Graph AI Dilemma: Let LLMs Write Cypher or Suffer Template Maintenance Hell?. Production bottlenecks, research review, Intent-first hypothesis

Part 2: The LLM Never Writes Cypher: Intent-First Design and Implementation. How the split works in practice

Part 3: Evaluation and Roadmap (This article). What the measurements support, what they do not, and what comes next

Japanese edition: Zenn article

Documentation: edgesentry.github.io/ontographia

Repository: github.com/edgesentry/ontographia

Executive Summary: What This Evaluation Answers

In Part 1, I laid out why the standard compromise in graph AI—agent skills paired with template selection—hits a maintenance ceiling as schemas expand, and proposed the Intent-first hypothesis. In Part 2, I walked through the architecture that enforces this division of labor: the model extracts an Intent JSON payload, and a deterministic core validates it against an ontology and builds parameter-bound Cypher. Part 2 also adapted three major directions from recent Text-to-Cypher literature to the Intent layer: dynamic ontology subsetting, self-repair driven by execution feedback, and difficulty-adaptive resource allocation.

This final installment examines whether the Intent-first architecture holds up under stress, where it succeeds, and where its limits lie. To answer those questions, I tested the system across a synthetic manufacturing stress benchmark of up to 2,000 properties and 15 third-party public database schemas from Neo4j's official text2cypher benchmark.

Before digging into the full data, here are the five primary takeaways from the evaluation:

First, static ontology validation breaks down against silent false positives as schemas grow. When an ontology contains thousands of properties, an LLM easily picks a distractor that exists in the schema but is wrong for the question (such as picking factory_id instead of plant_id). Because the chosen property is a real, registered vocabulary item, static schema validators and AST compilers accept it without error. The query compiles cleanly, runs without failing, and returns wrong numbers.

Second, dynamic ontology subsetting structurally resolves this silent failure mode. By extracting a minimal connected subgraph of classes and relationships matched to the user's question, irrelevant distractors are physically pruned from the prompt. In deterministic simulation, this raised property accuracy by 50 percentage points and cut prompt tokens by about 91%. In live repeated evaluations using Gemini 3.8 Flash on a 2,000-property ontology, our recommended subset-with-fallback policy lifted property accuracy by an average of 19.5 percentage points over the full schema while cutting prompt tokens by roughly 90.5% to 91% (13,765 down to 1,301 p95).

Third, execution feedback repairs empty results without compromising query governance. When a query is syntactically valid but returns zero rows because of an invalid filter value (an Empty@Valid condition), feeding the empty execution result back into an Intent revision prompt achieved 100% task recovery in deterministic simulation. Crucially, the model never touches raw Cypher strings; all regenerated queries matched the deterministic compiler output exactly.

Fourth, difficulty-adaptive resource allocation drastically reduces operational total cost of ownership. For routine questions, a subset-first policy with a single initial attempt matched the quality of an always-full-schema baseline while trimming token consumption by about 91%. For challenging questions where the subset pruned necessary vocabulary, the pipeline automatically escalated to the full schema with extended retries, recovering cleanly without human intervention.

Fifth, third-party schemas integrate smoothly, but public benchmark answer quality highlights fundamental evaluation constraints. All 15 external schemas from Neo4j's text2cypher benchmark converted into Ontographia's Canonical Ontology Model, achieving 100% relationship coverage, about 96% node label coverage, and about 81% property coverage. However, an end-to-end execution spot check on live public demo databases scored an exact row-set match of roughly 3%. Because public benchmarks treat raw Cypher as the gold standard rather than Intent structures, semantically valid Cypher queries written differently from the benchmark reference are penalized as total failures.

The Evaluation Framework: Why, Value, and How

Before reviewing the numerical data, it helps to establish the experimental logic: why each test was necessary, what concrete value it delivers to operational teams, and why specific evaluation techniques were chosen.

The evaluation is split across two experimental tracks:

Track A is a synthetic stress test based on an enterprise manufacturing domain. It introduces controlled distractor properties across three scale tiers (16, 200, and 2,000 properties) to stress-test hypotheses H1, H2, and H3 under isolated conditions.

Track B is an external validity benchmark using 15 third-party database schemas published by Neo4j, spanning cinema, social networks, healthcare, and retail. It evaluates whether Ontographia's Canonical Ontology Model generalizes beyond schemas authored by its creators.

Why Evaluate? Motivations and Failure Modes

The primary motivation for this evaluation was the realization that the common belief—"constraining an LLM with an ontology guarantees query safety"—is flawed.

Static ontology validation cleanly rejects hallucinated vocabulary that does not exist in the database. But when an enterprise ontology swells to thousands of properties after years of system integrations, mergers, and ERP migrations, it fills with plausible near-duplicates. If an LLM selects an incorrect property that happens to be registered in the schema, mechanical validation passes it through. The failure is completely silent.

A second motivation was validating self-repair. When a query compiles cleanly but returns zero rows because a filter value does not exist, can an agent recover simply by correcting its structured Intent, without letting the model edit raw Cypher text directly?

A third motivation was verifying architectural generality. Any developer can build an abstraction that works on their own demo schema. Demonstrating that the pipeline functions across 15 external database models without modification was necessary to prove that the Canonical Ontology Model is not an overfitted toy.

What Value Does It Deliver? Operational Benefits

For teams running graph databases in production, these verifications address real economic and operational bottlenecks:

Substantial Cost and Latency Reductions (H1): Trimming prompt tokens by roughly 91% (p95 dropped from 13,765 to 1,301 tokens on a 2,000-property schema) dramatically lowers commercial LLM API expenses and speeds up response times.
Structural Immunity to Silent False Positives (H1): Pruning irrelevant distractor properties from the prompt physically prevents the model from choosing them.
Reconciling Governance with Task Completion (H2): Agents can self-heal from empty queries without opening the security risks and nondeterminism of raw Cypher generation.
Lowering Total Cost of Ownership Across Mixed Workloads (H3): Routine questions are answered quickly with minimal context, while complex edge cases automatically receive deeper reasoning budgets and full schema context.
Broad Domain Portability (Track B): Teams can apply the engine across diverse operational domains without custom compiler re-engineering.

How Was It Measured? Experimental Rigor and Boundaries

To ensure that results reflect real architectural properties rather than random benchmark noise, the evaluation balances synthetic isolation with real-world testing:

Why use a synthetic benchmark for Track A?
In production datasets, failure causes are heavily confounded by missing data, inconsistent terminology, ambiguous questions, and network timeouts. By synthesizing near-duplicate distractors across three distinct scale milestones (16, 200, and 2,000 properties), we can cleanly isolate exactly how schema size distorts model attention and triggers silent false positives.

Why use public benchmarks for Track B?
Testing against 15 third-party schemas from Neo4j's official text2cypher-2025v1 dataset eliminates author bias and proves that the ontology ingestion engine handles diverse graph designs.

Why use deterministic simulation arms?
In clinical trials and statistics, an "arm" refers to a distinct experimental condition group. In our evaluation, a "deterministic simulation arm" replaces the commercial LLM with fixed, predictable heuristic rules.

Commercial LLMs introduce nondeterministic variance, daily API updates, rate limits, and sampling noise. If an evaluation mixes pipeline mechanics with live LLM calls from day one, it becomes impossible to determine whether a failure was caused by a bug in the state machine or a temporary lapse in model attention.

Just as software engineers separate unit and integration tests from third-party API dependencies, using deterministic simulation arms allows us to verify the correctness of the pipeline control loops with 100% mathematical reproducibility.

However, a deterministic simulation tests the mechanics of the control pipeline, not the intelligence of a model. To verify how real models perform, we follow simulation testing with repeated live commercial LLM evaluations using Gemini 3.8 Flash.

1. Three Core Hypotheses and Pre-Registered Pass Criteria

To evaluate the architectural extensions proposed in Part 2, I established three formal hypotheses. To prevent post-hoc rationalization, strict pass, partial, and fail criteria were defined directly in the evaluation harness code before running benchmarks.

Hypothesis H1: Dynamic Ontology Subsetting

Premise: In large-scale ontologies, providing an LLM with a dynamically pruned ontology subset based on question vocabulary yields higher property extraction accuracy and drastically lower token usage than providing the full schema.
Practical Value: Drastically reduces API costs and response latency while structurally blocking distractor properties from model context.
Implementation: Dynamic ontology subsetting via graph traversal and lexical matching, paired with an optional fallback to the full schema.
Pre-Registered Criteria:
- Supported: On the 2,000-property ontology, Property Hit rate exceeds the full schema by at least 10 percentage points, prompt token consumption at p95 drops by at least 30%, and compile success rate does not degrade.
- Partially Supported: Prompt token consumption drops by at least 30%, but property extraction accuracy remains roughly identical.
- Not Supported: Subsetting strips necessary vocabulary, causing compile rates or property accuracy to drop significantly below the full schema baseline.

Hypothesis H2: Intent Repair via Execution Feedback

Premise: When an Intent compiles cleanly but returns zero rows or encounters runtime execution errors, feeding execution feedback into an Intent revision prompt allows the agent to self-heal and improve final task completion.
Practical Value: Preserves the deterministic AST compiler invariant—the LLM never writes or edits Cypher—while allowing agents to recover from empty database responses.
Implementation: Execution feedback loop enabled via the --refine-on-exec flag.
Pre-Registered Criteria:
- Supported: Across a zero-result test suite, enabling execution feedback improves final execution success by at least 20 percentage points, recovers Intent exact match, and ensures 100% of emitted Cypher queries match the deterministic compiler output.
- Partially Supported: Final execution success improves, but the gain is under 20 percentage points.
- Not Supported: Execution feedback fails to improve final execution success, or introduces invalid query syntax.

Hypothesis H3: Difficulty-Adaptive Cost Allocation

Premise: Routing simple questions to an ontology subset with minimal retries, while reserving the full schema and extended retries for difficult questions, preserves answer quality while slashing overall token spend.
Practical Value: Minimizes enterprise operational TCO by tailoring compute budgets to query complexity.
Implementation: Adaptive schema policy (--schema-policy auto) with initial and escalated retry ceilings.
Pre-Registered Criteria:
- Supported: On the 2,000-property ontology, prompt token consumption at p95 on easy questions drops by at least 30% compared to an always-full baseline while maintaining identical LLM call counts, compile rates, and property accuracy, while hard questions automatically escalate to the full schema and recover.
- Partially Supported: Token consumption drops, but accuracy exhibits instability across query subsets.
- Not Supported: Adaptive allocation degrades query accuracy or fails to reduce token consumption.

2. Track A: Reproducing Silent False Positives at Scale

The Experimental Setup

In enterprise knowledge graphs, property namespaces frequently become polluted. A manufacturing database might simultaneously contain plant_id, plant_code, factory_id, facility_name, and plant_name.

To measure how schema growth impacts model behavior, Track A uses a manufacturing ontology evaluated against 40 benchmark questions, each associated with an authoritative Gold Intent. We synthesized three scale tiers by systematically injecting distractors—near-duplicate property names and redundant entity types:

Small: 16 properties (clean, minimal schema without artificial distractors).
Mid: 200 properties (moderate schema containing realistic enterprise near-duplicates).
Large: 2,000 properties (dense, large-scale enterprise schema heavily populated with distractors).

The test suite evaluates what happens when an Intent contains an incorrect property that nonetheless exists within the ontology.

Baseline Results on the Full Schema

Running the baseline pipeline across all three schema tiers produced the following results:

Small Schema (16 properties):
- Gold Intent compile rate: 40 of 40 (100%).
- Silent false positive compile rate: 0 of 40 (0%).
- Silent false positives detected as unknown vocabulary: 40 of 40 (100%).
- Full schema prompt tokens: 1,359 tokens.
- Subset schema prompt tokens: 1,246 tokens.
Mid Schema (200 properties):
- Gold Intent compile rate: 40 of 40 (100%).
- Silent false positive compile rate: 40 of 40 (100%).
- Silent false positives detected as unknown vocabulary: 0 of 40 (0%).
- Full schema prompt tokens: 2,460 tokens.
- Subset schema prompt tokens: 1,247 tokens.
Large Schema (2,000 properties):
- Gold Intent compile rate: 40 of 40 (100%).
- Silent false positive compile rate: 40 of 40 (100%).
- Silent false positives detected as unknown vocabulary: 0 of 40 (0%).
- Full schema prompt tokens: 13,710 tokens.
- Subset schema prompt tokens: 1,247 tokens.

What This Means: The Mechanism of Silent Failure

On the clean 16-property schema, the system performed as intuitively expected: because distractor properties did not exist in the ontology, any Intent attempting to use an incorrect property was rejected by static validation as an unknown name.

However, once the schema grew to 200 and 2,000 properties, the safety net vanished. All 40 deliberately incorrect Intents sailed through ontology validation and compiled into executable Cypher (40 of 40).

This quantitatively confirms the primary failure mode of graph AI at scale: static validators and AST compilers verify lexical existence, not contextual appropriateness. If an LLM becomes confused by a wall of prompt text and selects factory_id instead of plant_id, the validator sees a registered, valid attribute and signs off on the query. The database executes it without raising an error, producing empty rows or corrupt business metrics.

Simultaneously, prompt token overhead on the full schema grew tenfold, jumping from 1,359 tokens on the small schema to 13,710 tokens on the large schema.

To reproduce this baseline experiment locally:

uv run python examples/llm/eval/run_track_a.py --record

3. Hypothesis H1 in Deterministic Simulation

Pruning the Vocabulary Space

To combat both silent false positives and prompt bloat, Ontographia implements dynamic ontology subsetting. When a user asks a question, the engine scans the text for vocabulary matching ontology classes and properties. It identifies the target nodes, computes the minimal connecting relationships between them, and prunes all unreferenced entities and distractor properties from the prompt.

We evaluated three schema delivery configurations on the 2,000-property ontology:

Full Schema: The entire 2,000-property ontology is serialized into the model prompt.
Standalone Subset: Only the dynamically pruned subgraph is provided.
Subset with Fallback (subset_fallback): The model receives the subset first; if Intent extraction or validation fails, the engine automatically falls back to the full schema. This is our recommended production configuration.

Simulation Results at 2,000 Properties

Testing these configurations in a deterministic simulation arm yielded the following measurements:

Full Schema:
- Property Hit rate: 0.50
- Compile success rate: 40 of 40 (100%)
- Distractor properties exposed in prompt: 40
- Induced incorrect choices: 40
- Fallback invocation rate: 0%
- Prompt tokens at p95: 13,710 tokens
Standalone Subset:
- Property Hit rate: 1.00 (+50 percentage point improvement)
- Compile success rate: 40 of 40 (100%)
- Distractor properties exposed in prompt: 0
- Induced incorrect choices: 0
- Fallback invocation rate: 0%
- Prompt tokens at p95: 1,247 tokens (roughly 91% reduction)
Subset with Fallback:
- Property Hit rate: 1.00 (+50 percentage point improvement)
- Compile success rate: 40 of 40 (100%)
- Distractor properties exposed in prompt: 0
- Induced incorrect choices: 0
- Fallback invocation rate: 0%
- Prompt tokens at p95: 1,247 tokens (roughly 91% reduction)

Simulation Verdict: Supported

The simulation arm met all pre-registered criteria for Hypothesis H1:

Property accuracy jumped from 0.50 to 1.00. Because distractors were physically absent from the prompt, the model could not be misled by them.
Prompt tokens at p95 dropped from 13,710 to 1,247, achieving a 90.9% reduction.
Compile success remained at 40 of 40.
In this clean benchmark fixture, the fallback rate remained at 0% because all necessary vocabulary was captured in the initial subset.

However, a deterministic simulation relies on a key simplifying assumption: that the model only errs when distractors are visible in its prompt. To observe how real models behave when dealing with ambiguous phrasing and vocabulary pruning, we turn to commercial LLM evaluation.

To reproduce the simulation locally:

uv run python examples/llm/eval/run_track_a_h1.py --record

4. Live Commercial LLM Remeasurement with Gemini 3.8 Flash

Experimental Protocol and Sampling Design

To measure real-world performance, we evaluated the published package (ontographia[llm]==0.1.6) against commercial LLM APIs via LiteLLM. We selected Gemini 3.8 Flash (model identifier ontographia-gemini-3.8, temperature 0).

Because commercial LLM calls are subject to sampling variance, single-run benchmarks can be deeply misleading. We designed a multi-run sampling plan across both the 200-property (mid) and 2,000-property (large) schemas:

Mid profile (200 properties): Evaluated across 3 complete runs (mid n=3).
Large profile (2,000 properties): Initial evaluations revealed noticeable trial-to-trial variance in the standalone subset condition. To obtain statistically rigorous estimates, we extended the large profile evaluation to 5 complete runs (large n=5, 3 initial runs plus 2 extensions).

Large Schema Evaluation Results (2,000 Properties, n=5)

The summary below details the mean and sample standard deviation across all 5 live runs on the 2,000-property ontology:

Full Schema:
- Property Hit rate (mean ± sd): 0.683 ± 0.069
- Compile success rate (mean ± sd): 28.4 ± 3.0 of 40
- Fallback invocation rate: 0.000
- Average LLM calls per query: 0.710
- Prompt tokens at p95: 13,765 tokens
Standalone Subset:
- Property Hit rate (mean ± sd): 0.720 ± 0.086
- Compile success rate (mean ± sd): 29.8 ± 3.7 of 40
- Fallback invocation rate: 0.000
- Average LLM calls per query: 0.745
- Prompt tokens at p95: 1,301 tokens
Subset with Fallback (Recommended):
- Property Hit rate (mean ± sd): 0.878 ± 0.065
- Compile success rate (mean ± sd): 36.4 ± 2.3 of 40
- Fallback invocation rate (mean ± sd): 0.170 ± 0.027
- Average LLM calls per query: 1.080
- Prompt tokens at p95: 1,301 tokens

Crucially, across all 5 runs and all conditions, the rate of hallucinated vocabulary ("invent-without-seeing", where the model invents terms absent from the prompt) was exactly 0. The rate of choosing near-duplicate distractors was also 0.

Mid Schema Reference Results (200 Properties, n=3)

For comparison, the 3 runs on the 200-property ontology produced consistent trends:

Full Schema:
- Property Hit rate (mean ± sd): 0.683 ± 0.031
- Compile success rate: 29.0 of 40
- Prompt tokens at p95: 2,515 tokens
Standalone Subset:
- Property Hit rate (mean ± sd): 0.717 ± 0.051
- Compile success rate: 29.7 of 40
- Prompt tokens at p95: 1,301 tokens
Subset with Fallback (Recommended):
- Property Hit rate (mean ± sd): 0.858 ± 0.038
- Compile success rate: 36.3 of 40
- Prompt tokens at p95: 1,301 tokens

Empirical Analysis and Engineering Implications

Reviewing these measurements yields four critical insights:

First, Hypothesis H1 is supported under live commercial conditions. Across every single run in both the mid (n=3) and large (n=5) configurations, the pre-registered criteria were fully satisfied. The formal verdict across all runs is Support.

Second, the recommended subset_fallback policy demonstrates clear superiority over the full schema. On the 2,000-property ontology, subset_fallback boosted Property Hit rate by an average of +19.5 percentage points (improving from 0.683 to 0.878). Across the 5 individual runs, the measured accuracy gains over the full schema were +21.25, +31.25, +22.50, +11.25, and +11.25 percentage points. Clean compile completions rose from 28.4 to 36.4 of 40. Prompt tokens at p95 plunged from 13,765 down to 1,301, representing a 90.5% reduction (roughly 91%).

Third, standalone subsetting exhibits high variance and must not be used in production. While the standalone subset achieved a higher mean accuracy than the full schema (0.720 vs 0.683), its sample standard deviation was large (sd = 0.086). In individual runs, aggressive keyword pruning stripped out necessary vocabulary, causing the standalone subset to score lower than the full schema.

Fourth, the fallback mechanism is essential for operational stability. Under subset_fallback, the engine automatically escalates to the full schema whenever the subset fails extraction or validation. This fallback triggered on 17.0% ± 2.7% of queries, requiring an average of 1.08 LLM calls per question. By using the full schema as a safety net, subset_fallback recaptured missing vocabulary and maintained a high, stable success rate (0.878 ± 0.065). In production environments, subsetting without a fallback path is an anti-pattern.

To reproduce the live evaluation harness:

uv run python examples/llm/eval/run_track_a_h1_live.py --profiles mid,large --record

5. Track B: External Validity Across 15 Third-Party Schemas

Overcoming Author Bias

Any architecture can achieve high benchmark scores on an ontology designed by its own authors. To test whether Ontographia's Canonical Ontology Model (COM) generalizes to third-party databases, Track B evaluates the engine against external benchmarks.

We sourced 15 database schemas from Neo4j's official neo4j/text2cypher-2025v1 dataset on Hugging Face. These schemas represent diverse domains, including cinema, recommendations, social networks, retail, logistics, and healthcare. We built an automated ingestion tool (schema_convert.py) to parse these external schemas and map them directly into Ontographia's COM structure.

Schema Ingestion Coverage

Across all 15 third-party databases, the ingestion tool produced the following results:

Schema conversion and Engine.load success rate: 15 of 15 (100%).
Average node label coverage: approximately 96%.
Average relationship type coverage: 100%.
Average property coverage: approximately 81%.

Assessing Ingestion Gaps

The engine ingested every single external database schema without code changes, achieving perfect coverage on relationship topologies and 96% coverage on node entity types.

The remaining 19% gap in property coverage stems from structural quirks in the source benchmark dataset rather than architectural limitations. Several schemas in text2cypher-2025v1 contain inconsistent textual property descriptions, unstructured comments, or property names embedded inside freeform prose. While automated regex parsers missed these edge cases, the underlying COM schema representation accommodates all standard graph primitives.

6. Track B Exec Match Spot Check and Honest Reading

Live Database Execution on Public Benchmarks

To evaluate whether the pipeline could answer questions on these third-party databases, we conducted an end-to-end execution spot check. We stratified 60 sample questions (seed=53) from the movies and recommendations databases, pointed the engine at Neo4j's public demo instance (demo.neo4jlabs.com), and queried the database using Gemini 3.7 Flash (gemini-3.7-flash via LiteLLM).

The pipeline executed the full cycle: Natural Language Question -> Intent JSON -> Deterministic Engine.build -> Neo4j Database Query -> Jaccard Row-Set Comparison against benchmark Gold Cypher queries.

The spot check yielded the following metrics:

Compile success rate: 41 of 60 queries compiled cleanly (approximately 68%).
Exec Match (exact row-set match against Gold Cypher): 1 of 38 executed queries (approximately 3%).
Average Jaccard similarity across row sets: 0.03.
Structural F1 scores:
- Node labels: 0.82
- Relationship types: 0.67
- Properties: 0.61

An Honest Assessment of Public Benchmark Limitations

An exact match score of roughly 3% is undeniably low. However, analyzing the failure modes reveals the structural challenges of evaluating Intent-first architectures against raw Cypher benchmarks:

Benchmark Reference Queries Are Raw Cypher: In standard Text-to-Cypher benchmarks, the gold standard is an arbitrary Cypher query written by a human. The benchmark judges correctness by running both queries and verifying whether the returned row sets match identically.
Syntactic Equivalence vs. Identity: Ontographia's deterministic AST compiler builds standardized, parameter-bound Cypher queries. For example, where a benchmark author wrote a custom nested pattern comprehension, Ontographia might generate a clean multi-step MATCH ... WHERE block with distinct aliasing. Both queries may extract valid operational data, but minor differences in column ordering, aggregation grouping, or projection limits cause the strict row-set Jaccard comparison to score 0.
Compound Pipeline Errors: The 3% figure conflates Intent extraction errors, schema conversion gaps, and alternative Cypher formulations.

This experiment proves that Ontographia's ingestion and compilation pipeline executes end-to-end on arbitrary third-party databases without crashing. It does not prove that Ontographia currently matches human-level accuracy on open-domain Cypher benchmarks. We make no claims of State-of-the-Art (SOTA) execution parity on Track B.

Generalizing Business Terminology Mappings

A key takeaway from Track B was the necessity of decoupling domain heuristics from the core engine. In early prototypes, terminology repair rules were hardcoded for manufacturing concepts. In Track B, we moved these rules into declarative YAML configuration mappings. We verified that switching from manufacturing mappings to cinema domain mappings allowed the same compiler engine to handle movies without modification.

To run the Track B execution benchmark:

uv run --with datasets --with neo4j python examples/llm/eval/run_track_b_exec.py --record

7. Hypothesis H2: Safe Self-Repair via Execution Feedback

The Empty@Valid Problem

In real enterprise operations, a query can be syntactically flawless, fully compliant with the ontology, and compile without warnings, yet still return zero records when executed against the database.

Consider a user asking for production numbers from "Plant 4." The ontology confirms that Plant nodes have a plant_code attribute, and the query compiles cleanly. But if the physical database only contains plants 1, 2, and 3, Neo4j returns an empty result set. We term this failure mode Empty@Valid.

In traditional Text-to-Cypher systems, the agent handles empty results by passing the failed Cypher query back to the LLM and asking it to rewrite the query. This reintroduces non-deterministic behavior: the model frequently hallucinates alternative relationship paths, drops required filters, or introduces syntax errors.

Ontographia takes a fundamentally different approach. The engine captures the empty execution result and feeds it back into the Intent extraction prompt, requesting a revised Intent JSON. The LLM never touches the Cypher string. The revised Intent passes through ontology validation and is compiled into Cypher by the deterministic AST engine.

Deterministic Simulation Results

To test whether this repair loop operates reliably, we constructed a test suite of 40 questions where filter values in the Gold Intents were deliberately corrupted. The queries compiled successfully, but mock database executions returned zero rows.

We evaluated two conditions in a deterministic simulation arm:

No Feedback (no_refine): The pipeline terminates immediately after the initial query returns an empty result.
With Feedback (with_refine): The empty result triggers an Intent refinement prompt, and the revised Intent is re-compiled.

The simulation produced the following measurements:

Without Feedback (no_refine):
- Initial Empty@Valid rate: 1.00 (100% of queries returned empty results)
- Final execution success rate: 0.00
- Intent exact match rate: 0.00
- Task recovery rate: 0.00
- Emitted Cypher matching Engine.build: 1.00 (100%)
With Feedback (with_refine):
- Initial Empty@Valid rate: 1.00 (100% of queries returned empty results)
- Final execution success rate: 1.00 (+100 percentage point recovery)
- Intent exact match rate: 1.00
- Task recovery rate: 1.00
- Emitted Cypher matching Engine.build: 1.00 (100%)

Simulation Verdict: Supported

The simulation supported Hypothesis H2, achieving a 100 percentage point recovery from empty execution results.

Crucially, in 100% of cases, the final Cypher queries matched the output of the deterministic AST compiler. This demonstrates that self-repair can be achieved without allowing the LLM to write or alter query text directly.

Operational Context and Caveats

This simulation confirms the integrity of the state machine and repair control loop: when provided with corrective feedback, the pipeline successfully re-parses, re-validates, and re-compiles the query.

In live production environments, the ultimate recovery rate depends on whether the LLM can infer the correct filter value from the execution error. The core value of Ontographia's approach is that even when an LLM fails to find the right filter, it cannot violate query safety or database governance while trying.

Teams can enable this self-repair loop in production via the CLI:

uv run python examples/run_llm_e2e.py \
  --question "Which plant hosts production Line-1?" \
  --execute \
  --refine-on-exec \
  --password ontographia

To reproduce the simulation locally:

uv run python examples/llm/eval/run_track_a_h2.py --record

8. Hypothesis H3: Difficulty-Adaptive Cost Allocation (Adaptive Spend)

The Cost of Uniform Resource Allocation

In enterprise conversational systems, user queries vary wildly in complexity. Asking "What is the name of Line 1?" requires minimal schema context and can be resolved in a single call. Conversely, asking for a cross-facility dependency bottleneck requires inspecting multiple relationship hops.

Defaulting to the most expensive configuration—dumping the entire 2,000-property ontology into every prompt and allowing up to 5 retries—wastes token budgets and introduces unnecessary latency. Conversely, capping all queries to minimal prompts causes difficult questions to fail.

Hypothesis H3 evaluates an adaptive resource allocation policy: start every query with an ontology subset and a single initial attempt (initial_retries=1); if validation or execution fails, automatically escalate to the full schema with extended retries (escalated_retries=5).

Simulation Setup at 2,000 Properties

We evaluated this adaptive policy against an always-full control group on the 2,000-property ontology, testing across two query slices:

Easy Slice: Routine questions where the required vocabulary is readily captured by the initial subset (representing the vast majority of day-to-day enterprise queries).
Hard Slice: Complex questions where the subset intentionally prunes required vocabulary, forcing the pipeline to escalate.

We compared two system configurations:

Always Full Fixed (always_full_fixed): Always sends the full 2,000-property ontology with a fixed ceiling of 5 retries.
Adaptive Allocation (adaptive): Starts with the subset and 1 retry; escalates to the full schema and 5 retries only upon failure.

Simulation Results

The simulation arm yielded the following metrics:

Easy Slice under always_full_fixed:
- Property Hit rate: 1.00
- Compile success rate: 40 of 40 (100%)
- Escalation rate: 0.00
- Average LLM calls: 1.00
- Prompt tokens at p95: 13,710 tokens
Easy Slice under adaptive:
- Property Hit rate: 1.00
- Compile success rate: 40 of 40 (100%)
- Escalation rate: 0.00
- Average LLM calls: 1.00
- Prompt tokens at p95: 1,247 tokens (roughly 91% reduction)
Hard Slice under always_full_fixed:
- Property Hit rate: 1.00
- Compile success rate: 40 of 40 (100%)
- Escalation rate: 0.00
- Average LLM calls: 1.00
- Prompt tokens at p95: 13,710 tokens
Hard Slice under adaptive:
- Property Hit rate: 1.00
- Compile success rate: 40 of 40 (100%)
- Escalation rate: 1.00 (100% escalation)
- Average LLM calls: 2.00
- Prompt tokens at p95: 13,710 tokens

Simulation Verdict: Supported

The simulation supported Hypothesis H3 across all criteria:

On easy questions, the adaptive policy matched the 100% accuracy of the full-schema baseline while slashing prompt tokens from 13,710 to 1,247 (a 90.9% reduction) and requiring only 1.00 model calls.
On hard questions, the engine detected the vocabulary deficit, triggered escalation on 100% of queries, and recovered complete compile success (40 of 40) at an average cost of 2.00 model calls.

Production Reality

This adaptive mechanism is already built into Ontographia as the default production mode (--schema-policy auto). Because this policy mirrors the subset_fallback behavior verified in our live commercial LLM evaluations, teams do not need a separate query classifier to benefit from adaptive spend.

To test the adaptive policy:

uv run python examples/run_llm_e2e.py \
  --question "Which plant hosts production Line-1?" \
  --schema-policy auto \
  --initial-retries 1 --escalated-retries 5

To reproduce the simulation locally:

uv run python examples/llm/eval/run_track_a_h3.py --record

9. Comprehensive Synthesis: Commercial LLMs and Real-World Generalization

Synthesizing our empirical findings across simulation arms, commercial LLM remeasurements, and public benchmarks yields a grounded assessment of the Intent-first paradigm:

Track A Live Commercial Remeasurement (Gemini 3.8 Flash via LiteLLM): The recommended subset_fallback policy achieved consistent, statistically robust support for Hypothesis H1 across both mid (n=3) and large (n=5) configurations. On a 2,000-property ontology, it lifted property accuracy by an average of +19.5 percentage points over the full schema while cutting prompt tokens by roughly 90.5% to 91%. The data also revealed that standalone subsetting suffers from high variance (sd = 0.086), proving that a full-schema fallback path is mandatory in production.
Track B Public Benchmark Execution: Ontographia's Canonical Ontology Model successfully ingested 15 third-party schemas from Neo4j's official benchmark, demonstrating strong structural generality. However, live execution against demo databases yielded an exact match rate of roughly 3%, underscoring the gap between deterministic AST generation and human-authored benchmark Cypher.
Declarative Domain Portability: Abstracting domain-specific repair heuristics into declarative YAML mapping configurations proved successful, enabling the same compiler to process manufacturing and cinema domains without code modifications.

These results indicate that Intent-first is not merely a theoretical framework. It operates effectively on commercial LLM APIs and external database schemas, delivering substantial cost reductions and structural error protection. At the same time, it highlights the need for better evaluation standards that look beyond syntactic Cypher matching.

10. Roadmap: Delivered Proofs, Scope Boundaries, and Next Steps

The Ontographia project set out to test whether natural language interfaces to graph databases could achieve deterministic auditability without drowning in template maintenance. Here is an accounting of what has been proven, what was left out of scope, and what lies ahead.

What Was Delivered and Proven

Verification of H1, H2, and H3 in Deterministic Simulation: Proven in code across synthetic benchmarks, demonstrating the structural validity of dynamic subsetting, execution-guided self-repair, and difficulty-adaptive spend.
Live Commercial LLM Remeasurement for H1: Confirmed across repeated runs with Gemini 3.8 Flash (large n=5, mid n=3), showing an average +19.5 percentage point accuracy boost and a 90.5% to 91% token cut.
Schema Generalization Across 15 Third-Party Databases: Confirmed 100% conversion and loading into the Canonical Ontology Model across Neo4j's public benchmark schemas.
Empirical Baseline for Live Execution: Established an honest ~3% Exec Match baseline on public demo databases, identifying key evaluation bottlenecks.
Declarative Terminology Mapping: Successfully decoupled domain repair logic into portable YAML schemas.

What Is Intentionally Out of Scope

SOTA Claims on Public Text-to-Cypher Benchmarks: Because public benchmarks treat arbitrary, human-written Cypher as the gold standard, evaluating a deterministic compiler against them penalizes valid semantic variations. We deliberately refrain from claiming benchmark supremacy.
Multi-Model Bake-Offs: While the architecture supports any model via LiteLLM, this evaluation focused on validating the control loops using Gemini 3.8 Flash. Cross-model comparisons against GPT-4o or Claude 3.5 Sonnet are left to future community evaluations.
Exhaustive Live Matrices for H2 and H3: The mathematical integrity of the repair and escalation state machines was established in simulation. In production, schema_policy=auto already serves as the default operational path.

Future Roadmap and Research Priorities

Intent-Level Gold Benchmarks and Semantic Equivalence Metrics: Developing benchmark datasets that evaluate graph agents based on structured Intent representations and semantic data equivalence, rather than brittle row-set equality against specific Cypher strings.
Synonym- and Typo-Aware Ontology Subsetting: Upgrading the dynamic subsetting engine from exact lexical matching to embeddings or phonetic indices capable of catching colloquial enterprise jargon and spelling variants.
Multi-Domain Production Deployments: Validating the H1–H3 control loops on active enterprise knowledge graphs and production databases across finance, supply chain, and life sciences.

The full project roadmap is maintained on the Ontographia repository roadmap.

11. Trilogy Retrospective and Getting Started

Over this three-part series, we have explored a fundamental architectural question: how do we build natural language interfaces to enterprise knowledge graphs that are flexible enough to answer ad-hoc questions, yet strict enough to deploy in production?

In Part 1, we examined the graph AI dilemma: raw LLM Cypher generation creates unacceptable security and accuracy risks, while hardcoded query templates trap engineering teams in maintenance backlogs.
In Part 2, we introduced the Intent-first architecture: dividing labor so the LLM extracts structured Intent JSON, while a deterministic, ontology-enforced compiler builds parameter-bound Cypher. We then mapped cutting-edge Text-to-Cypher research techniques directly into the Intent layer.
In Part 3, we evaluated this architecture empirically: uncovering the silent false positives that emerge when ontologies scale, proving that dynamic subsetting delivers an average +19.5 percentage point accuracy gain and cuts prompt tokens by ~91% in live commercial testing, and establishing clear operational boundaries across 15 external database schemas.

The central takeaway is clear: enterprise graph AI does not require choosing between safety and scalability. By drawing a clean boundary between semantic extraction and query compilation, teams can deploy conversational graph agents that are both cost-effective and strictly auditable.

Ontographia is open source. All code, ontology specifications, evaluation harnesses, and raw benchmark data are publicly accessible.

You can install the experimental Intent layer directly from PyPI:

pip install "ontographia[llm]"

Explore the test suites under examples/llm/eval/, review the Evaluation Guide, and test the pipeline against your own domain ontologies. Feedback, benchmark reports, and contributions are welcome on the GitHub repository.

Open source project: github.com/edgesentry/ontographia
Documentation: edgesentry.github.io/ontographia
Release notes: v0.1.6
Author: Yohei Onishi
Tags: #Neo4j #Cypher #KnowledgeGraph #GraphRAG #AIAgents #EnterpriseAI #LLMEvaluation

The LLM Never Writes Cypher: Intent-First Design and Implementation (Ontographia Part 2)

Yohei Onishi — Sun, 25 Oct 2026 08:34:54 +0000

The LLM Never Writes Cypher: Intent-First Design and Implementation (Ontographia Part 2)

Series overview: Ontographia

Teams that query a graph database such as Neo4j in natural language usually choose between letting an AI write the query and locking the system to handwritten templates. This series examines the common compromise (agent skills plus template selection) and proposes another split: the AI extracts a structured Intent, and an ontology-checked, deterministic pipeline compiles the Cypher (Ontographia).

Part 1: The Graph AI Dilemma: Let LLMs Write Cypher or Suffer Template Maintenance Hell?. Production bottlenecks, research review, Intent-first hypothesis

Part 2: The LLM Never Writes Cypher (This article). How the split works in practice

Part 3: Wrong but Real. Stress tests, public datasets, and limits of the claims

Documentation: edgesentry.github.io/ontographia

Repository: github.com/edgesentry/ontographia

Part 1 ended with a question. Can a natural-language interface follow what users actually ask, and still produce database queries that are deterministic and auditable, without engineering work growing with every new template?

Ontographia's answer is to give the AI exactly one job. It reads the question and writes a structured Intent: where to start, which relationships to follow, how to filter, and what to return. A fixed core checks every name in that Intent against the ontology and compiles it into Cypher with values bound as parameters. The AI never writes Cypher, and nothing it writes reaches the database as query text.

The published packages already follow that split. v0.1.5 started rejecting Intents with unknown fields. v0.1.6 packaged the experimental Intent layer as an installable extra and added an MCP server that builds Cypher only after validation. Later sections walk through that design and a small end-to-end run of v0.1.6 against a real Neo4j setup.

The rest of this piece stays at design level: how a question becomes a query, why the template catalog can go away, why the model and database driver stay outside the core, and how three research ideas move from "fix the query string" to "fix the Intent." Step-by-step implementation belongs in the public docs and repo, linked below.

Where Part 1 left off

In Part 1 I argued that the template-selection hybrid looks safe but hits a structural wall. Flexibility is counted in templates, so every new kind of question means another template to write, review, and regression-test. Research such as schema filtering and iterative refinement helps, but it usually repairs queries after the fact and still assumes the model writes raw Cypher.

Ontographia moves flexibility into the Intent and keeps query assembly deterministic. Only Intents that pass ontology validation are compiled. Prompting, model calls, and database execution live outside the core, in the application or agent layer. The architecture guide describes the boundary in detail.

How a question becomes a query

The LLM does not write Cypher. It writes a structured Intent; a deterministic pipeline validates that Intent against the ontology and compiles the query.

In plain terms the path is:

Load the company's ontology (YAML, Turtle, or another supported form) into an internal Canonical Ontology Model. That model is the vocabulary source of truth.
Give the model a schema derived from that vocabulary and ask for a structured Intent as output (start, traverse, filter, return). The model stops there; it does not emit Cypher.
Validate every class, relationship, and property name in the Intent. Unknown names stop here; the error goes back so the model can retry.
Compile a validated Intent into a { query, params } pair. The same ontology, Intent, and dialect always produce the same pair. Filter values travel in the parameter map, not inside the query text.
Run that pair against Neo4j in the application layer.

A manufacturing sample in the repository asks for suppliers of the parts that make up a product SKU. The Intent names the start concept, the relationships to walk, the SKU filter, and the supplier name to return. The compiled query binds the SKU as a parameter so the literal never appears in the query string. To see the exact Intent shape and emitted Cypher, use the Neo4j end-to-end walkthrough and the samples under the repository rather than treating this article as an implementation guide.

Why the template catalog can go away

In the hybrid approach, someone writes a Cypher string in advance for each pattern: one for a supplier list, another for a recursive bill-of-materials expansion, and so on.

In Ontographia a query is a combination of a small set of Intent fields (start, traverse, filter, return, plus optional ordering, paging, and an optional-match flag). Any question that stays inside the ontology's vocabulary can be expressed with those fields, so a new pattern does not need a new template.

The field set is closed. Since v0.1.5, an unknown key is a validation error that names the offending field. Before that, unknown keys were ignored, so a model that wrote filters instead of filter could build a query with the filter silently dropped. Unknown dialect names are rejected the same way instead of falling back quietly.

Why the core has no model and no database driver

The core is a compiler. It has no networking, no model API, and no Neo4j driver. Many AI toolkits put those inside the engine; Ontographia keeps them out on purpose so query assembly can be tested without tokens, without mocked model replies, and without a live database. The rule that the model never edits Cypher is enforced by that boundary, not only by convention. See the architecture guide.

Leaving prompting and execution to the application has a cost: every team would otherwise invent its own wiring. The project started with helpers as examples in the repository. In v0.1.6 they became an installable, experimental Python extra for Intent extraction and related loops. Cypher still comes only from the core, after validation.

Three research ideas, moved from Cypher to Intent

Part 1 covered schema filtering, iterative refinement, and difficulty-adaptive spend. Those lines usually assume the model writes Cypher. The Intent layer applies the same ideas one step earlier.

Schema filtering becomes ontology subsetting. Large enterprise ontologies can have hundreds or thousands of classes and properties. Putting all of them in the prompt costs tokens and makes it easier for the model to pick a property that looks right but is not. The Intent layer matches words in the question against ontology names (ignoring case and normalizing plurals), adds the minimal relationships that connect the matched classes, and sends only that subset. If extraction or validation fails on the subset, it retries with the full schema. If nothing matches, it uses the full schema from the start.

Iterative refinement becomes Intent repair. An Intent can pass validation, compile, and still return zero rows, or the driver can raise a type error. Text-to-Cypher work often hands the failed Cypher back to the model, which brings non-determinism back into the query string. Here the model gets the execution result and revises the Intent. The revised Intent goes through validation and compilation again, so every query that runs still comes from the compiler. The loop is opt-in, defaults to at most two rounds, and stops if a revised Intent fails validation.

Difficulty-adaptive spend becomes staged retries. Everyday questions are often single-node lookups or shallow walks. Extraction starts with the subset and a single retry, and only then moves to the full schema with more retries. Execution failures on Neo4j do not trigger that escalation; the repair loop handles those. The aim is lower tokens and latency on easy questions. Whether it works is a measurement question; Part 3 has the numbers.

What changes for agent skills

With Intent-first, agent skills have less to do. In the hybrid, the model classifies which template to call, and developers maintain a growing catalog. Here the model turns a question into an Intent, and developers maintain the ontology and the extraction prompt.

Schema changes show the difference most clearly. In the hybrid, renaming a property means finding every affected template, fixing it, and re-testing classification. In Ontographia you update the ontology and validation follows it. Query safety stops depending on whoever wrote each template, because names are checked and values are bound before any query string exists.

v0.1.6 in practice: MCP and a small end-to-end run

v0.1.6 also added an experimental MCP (Model Context Protocol) server with tools to list the ontology, expose the Intent schema, build Cypher from an Intent, and execute. The server does not call a model. The MCP client supplies the Intent; the server validates before building Cypher, so the model still never writes Cypher when it runs inside an MCP client. Setup is in the MCP docs.

Those tools worked against a real setup in my checks, including replaying a saved Intent through build and execute. Integration tests for execute also run in CI against a live Neo4j instance.

Separately, I ran questions end to end with Gemini 3.7 Flash and a local Neo4j. When the model put a property name that does not exist in the ontology into an Intent (for example a garbled name with a stray character between name and as_name), validation failed, no Cypher was emitted, and the bad name never reached the database.

Answering the question was a different matter. I kept all four attempts, and only one succeeded end to end; that one returned two rows, MikroMotors and FormTech. Stopping a bad name and answering correctly are separate results, and in this small sample the second rate is low.

The protection is also narrow. Validation stops names that are not in the ontology. If the model picks a real name that is the wrong one for the question, the Intent passes; Part 1 called these silent false positives. Factual errors are out of scope too. This is not a claim that Ontographia prevents hallucination in general.

Known limits

Intent-first covers questions that fit the ontology's vocabulary. A question that needs a class or relationship the ontology does not define fails validation instead of producing a query.

The project is pre-1.0, and releases can still break compatibility; v0.1.5 started rejecting Intents with unknown keys that earlier versions built. The Intent layer and the MCP server are experimental optional extras. Subsetting, adaptive retries, and execution feedback live there, not in the core compiler. GQL output is a prototype. The roadmap lists what comes next, without dates.

Next: does it hold up at scale?

Moving natural-language flexibility into the Intent and making Cypher assembly deterministic means a new kind of question no longer needs a new template. Whether that holds against ontologies with thousands of properties and public Text-to-Cypher benchmarks is a separate question. Part 3 reports a stress test at that scale and validation on public datasets, including where the approach falls short, plus a re-run on the published v0.1.6 package with two model versions.

To try it: install the core with pip install ontographia, and add the experimental Intent layer or MCP extras if you need them (ontographia[llm], ontographia[mcp]). The CLI is also on Homebrew (brew tap edgesentry/tap && brew install ontographia). The Neo4j walkthrough runs the manufacturing sample end to end. If you try it on your own ontology, I would like to hear which questions failed validation and which passed but returned the wrong thing; issues are open on the repository.

Open source project: github.com/edgesentry/ontographia
Documentation: edgesentry.github.io/ontographia
Release notes: v0.1.5, v0.1.6
Author: Yohei Onishi
Tags: #Neo4j #Cypher #KnowledgeGraph #GraphRAG #AIAgents #EnterpriseAI #MCP

The Graph AI Dilemma: Let LLMs Write Cypher or Suffer Template Maintenance Hell? (Ontographia Part 1)

Yohei Onishi — Sun, 25 Oct 2026 08:19:57 +0000

The Graph AI Dilemma: Let LLMs Write Cypher or Suffer Template Maintenance Hell? (Ontographia Part 1)

Series overview: Ontographia

Business teams want to ask a graph database (for example Neo4j) questions in ordinary language. The usual choices are risky: let an AI write the database query itself, or lock the system to a fixed list of hand-written query templates.

This three-part series looks at the common middle ground (AI picks a pre-approved template) and why it still stalls in production. It then proposes a different split: the AI only describes what the user wants to know, and a fixed, ontology-checked pipeline turns that description into a safe query (Ontographia).

Part 1: The Graph AI Dilemma (This article). Why templates become the bottleneck, what recent research helps, and the Intent-first idea

Part 2: The LLM Never Writes Cypher. How the split works in practice; step-by-step walkthroughs stay in the docs and repo

Part 3: Wrong but Real. Measurements on large ontologies and public datasets, plus what is still open

Documentation: edgesentry.github.io/ontographia

Repository: github.com/edgesentry/ontographia

Enterprise knowledge graphs (manufacturing bills of materials, multi-tier supply chains, quality and asset data) are built around relationships. Business teams increasingly want to ask those graphs questions in plain language.

Letting an AI write the database query from scratch is hard to approve in governed environments. The model can invent names that are not in the schema, produce broken queries, mix filter values into the query text in ways that are hard to audit, or trigger expensive full-graph scans that hurt a shared database.

So many teams pick a hybrid. Engineers write pre-approved, parameterized query templates, register them as agent tools, and ask the model to choose the right template and fill in the parameters. That looks like a safe compromise at first. Over months, the pain moves: less "will the model invent a bad query?" and more "we cannot keep writing, reviewing, and regression-testing templates forever."

Here is why that hybrid hits a maintenance wall, what recent Neo4j and NLP research still assumes, and a different hypothesis: keep the AI on structured intent only, and compile the actual query with a deterministic, ontology-constrained pipeline.

Three choices teams actually make

Teams usually land somewhere on this spectrum.

Let the model write the query. High flexibility. Weak guarantees. The model sees the schema (or a large dump of it) and invents a full query. Demos look great. Production risk is high: wrong or misspelled names, intermittent syntax failures, hard-to-govern string concatenation, and runaway graph scans.

Only hand-written queries. Maximum safety and auditability. Almost no natural-language agility. Every new nuance needs another endpoint. The product starts to feel like forms and dropdowns with an AI label.

Hybrid: agent skills plus templates. Engineers maintain a catalog of vetted templates. The model classifies the user's question into one of them and extracts arguments. The app binds those arguments safely and runs the query. Syntax errors drop, and tool choice is a task modern models handle well. That is why the hybrid feels like a win early on.

Why templates become the wall

In the hybrid, flexibility is counted in templates. A few examples from a supply-chain setting:

"Who is the primary supplier of Component A?" → template 1
"Expand secondary suppliers for Component A." → template 2
"List suppliers of Component A at domestic plants." → template 3
"Which SKUs break if Supplier X stops?" → template 4

Users combine filters, path depth, aggregations, and sort orders in ways catalogs never finish covering. Each unexpected phrasing becomes another engineering ticket: "please add a template."

Schemas also change. Labels rename, junction nodes appear, properties split. One schema change can mean auditing many templates, rewriting them, updating tool descriptions so the model still picks the right one, and re-running classification tests.

The hybrid does not remove the core tension. It moves friction from model risk at runtime to human template authoring and regression cost.

What recent research helps, and what it still assumes

The Neo4j and broader Text-to-query community has been working the same tension. Three lines matter for this series.

Schema filtering. Large schemas in the prompt hurt attention and raise misclassification. Approaches that prune the schema before prompting (exact match, named-entity cues, similarity) reduce that load. Key source: Ozsoy (2025), Enhancing Text2Cypher with Schema Filtering.

Iterative refinement. One-shot generation is weak for multi-hop questions. Checkers, formal verifiers, and database feedback drive a verify-and-correct loop. Key source: Neo4j Developer Blog (2025), Explore iterative refinement for Text2Cypher.

Difficulty-adaptive spend. Not every question deserves the same compute. Easy questions get a cheap pass; hard ones get more context and retries. Sources include Wang et al. (NAACL 2025 Findings) and Damani et al. (ICLR 2025).

These ideas are useful. They still share a starting assumption: the model writes the query string. Post-check loops then struggle with three structural issues.

Silent false positives. A query can be syntactically fine, match the schema, and run without error while following the wrong business path or aggregating the wrong fields. Formal checkers do not see unspoken business rules.
Filter values mixed into query text. That complicates strict parameter binding and security review.
Unstable query shape over time. The same question can produce different query structures on different runs, which makes performance and compliance review harder.

Filtering and repair on top of unconstrained generation are helpful treatments. They are not a full architectural answer for environments that need both language flexibility and deterministic queries.

How the options compare on what enterprises care about

Hand-written queries only: almost no language agility; strong safety and auditability; maintenance grows with each business question pattern.

Model writes the query: high agility; fragile safety; frequent schema invention; ongoing prompt and guardrail work; weak auditability.

Hybrid templates: agility capped by the catalog; strong safety inside approved templates; maintenance dominated by template growth and regression.

Research overlays (filtering + refinement): higher agility; safety depends on after-the-fact checks; silent false positives remain; more moving parts to operate.

Intent-first (Ontographia, proposed here): the model only fills a structured description of the question; names must exist in the ontology; the query string is produced by a fixed compiler with values bound as parameters; teams maintain the ontology rather than a growing template catalog. Details of that pipeline are in Part 2 and in the architecture guide.

What those environments need at once:

Remove the failure modes that come from letting the model invent query text (invented names, syntax failures, injection-prone string building, runaway scans).
Remove the exponential template maintenance that makes the hybrid stop scaling.

The Intent-first hypothesis

The idea is to separate what the user wants to know from how the database query is assembled.

Application layer (where the model lives). The user asks in natural language. The model outputs a structured Intent: which concepts to start from, which relationships to follow, which filters apply, and what to return. It does not write the database query.

Core layer (fixed rules). Every name in that Intent is checked against a Canonical Ontology, the single vocabulary source of truth. Unknown vocabulary is rejected. A validated Intent is lowered into an internal query structure and emitted as parameterized Cypher. The same ontology, Intent, and dialect always produce the same query text and parameter map. The application then runs that against Neo4j.

Four design principles follow.

The model only extracts Intent. Its job is mapping language into a closed, structured form, not inventing query syntax.
The ontology is the gate. Class, relationship, and property names must exist in the registered ontology before any query is built.
Query text is compiled, not sampled. Filter values travel as parameters, which removes a common injection path.
Research ideas move to the Intent boundary. Schema filtering becomes sending a smaller ontology slice with the prompt. Iterative refinement becomes revising the Intent after execution feedback, then validating and compiling again. Difficulty-adaptive spend becomes spending more retries only when early attempts fail. Part 2 and Part 3 take these up in turn.

Under this split, new questions that stay inside the ontology's vocabulary do not each need a new template. The database never sees model-written query text.

What Parts 2 and 3 cover

Part 2. How the layers are split, why the model and database driver stay outside the core, how the three research ideas map onto Intent handling, and what a small real end-to-end run showed. Implementation detail lives in the docs and repo, not in the article body.
Part 3. Stress tests on ontologies with hundreds to thousands of properties (including the silent false-positive case), public Neo4j benchmark schemas, measured token and accuracy effects of subsetting, and an honest list of what the numbers do not prove.

Takeaways

Raw "model writes the query" is hard to govern; hand-written-only is too rigid for natural language.
The popular hybrid moves the bottleneck to template authoring and regression.
Schema filtering and iterative repair help, but while the model still writes query text, silent wrong-but-valid results remain.
Intent-first keeps language flexibility in a structured description and keeps query assembly deterministic and ontology-checked.

If you want the mechanism next, continue to Part 2. If you want the measurements, skip to Part 3. To try the open-source project, start at the documentation or the repository.

Open source project: github.com/edgesentry/ontographia
Documentation: edgesentry.github.io/ontographia
Author: Yohei Onishi
Tags: #Neo4j #Cypher #KnowledgeGraph #GraphRAG #AIAgents #EnterpriseAI #DataEngineering

Accelerating Multimodal Vector DB with Hugging Face + LanceDB

Yohei Onishi — Wed, 18 Feb 2026 01:00:24 +0000

Introduction

In February 2026, Hugging Face Hub announced native support for the Lance format.

https://lancedb.com/blog/lance-x-huggingface-a-new-era-of-sharing-multimodal-data/

This integration is bringing about a significant shift in the world of multimodal AI. The foundation is now in place to address a growing need: "Take the massive multimodal datasets (images, videos, text, and other data types) available on Hugging Face, convert them into AI-friendly numerical representations (embedding vectors), and search them at high speed as a multimodal vector DB."

This article provides an overview of the technology stack behind this integration — Lance, LanceDB, and Apache OpenDAL — and explores the role these technologies can play in the manufacturing industry, where the author works.

Why I Paid Attention to Lance Format Support on Hugging Face Hub

What caught my attention about this announcement was not just the Hugging Face–Lance integration itself, but rather the underlying technology components mentioned — embedded vector DB, multimodal lakehouse format, and storage abstraction layer — which I intuitively felt could trigger an explosive adoption of multimodal vector DBs on edge devices. By embedding these technologies into the massive Hugging Face ecosystem, developer awareness and adoption could accelerate rapidly.

To be clear, however, the manufacturing industry I work in is not expected to use Hugging Face Hub directly as a production data infrastructure. Hugging Face is fundamentally a platform for sharing and distributing AI models and datasets — it is not a place to store production data from manufacturing floors. What this article focuses on is how the underlying technologies themselves — Lance / LanceDB / OpenDAL — which became visible through the Hugging Face integration, can benefit data infrastructure spanning edge to cloud in manufacturing.

The Role of Multimodal Vector DB in Manufacturing

In manufacturing, where the author works, a diverse range of data modalities is generated in large volumes every day: text (work instructions, inspection records), images (visual inspection photos, microscope images), video (production line footage), audio (equipment operating sounds), sensor data, and more. A multimodal vector DB that can search and analyze across these modalities has the potential to play a central role in AI adoption in manufacturing going forward.

The following scenarios illustrate concrete use cases:

Anomaly detection in visual inspection: Instantly vectorize images captured by production line cameras and perform similarity searches against past defective product images. "Past cases similar to this scratch" can be found in milliseconds.
Predictive maintenance of equipment: Integrate operating sounds, vibration data, and temperature sensor data in a multimodal fashion, and detect anomalous patterns early through vector search.
Knowledge transfer from skilled workers: Store work videos and procedure manual text in the same table, and search for relevant video scenes using natural language queries like "how to use this tool."

Traditionally, these tasks were implemented by data scientists on a case-by-case basis using classical machine learning algorithms and statistical methods. For example, a dedicated image classification model would be trained for each product in visual inspection, and thresholds or rules would be manually adjusted for each piece of equipment in predictive maintenance. Because the pursuit of high accuracy required individual optimization, development and maintenance costs tended to balloon.

Introducing embedding vector DBs fundamentally changes this approach.

At its core, embedding vectors become a "universal language that transcends data modalities." Images required computer vision expertise, audio required signal processing, text required NLP — traditionally, each modality demanded different methods, different tools, and different specialists. However, with multimodal embedding models (CLIP, ImageBind, etc.), any type of data can be projected into the same vector space. Once converted to vectors, comparing images and audio, or searching text against video, can all be accomplished through a single, unified interface: distance computation between vectors.

This simplicity brings the following concrete changes compared to the traditional tailor-made approach:

Horizontal scaling through general-purpose embedding models: Instead of training individual models, images, audio, and text can be vectorized using general-purpose multimodal embedding models (CLIP, ImageBind, etc.) and anomalies detected via similarity search. Even when new products or equipment are added, it is possible to respond simply by adding reference data to the vector DB without retraining the embedding model.
Unified search across multiple modalities: Previously, image inspection and audio inspection were operated as separate systems, but with a multimodal vector DB, images, audio, and sensor values can be stored in the same table, enabling similarity searches that span modalities. Complex searches such as "past cases with sounds similar to this abnormal noise where visual anomalies also occurred simultaneously" become possible.
Opening up to non-data-scientists: Tailor-made ML models required data science expertise for both building and operating them. In contrast, similarity search via vector DB can be used through a simple interface of "register reference data and submit queries," opening up the possibility for on-site engineers and quality control personnel to perform searches and analysis themselves.

A key point here is that LanceDB is an "embedded" vector DB. Edge devices on the manufacturing floor (line-side industrial PCs and gateway equipment) have limited memory and computational resources compared to cloud servers. Since LanceDB operates directly embedded in the application without requiring a server process, it can execute vector search even in resource-constrained edge environments.

Furthermore, LanceDB is designed with a disk-first architecture, meaning it does not need to load the entire database into memory as many vector DBs do. Only the necessary index fragments and data are read from disk on demand during search, and zero-copy access leveraging Apache Arrow minimizes memory copy overhead. This means that even a DB with millions of vector records can operate without issue on edge devices with limited memory (around a few GB).

Additionally, the Lance format adopts a lakehouse architecture, allowing data to be handled in the same format on both local file systems and cloud object storage (S3, GCS, Azure Blob Storage). In other words, data generated and searched on edge devices can be integrated directly into the cloud-scale data lake. The fact that no format conversion or data migration overhead occurs between edge and cloud is a significant advantage in distributed environments like manufacturing.

The above explains why the author, from a manufacturing industry perspective, is paying attention to multimodal vector DBs. The technical terms introduced here — Lance, LanceDB, OpenDAL, etc. — will be explained one by one in the following "Technology Stack Overview" section, so that readers without prior knowledge can follow along.

Technology Stack Overview

What Is Hugging Face?

Hugging Face is the "GitHub of AI" in the AI / machine learning community.

https://huggingface.co

Originally known for its Transformers library for natural language processing (NLP), it has grown into a massive ecosystem for sharing and distributing all kinds of AI-related resources through Hugging Face Hub.

Models: Pre-trained AI models (LLMs, image recognition, speech recognition, etc.) can be published and downloaded
Datasets: Training and evaluation datasets can be published and downloaded
Spaces: Demo applications can be easily hosted

For AI developers, Hugging Face Hub has become "the first place to look for models and data."

What Is Multimodal?

Multimodal refers to handling multiple types (modalities) of data simultaneously.

Traditional AI primarily dealt with a single type of data — text for text, images for images. However, real-world information is always a mixture of multiple modalities.

Modality	Examples
Text	Documents, logs, manuals
Images	Photographs, X-ray images, microscope images
Video	Surveillance footage, production line footage
Audio	Equipment operating sounds, voice instructions
Numerical	Sensor data, quality inspection measurements
Vectors (embeddings)	The above data converted into numerical vectors by AI

Recent AI can understand all of these together. For example, Google's Gemini and OpenAI's GPT-4o can accept images and text simultaneously as input to answer questions.

In the era of multimodal AI, a data platform that can "store and search" this diverse data "in one place, in a unified manner" is essential. This is where Lance and LanceDB come in.

What Is Lance (the Format)?

Lance is an open-source columnar lakehouse format designed from scratch for multimodal AI.

https://lance.org

A "lakehouse" is an architecture that combines the benefits of a large-scale data lake (a big repository where anything can be stored) and a data warehouse (an organized, high-performance query foundation).

The features of the Lance format, compared to familiar technologies, are as follows:

Feature	Parquet / Iceberg	Lance
Storing text & numbers	✅ Strong	✅ Strong
Storing images, video, audio	❌ Requires separate storage	✅ Can be stored in the same table
Storing vector embeddings	⚠️ Possible but not optimized	✅ Native support
Vector search & full-text search	❌ Requires separate index	✅ Index built into the format
Random access performance	⚠️ Slow (scan-oriented)	✅ Up to 100x faster than Parquet
Column-level append/update	❌ Requires full rewrite	✅ Zero-copy append

In other words, Lance is a format that can store binary data like images and video, their metadata, and vector embeddings all in a single table. Data that was previously managed separately — "images in object storage, metadata in an RDB, embeddings in a vector DB" — can be unified.

The Relationship Between Lance and Iceberg / Delta Lake

One point worth clarifying: while Lance is positioned as a "lakehouse format," lakehouse ≠ Iceberg or Delta Lake. Lakehouse is fundamentally an architectural philosophy of "combining the flexibility of a data lake with the manageability of a data warehouse," and Iceberg, Delta Lake, and Lance are table/data formats that implement this philosophy.

Lance is not compatible with Iceberg or Delta Lake — they are independent formats. However, they are complementary, not competitive.

Format	Strength	Primary Data
Iceberg / Delta Lake	Structured data table management (schema evolution, ACID transactions, time travel)	Sales data, logs, master data, and other structured tables
Lance	Storage and search of multimodal data and embedding vectors	Images, video, audio, vector embeddings and their metadata

In practice, a combined configuration using both is conceivable. For example, management metadata for multimodal embedding vectors stored in Lance (who generated them with which model, which product line they belong to, etc.) could be managed in Iceberg / Delta Lake tables. Furthermore, using the metadata catalog Unity Catalog from Databricks (the developer of Delta Lake), it may become possible in the future to manage Lance tables and Iceberg / Delta Lake tables in a unified manner, with consistent access control and data lineage.

What Is LanceDB?

LanceDB is a serverless, embedded vector database built on the Lance format.

https://lancedb.com

What does "embedded" mean? Databases broadly fall into two architectures:

	Client-Server	Embedded
Examples	PostgreSQL, MySQL, Pinecone	SQLite, DuckDB, LanceDB
How it works	Start a separate server process & access via network	Runs directly within the application process
Operations	Server management & monitoring required	No server needed (zero ops)
Best suited for	Large-scale cloud systems	Edge devices, laptops, CI/CD pipelines

If SQLite is the embedded DB for relational data and DuckDB is the embedded DB for analytical queries, then LanceDB can be positioned as the "embedded DB for vector search."

Key features of LanceDB include:

🔍 Supports vector similarity search, full-text search, and hybrid search
📦 Can directly store and query multimodal data (images, video, text, and vectors stored together)
🦀 Low latency through a high-performance Rust-based architecture
🔄 Automatic versioning enables data history management (time travel)
🐍 SDKs for Python, JavaScript (Node.js), and Rust
🔗 Integrates with major AI frameworks such as LangChain and LlamaIndex

What Is Apache OpenDAL?

Apache OpenDAL (Open Data Access Layer) is an open-source data access layer that abstracts access to any storage service through a unified API. It is developed as an Apache Software Foundation top-level project.

https://opendal.apache.org

Why Storage Abstraction Is Needed

In modern data infrastructure, data is rarely stored in a single location. For example, the following situations commonly occur:

Development environments store data on local file systems
Staging environments use AWS S3
Production environments use Azure Blob Storage
Some data is distributed across GCS (Google Cloud Storage)
Edge devices store data on local SSDs
Dataset sharing uses Hugging Face Hub

Traditionally, code to access these storage services had to be written separately for each storage provider: boto3 for S3, google-cloud-storage for GCS, azure-storage-blob for Azure — each requiring different libraries and different APIs. Every time the storage changed, application code had to be rewritten, making multi-cloud and hybrid environment support a significant burden.

What OpenDAL Solves

OpenDAL provides a common read / write / list / delete API for these storage services. Application code depends only on the OpenDAL API, and the target storage can be switched by simply changing configuration.

# OpenDAL API Image (pseudocode)

# Writing to a local file system
op = Operator("fs", root="/data/lance")
op.write("dataset.lance", data)

# Writing to S3 with the exact same API
op = Operator("s3", bucket="my-bucket", region="ap-northeast-1")
op.write("dataset.lance", data)

# Hugging Face Hub also uses the same API
op = Operator("huggingface", repo="my-org/my-dataset")
op.read("dataset.lance")

Benefits for Manufacturing

Manufacturing is a textbook example of an environment where multi-cloud and on-premises coexist.

Environment	Storage Example	Use Case
Factory edge devices	Local SSD / NAS	Temporary storage for real-time inspection data
Domestic cloud	AWS S3 / Azure Blob	Aggregation and analysis of factory data
Overseas cloud	GCS / S3 (different region)	Data sharing across global sites
On-premises servers	HDFS / MinIO	Data with high security requirements

With OpenDAL, data access code for Lance / LanceDB only needs to be written once and works on any storage. Operations like syncing a Lance table saved on a local SSD on an edge device to S3 in the cloud can be achieved without any code changes — just a configuration change.

This is the key technology that realizes the vision of "a lakehouse that integrates edge and cloud" at the storage layer. Lance handles the unification of data format, and OpenDAL handles the unification of storage access — when these two combine, a seamless data pipeline from edge to cloud becomes possible.

The Impact of Hugging Face × Lance Integration

With the technology components introduced so far, let us now examine what the Hugging Face × Lance integration changes.

Challenges with the Traditional Workflow

Previously, attempting to use multimodal datasets on Hugging Face Hub required the following cumbersome workflow.

First, use the datasets library to download the dataset in Parquet or CSV format to local storage. Next, extract binary data such as images and video to the local file system. Then, vectorize them using an embedding model and build the generated embedding vectors as an index in a dedicated vector DB such as Pinecone or Weaviate. Additionally, metadata (file names, labels, creation dates, etc.) had to be managed separately in an RDB or spreadsheet.

In other words, using a single dataset required synchronizing at least three different systems (file storage, vector DB, metadata management).

In this workflow, every time a dataset version was updated, all steps had to be repeated, making reproducibility difficult to ensure.

Workflow After Lance Integration

With Hugging Face Hub's native support for the Lance format, this workflow is dramatically simplified.

A dataset published in Lance format contains binary data (images, video), text, metadata, embedding vectors, and even vector search indexes all integrated in a single dataset. Developers simply specify an hf:// URI to directly access the remote dataset without downloading locally, and can execute scans, filtering, and vector search.

The Innovation of Index Sharing

Particularly important in this integration is the ability to share vector indexes together with the dataset.

Traditionally, performing vector search required running an embedding model yourself to vectorize data after downloading the dataset, and then building a vector index from scratch. The larger the dataset, the more computational resources and time this process demanded.

With the Lance format, dataset creators can publish their work with the vector index pre-built. Users can begin vector search immediately without rebuilding the index. For cases like performing vector search on image datasets with millions of records, this means setup work that previously took hours to days is reduced to zero.

Utilization as a Multimodal Vector DB

This integration has a particularly significant impact for developers who want to use LanceDB as a multimodal vector DB.

Consider, for example, a large-scale multimodal dataset published on Hugging Face Hub — images with their captions, plus embedding vectors generated by CLIP or ImageBind. Previously, using such a dataset required the cumbersome workflow described above.

After Lance integration, simply passing an hf:// URI to LanceDB makes that dataset function as a multimodal vector DB as-is. Cross-modal searches like "search for images similar to this image" or "search for images semantically close to this text" can be executed immediately without any setup.

No format conversion or index rebuilding is needed between data providers and consumers. This removal of friction is the greatest impact of the Hugging Face × Lance integration.

Looking Ahead: The Future of an Enterprise "Multimodal Vector Data Hub"

The Hugging Face × Lance workflow introduced so far is currently realized only on the public platform of Hugging Face Hub. However, the essence of this mechanism — unifying multimodal data, embedding vectors, and indexes for sharing and distribution — is a concept that can be applied to enterprise internal infrastructure as well.

Imagine if a multimodal vector data hub — what might be called an internal "Hugging Face Hub for multimodal vectors" — were built within an enterprise. Developer workflows in manufacturing could fundamentally change as follows.

Current State: Building Individually for Each Edge Device

Currently, to perform vector search on edge devices at manufacturing sites, data collection, vectorization, and index building must be done individually for each device, each line, and each factory. Even deploying a visual inspection vector DB built at one factory to another requires data migration and index rebuilding, making horizontal scaling costly.

Future: Register Metadata in the Data Hub, Fetch Vector Data On-Demand Only When Needed

If an enterprise multimodal vector data hub existed, the workflow would look like the following.

The key point here is that in a lakehouse architecture, no data copying occurs. What is registered in the data hub is purely metadata (dataset location, schema, index information, etc.) — the actual data such as images and embedding vectors remain on the object storage as-is. When an edge device executes a search, only the necessary portions are fetched on-demand from the object storage. This means that even petabyte-scale datasets do not need to be fully copied or synced.

This workflow, once realized, would bring about the following revolutionary changes.

1. "Deploying a vector DB" becomes as natural as deploying software

Once the data science team publishes a vector dataset generated with a new embedding model to the data hub, each edge device can retrieve it by simply specifying a URI. Like pulling a container image, pulling the contents of a vector DB and running it immediately becomes possible. Deploying a new product line's visual inspection model across all factories — previously taking weeks — could be completed in minutes.

2. A feedback loop between edge and cloud emerges

New data generated on-site at edge devices (newly detected defective product images, previously unseen abnormal sound patterns, etc.) accumulates in local Lance tables and is synced to the cloud via OpenDAL. The data science team continuously improves the embedding model using data aggregated from all factories and republishes the updated vector dataset to the data hub. This edge → cloud → model improvement → edge cycle completes within the same Lance ecosystem without format conversion.

3. Data democratization and reuse accelerate

A "welding defect embedding vector collection" built by one factory's quality control team becomes instantly available to other factories and departments through the data hub. Previously, each team independently collected, processed, and built models from their own data, but by managing vector datasets as shared assets, organization-wide knowledge accumulation and horizontal scaling accelerate.

At present, such enterprise multimodal vector data hubs are not widely available. However, the architecture pattern validated by the Hugging Face × Lance integration is designed to be brought directly into enterprises. The Lance format is open source, and OpenDAL supports enterprise object storage services such as S3, GCS, and Azure Blob Storage. The pieces are already in place.

Getting Started

If reading this far has piqued your interest in Lance / LanceDB, I recommend starting with the tutorial published on the official blog.

https://lancedb.com/blog/lance-x-huggingface-a-new-era-of-sharing-multimodal-data/

This tutorial walks you through accessing Lance datasets on Hugging Face Hub via hf:// URI and experiencing vector search and filtering firsthand. You can get a hands-on feel for the "searching remote multimodal datasets instantly without downloading" experience described in this article.

Detailed tutorial steps and practical hands-on exercises using manufacturing data will be covered in a separate article.

Conclusion

This article introduced the following technology stack and its implications for manufacturing, prompted by the native support of the Lance format on Hugging Face Hub.

Lance (Format): A lakehouse format that can store multimodal data and embedding vectors in a single, unified table. It is complementary to, not competitive with, Iceberg / Delta Lake.
LanceDB (Embedded DB): An embedded vector DB built on the Lance format. It runs without a server and operates even on resource-constrained edge devices.
Apache OpenDAL (Storage Access): Abstracts storage access, unifying data access across multi-cloud and on-premises environments without code changes.

When these technologies combine, the construction of a seamless multimodal data platform from edge to cloud becomes realistic: unified data format (Lance) × unified storage access (OpenDAL) × vector search running on edge (LanceDB).

In manufacturing, use cases such as visual inspection, predictive maintenance, and knowledge transfer — traditionally built individually by data scientists in a tailor-made fashion — have the potential to evolve into generalized, horizontally scalable forms through the adoption of embedding vector DBs. Looking further ahead, the emergence of enterprise multimodal vector data hubs may bring an era where deploying a vector DB feels as natural as deploying software.

In the next article, I will introduce the integration of DuckDB and LanceDB — an approach that combines SQL-based data retrieval and analysis with the multimodal lakehouse powered by the Lance format.

https://lancedb.com/blog/lance-x-duckdb-sql-retrieval-on-the-multimodal-lakehouse-format/