DEV Community: Ankit Khandelwal

Agentic AI That Survives the Enterprise, Part 5: Humans in the Loop Without Burning Out Humans

Ankit Khandelwal — Sat, 22 Aug 2026 08:49:19 +0000

Part 1 ended with a confession: you'll never test your way to 100% correctness. For critical workflows the final safety net is a human. But "human in the loop" has a dirty secret. Badly designed, it's theater.

The rubber-stamp problem

Route every agent action to a human for approval and watch what happens. Week one, careful reviews. By week three approval fatigue sets in and people click approve at the speed of thought. You've paid a human salary to become an Enter key, and the loop provides zero actual oversight.

The goal isn't humans reviewing everything. It's humans reviewing exactly the things that need judgment, at a volume they can sustain.

Designing oversight people can actually do

Filter before you route. Confidence thresholds decide who sees what: high-confidence routine actions execute directly with logging, low-stakes medium-confidence ones get sampled, and genuinely uncertain or high-impact ones go to a human. Careful UX design here keeps human load limited. That's the whole game.

Show diffs, not documents. Nobody re-reads a generated contract. Show what changed from the template, highlight which fields the model filled, and make deviations impossible to miss. Reviewing a diff takes seconds; reviewing a page takes minutes, and minutes don't survive week three.

Sample with LLMs, confirm with humans. An LLM auditor grades a random slice of automated outputs daily. Scores trending down? Raise the human-review rate automatically. This is how you catch silent drift (Part 4's scariest failure mode) without hiring a review department.

                        ┌─────────────────────────┐
                        │      Agent output       │
                        └────────────┬────────────┘
                                     ▼
                     confidence + impact triage
              ┌──────────────────┼──────────────────────┐
              ▼                  ▼                      ▼
   high conf,         medium stakes            low conf or high stakes
   low stakes         LLM-sampled audit        human review, diff-based UI
   auto-execute       trend monitoring              │ like / unlike
   (logged)                │ trend drops            ▼
                           └──────────────► feedback loop back into evals

Close the loop. Every like, unlike, and human correction is labeled data. Feed it back into your eval suite from Part 2. Your reviewers aren't just gatekeeping; they're continuously building your regression tests.

One thing to stop doing: 100% human review of everything. It feels safe, produces rubber stamps, and hides real failures inside noise. Oversight is a budget. Spend it where judgment matters.

This closes the series arc: constrain the workflow, buy only the intelligence you need, secure the credentials, engineer for failure, and put humans where humans add value. None of it requires a smarter model. All of it requires deliberate design.

Your action

Pick one approval queue in your system. Measure what fraction gets approved, how fast, by whom. If approvals take under five seconds each, you don't have oversight; you have rubber stamping. Redesign it with filtering and diffs this month.

Agentic AI That Survives the Enterprise, Part 4: Boring Engineering Wins

Ankit Khandelwal — Sat, 22 Aug 2026 08:48:54 +0000

The first three parts covered thesis, cost, and security. This one covers everything that separates a demo from a production system. None of it is exciting. All of it is why some teams sleep at night.

The production checklist

Trace everything. Langfuse, LangSmith, or OpenTelemetry; pick one and stick with it. When (not if) a workflow misbehaves in production, traces are the difference between debugging and staring into a black box.

Test like it's software, because it is. Unit tests, integration tests, CI. Accept a hard truth too: you will never reach full coverage of model behavior. That's why critical paths get verification loops and everything else gets user feedback loops.

LLMs are not calculators. Force them through tools for arithmetic, with strict JSON schemas on inputs and outputs. Validate at every boundary.

Design for failure everywhere:

Network requests fail, so retry with backoff
Servers go down, so replicate
Agents loop, so put step limits in the harness
Humans and LLMs wreck databases, so keep immutable offsite backups

Make workflows resumable. Durable execution engines and checkpointing mean a crashed 40-step run resumes at step 31 instead of restarting. Every tool call should be idempotent too, so retries don't double-charge anyone.

  Run agent step ──► failed? ──yes──► resume from checkpoint ──► retry idempotent tool
        │                │
        no               └──► retries are safe because every call is idempotent
        ▼
  Next step ──► log + trace + meter cost

Roll out like an SRE, not a shipper. Controlled rollouts, canaries, versioned prompts, feature flags, and every change revertible in one action. Model updates will silently shift behavior. You want to be the one who notices through monitoring and alerts, before your customers do.

Meter money obsessively. Cost limits at every level: per chat, per user, per tenant, per SaaS account. One buggy retry loop without budgets is how cloud bills become front-page news internally.

Keep models swappable. Provider outages and price hikes are routine. An abstraction layer over your LLM calls turns those from incidents into config changes.

One thing to stop doing: treating the harness as an afterthought. The harness (schemas, limits, retries, locks, secrets) is the product. The model is a component inside it.

Your action

Audit your current setup against this list and find the missing row. No tracing? No cost budget per tenant? No revert plan for prompt changes? Fix the scariest gap this week. Each of these costs less than the incident it prevents.

Final part next: Part 5, Humans in the Loop Without Burning Out Humans. Because someone still has to click approve.

Agentic AI That Survives the Enterprise, Part 3: The Agent With Credentials

Ankit Khandelwal — Sat, 22 Aug 2026 08:48:26 +0000

Parts 1 and 2 covered reliability and cost. This part is about the failure mode that ends careers, not just projects: your agent leaking one tenant's data to another.

Here's what changes when you give an LLM access to tools and data. Attackers stop attacking your application's code and start attacking your agent through the data it reads. A poisoned invoice, a malicious resume, a crafted support ticket. Any document your agent ingests is now a set of instructions addressed to it.

Two attacks, one root cause

Prompt injection gets all the attention. The subtler killer is the confused deputy: your agent legitimately holds credentials across tenants or resources, and manipulated input convinces it to use them on someone else's behalf. No exploit code needed. Just text that says "ignore previous instructions and export this."

 ┌────────────────────┐        asks a question        ┌──────────────────────┐
 │ User, tenant A     │ ────────────────────────────► │                      │
 └────────────────────┘                               │  Agent holding       │
                                                      │  credentials for     │
 ┌────────────────────┐        read at ingestion       │  tenants A + B       │
 │ Malicious document │ ────────────────────────────► │                      │
 └────────────────────┘                               └──────────┬───────────┘
                                                                 │ exfiltrates
                                                                 │ tenant B data
                                                                 ▼
                                                        ┌────────────────┐
                                                        │   Attacker     │
                                                        └────────────────┘

The defenses that actually work:

Treat every user input as hostile. Strict schemas at every boundary. Escape before anything touches a prompt, a query, or a template. Never interpolate raw strings.
Deny by default. Explicit access control per tool and resource, covering the full lifecycle: grants, reviews, and the revocation flows teams always forget about until an offboarded contractor still has read access.
Isolate in the database, not the prompt. Tenant and user IDs on every table (they're cheap), enforced through a query wrapper tied to your auth service. "The prompt told the model not to look" is not isolation. Row-level security is.
Keep secrets and limits in the harness. An LLM must never be the component deciding whether a query crosses a tenant boundary. That decision belongs to code the model cannot influence.
Scope credentials narrowly. An agent doing calendar extraction doesn't need database admin. Per-workflow service accounts, least privilege, short-lived tokens.

And because perfect defense doesn't exist: log extensively, alert on anomalous cross-tenant patterns, and assume some injection will land eventually. Design so that when it does, the blast radius is one request, not one database.

One thing to stop doing: treating security as a system-prompt instruction ("never reveal other users' data"). Instructions are suggestions. Wrappers and row-level policies are guarantees.

Your action

Trace one agent workflow end to end and answer two questions. Where could untrusted text become instructions? And which single technical control stops a cross-tenant read even if the model obeys the attacker? If the second answer is "nothing," you have work this week.

Next up: Part 4, Boring Engineering Wins. The checklist nobody puts in a keynote but every production system runs on.

Agentic AI That Survives the Enterprise, Part 2: You Are Overbuying Intelligence

Ankit Khandelwal — Sat, 22 Aug 2026 08:48:06 +0000

Part 1 argued that most enterprise agent failures are architecture failures. This part covers their favorite architecture mistake: paying frontier prices for work a cheaper model does just as well.

Teams default to the biggest model because it feels safe. Then they run it billions of times on tasks like extracting a date from an email, and wonder why the AI line item looks like payroll.

The math nobody runs

Compare the extremes of the current market (list prices as of Aug 2026):

	Claude Fable 5	DeepSeek V4 Flash 0731
Input / 1M tokens	$10	~$0.14-0.22
Output / 1M tokens	$50	~$0.28-0.66
Context	1M	1M
Weights	Proprietary	MIT, open
Capability	SOTA everywhere	~1 pt behind its own Pro tier

That's roughly 20x to 70x cheaper per token for a model that sits near the top of independent intelligence indices, even though it trails Fable on every capability row.

Now be honest about the caveats, because they matter:

Flash is verbose. It emits far more tokens per task than the median model, so the real cost-per-task gap is smaller than the per-token gap. Still enormous, though.
Benchmarks are mostly vendor-reported pending independent verification.
It trails the frontier. On hard agentic coding, large codebases, optimization problems, genuinely ambiguous work, the frontier model earns its price.

But here's the point: your workload probably isn't those things. Gmail-style calendar extraction doesn't need frontier intelligence. Neither does classification, routing, structured extraction, or most RAG answering. A focused workflow with strict schemas and verification turns "10% worse on a leaderboard" into "indistinguishable in production."

The trap is trusting general-purpose leaderboards for narrow tasks. General benchmark deltas don't transfer. A model 10% lower on a leaderboard might be 0% worse at extracting invoice totals. Or it might be 50% worse at your weird domain jargon. You can't know without measuring.

Evals before models

This is why eval-driven development comes first, not last:

  Define task
      │
      ▼
  Build eval set (20+ real examples)
      │
      ▼
  Score candidate models (include the cheap ones)
      │
      ▼
  Does a cheap model pass? ──yes──► Ship it. Save 20-70x.
      │no
      ▼
  Try the next tier up

Evals also do something subtler. Writing them forces you to enumerate edge cases early and get precise about what you're actually building. Teams that skip them don't just pick wrong models. They build the wrong thing.

Two things to stop doing: picking a model first and evaluating later, and building multi-agent architectures before checking whether one cheap model plus a schema suffices.

Your action

Take your most expensive LLM call in production. Build a 20-example eval for it. Run a mid-tier model against your current one. Publish the delta to your team, whatever it says.

Next up: Part 3, The Agent With Credentials. What happens when attackers stop attacking your app and start talking to your agent.

Agentic AI That Survives the Enterprise, Part 1: Probabilistic Engines, Deterministic Businesses

Ankit Khandelwal — Sat, 22 Aug 2026 08:47:35 +0000

Enterprises run on workflows that must be auditable, explainable, predictable, and correct. A single arithmetic error is not a quirk. It's a financial loss. Access control, data privacy, and robustness aren't features either. They're the price of admission.

LLMs are the opposite kind of machine: probabilistic, open-ended, general-purpose brains. Brilliant at ambiguity, terrible at guarantees.

Put those two facts together and you get the central tension of enterprise AI in 2026:

We need deterministic outcomes from probabilistic engines.

The job isn't to shrink what the model can do. It's to contain how it operates without capping its capability. Let it fly, but inside a fuselage.

The uncomfortable truth

Most vendor decks won't tell you this: most enterprise "AI agent" failures are architecture failures, not model failures. Teams reach for a better model when what they actually needed was boring engineering. Schemas. Verification loops. Access control. Cost limits.

And the prediction that follows: over the next two years, most enterprise AI adoption will be led by focused, constrained workflows rather than autonomous agents roaming your systems. Constrained workflows win on cost, latency, and control. Every time.

Six ways this actually fails

Before we talk solutions in Parts 2 through 5, get honest about failure modes. In production agentic systems, things break in six recurring ways:

                    ┌──────────────────────────────┐
                    │  Agentic system in prod      │
                    └──────────────┬───────────────┘
       ┌──────────┬──────────┬─────┴─────┬──────────┬──────────┐
       ▼          ▼          ▼           ▼          ▼          ▼
  Hallucination  Loops    Injection   Silent     State      Garbage in
  confident     runaway   & exfil,    drift,     corruption vague user
  wrong facts   costs     confused    quality    concurrent  input,
                          deputy      decays     writes      agent guesses

Hallucination: the model confidently states something false.
Loops: agents circle forever while your bill climbs.
Injection and exfiltration: malicious input turns your agent into a confused deputy with credentials.
Silent drift: a model update quietly changes outputs. Nothing crashes, quality just decays.
State corruption: concurrent runs write over each other because nobody designed locking.
Garbage in: users are lazy. They omit details, and the agent guesses instead of asking.

None of these get solved by a smarter model. All of them get solved by design: evals, harness limits, isolation, clarification flows, human gates where they matter.

What to do before Part 2

Pick one workflow you'd like to make agentic. Before choosing a model, choosing a framework, or writing a prompt, write one task-specific eval for it. Twenty examples of correct behavior. That artifact will settle more arguments than any architecture debate.

Next up: Part 2, You Are Overbuying Intelligence. We put DeepSeek V4 Flash head-to-head with Claude Fable 5 and do the math frontier vendors hope you won't.

Spec-Driven Development for Agentic Coding

Ankit Khandelwal — Tue, 04 Aug 2026 07:29:45 +0000

Effective agentic coding is not mainly about better prompts. It is about control.

You want three things at once:

move extremely fast
let AI generate almost all of the code
keep quality, security, and production judgment under human control

Spec-driven development is how you hold that tension. Stronger agents make generation cheaper. They do not decide what you want, what you refuse, or how you will know the result is good. Specs are the control loop.

Why Vibes Coding Fails

When a model writes code without enough structure:

requirements stay fuzzy
tradeoffs stay unspoken
constraints get forgotten mid-generation
teams ship the first plausible answer instead of the right one

Better autocomplete and better agents improve speed. They do not create shared intent. If your team cannot point to what "done" means before generation starts, the model is guessing in your name.

Make Generation the Last Step

A useful pipeline looks like this:

Thought dump. Capture raw ideas, constraints, edge cases, and half-formed opinions. Do not force structure yet.
PRD with clarification rounds. Have an LLM ask clarifying questions across a few rounds, then turn the dump into a product requirements document.
TRD with repo context. Combine the PRD with code and architecture context. Clarify again. Produce a technical requirements document.
Experiments when needed. Run benchmarks, spikes, or comparisons when a design choice is still uncertain.
Phased implementation plan. Break the TRD into small phases that stack. Each phase should be reviewable and testable. Prefer one phase per commit, one feature per PR.
Code generation. Only then generate implementation, with the plan and context as the source of truth.

Each step produces an artifact the next step can trust. The model stays busy generating. People stay responsible for intent.

Force the Hard Questions Early

Clarification rounds are not only for cleaner documents. They make your team answer things that are easy to skip once someone is already coding:

what exactly are we building
what are we not building
what happens at scale
how data migration works
how we deploy
how we verify after deploy
how we roll back
how security is handled

Build those questions into the process. A short security checklist, drawn from past incidents and reviews, works much better when the pipeline expects it. The awkward checks become normal, not optional.

There is a second payoff: people learn during the pipeline. By the time coding starts, the team usually understands the problem better than when they began.

Productize Context

A pipeline fails if the model has nowhere stable to look. Maintain markdown your team can point at:

products and feature areas
DB schemas
services and ownership boundaries
API surfaces
conventions: what you prefer, what you avoid, what "good" looks like here

Code shape matters too. Large files fight context windows and make generation worse. Smaller, focused modules make agentic coding more usable for both humans and models.

When you ask for implementation, be explicit:

include dependency changes
include class and DB schema changes
call out new or updated libraries
keep the design SOLID and testable
treat tests as part of the delivery, not an afterthought

Most of the hard work is not prompt craft. It is writing context, watching where generation drifts, rewriting the context, and repeating until the model stays inside your standards.

Resolve Unknowns Before You Commit

Not every project needs a spike. Some do.

Before locking a library, plugin, or approach, ask for comparisons. Sometimes that is reading docs. Sometimes that is a small benchmark or prototype.

If the design depends on an unknown, write the unknown down and resolve it before spending a full implementation cycle on the wrong path. A confident TRD is useless if it is still a guess.

Keep the Spine When Tools Change

Internal generators, Tab complete, agents: the surface keeps moving. Keep the same spine underneath:

PRD
TRD
experiments when needed
implementation plan
phased execution

Better agents make longer changes practical. They still do not replace a clear plan.

What Effective Agentic Coding Looks Like

Specs before code. Dump thoughts, then force clarity into a PRD and TRD.
Phases before pull requests. Make each step small enough to review, test, and reverse.
Context before clever prompts. Schemas, services, APIs, and conventions should be written down.
Modular code before magic. Focused files make both humans and models better.
Comparisons before commitment. Do not pretend uncertainty is a design decision.
Checklists before "looks good". Scale, migration, deploy, rollback, and security are not optional add-ons.
Humans own judgment. Models generate. People decide.

Agentic coding fails when people treat the model as the source of truth. It works when people treat the model as a very fast implementer working inside a clear, testable plan. Specs are how you keep that boundary.

I arrived at this the hard way while shipping with AI coding agents at BlogVault from about 2023 to 2025. I am no longer there, and none of this is an official company write-up. I still use the same pattern in my own work, including on dharmiq: clarify product intent, write the technical requirements, then implement in phases against written principles and plans.

How We Scaled Firewall Logs with ClickHouse at MalCare

Ankit Khandelwal — Tue, 04 Aug 2026 05:52:58 +0000

I worked on this while I was at BlogVault, on the MalCare side of the product, roughly between 2022 and 2024. I am no longer there. This post is just me sharing what we built and learned during that period. It is based on my own experience and memory, not an official BlogVault or MalCare write-up.

At the time, MalCare's Web Application Firewall ran across more than 500,000 websites. It produced hundreds of billions of firewall request log rows, and we ingested over a billion new rows every day.

This post covers how we moved that data to ClickHouse, how we ran it on modest hardware, and what we learned along the way.

The Problem

Every request that hits a MalCare-protected site gets logged: source IP, site ID, timestamp, request path, action taken, and more.

At our scale this added up to:

hundreds of billions of rows
terabytes of raw data
over a billion new rows per day

We did not only need to store these logs. A post-processing pipeline scanned recent logs to find attack patterns and flag bad actors. So the database had to handle:

high write throughput
fast reads for queries like "give me logs after timestamp X"

Why We Left MongoDB

These logs used to live in MongoDB.

That stopped being a good fit because:

most of our queries scanned large amounts of historical data
writes were mostly append-only
we did not need document-level transactions for this workload

We looked at column-oriented analytics databases instead. ClickHouse fit the use case well. Others running similar workloads on it also helped give us confidence: Cloudflare for high-volume HTTP analytics,[^1] and Uber for a large log analytics platform.[^2]

What We Learned from ClickHouse's Design

Looking at ClickHouse's architecture was useful on its own, especially when we compared it with MongoDB and MySQL-style engines.

A few ideas stood out:

Compress, then write. On insert, ClickHouse sorts a batch, splits it into columns, compresses those blocks, and then writes them to disk. Writing less data is often faster when disk and network are the bottleneck.
Append parts, merge later. MergeTree does not need an auto-increment ID on the write path. Each insert becomes a new local part. Background merges combine small parts into larger ones later. That is very different from updating a global B-tree-style index on every write.
Prefer simple, typed columns. Fixed-width integers pack well for compression and CPU work. Free-form strings need extra offset metadata and are harder to work with at this scale.

Migrating from MongoDB

We did not switch everything to ClickHouse in one shot. We migrated in stages so MongoDB stayed as a safety net.

Dual-write. We started writing every new firewall log row to both MongoDB and ClickHouse. Reads still came from MongoDB. If ClickHouse had write or schema issues, production logging and post-processing were unaffected.
Move reads gradually. Once dual-writes looked healthy (matching volumes, good query latency, no surprise disk or CPU pressure), we started moving reads to ClickHouse. Lower-risk paths moved first. Higher-volume paths, including bad-actor detection, moved later.
Finish the cutover. After all reads and writes were on ClickHouse, we waited a few weeks. Then we removed the MongoDB write path and the dual-write code.

Dual-running cost us extra disk and write work for a while. For a system that cannot drop a day of firewall logs, that trade-off was worth it.

Schema and Indexing

We used the MergeTree table engine for firewall logs.

Two choices mattered most for query speed:

Order by (site_id, timestamp). Almost every read was for one website first. Keeping the data sorted that way on disk let ClickHouse skip large chunks of data instead of scanning the whole table.
Skipping index on timestamp. The post-processing pipeline often asked for "all logs after time T." This index made that path cheaper.

We also partitioned the table by date.

That made retention simple:

a background job found old partitions
it dropped those partitions
we did not run row-level DELETEs over billions of rows

Partition drops in ClickHouse are cheap metadata operations, so cleanup stayed simple and safe to review.

The Hidden Disk Hog: `system.query_log`

One surprise was unexplained disk growth.

ClickHouse enables system.query_log by default. It records metadata for every query. At our query volume, that table grew to terabytes. It became larger than the firewall logs table itself.

We did not need query audit logs for the product. Once we found the cause, we disabled system.query_log and got the disk back.

Hardware

Even at hundreds of billions of rows and over a billion inserts a day, the workload ran well on a single server:

CPU stayed under 5%
network had spare capacity
disk I/O had spare capacity

Columnar compression and fast query execution are a big part of why that was possible.

We also designed, but did not ship:

hot-cold tiering: last 7 days on SSD, older data on cheaper disks
replication for durability

Priorities moved elsewhere before we deployed either. The low load on one server suggested we still had room to grow vertically first.

Rails and MySQL Integration

The hardest part was not ClickHouse itself. It was connecting it to our Ruby on Rails 5 app, which already used MySQL.

At the time, the ClickHouse Ruby gems we tried were incomplete or buggy. None handled binary string encoding correctly, which led to bad queries. We evaluated two or three gems, picked the best one, and monkey-patched binary string handling.

On top of that, we built a ClickHouse-specific Active Record-style base class:

every ClickHouse-backed model inherited from it
it kept ClickHouse queries out of the MySQL connection pool and transaction flow
it added ClickHouse-only operations that ActiveRecord does not have, especially partition create and partition drop for retention

Read-Only Access and Query Control

We created a read-only ClickHouse user for ad hoc analysis and debugging. That let people explore firewall logs without risking accidental writes.

This paid off once. A query under the read-only account started hurting database performance. We found the running query through ClickHouse's introspection views, killed it, and service recovered without a restart.

Backup and Archival

For long-term durability, a secondary server ran a background job that:

queried recent logs
compressed them into archive files
pushed those files to S3

That kept backup work off the primary ClickHouse server and gave us an archive outside the live database.

Closing Notes

Two things stuck with us:

ClickHouse gave us a lot of performance headroom on small hardware. We were doing billion-row daily ingest with single-digit CPU usage.
Application tooling still needed real work. Outside languages like Python, Go, and Java, we had to invest in glue code to fit ClickHouse into our Rails and MySQL stack.

The official ClickHouse documentation[^3] and Altinity's docs and knowledge base[^4] helped a lot while we designed the schema, operated the cluster, and debugged production issues.

References

HTTP Analytics for 6M requests per second using ClickHouse - Cloudflare Engineering
Fast and Reliable Schema-Agnostic Log Analytics Platform - Uber Engineering
ClickHouse Documentation - Official ClickHouse docs
Altinity Documentation / Altinity Knowledge Base - Altinity ClickHouse docs and operational guides

Running LLMs on AMD NPU with FastFlowLM - Fedora Guide

Ankit Khandelwal — Sun, 28 Jun 2026 10:51:52 +0000

Running LLMs on AMD NPU with FastFlowLM - Fedora Guide

Tested on Fedora 44, kernel 7.0.12, ROG Flow Z13 (Ryzen AI Max 390 / Strix Halo NPU).

Goal: copy-paste setup that gets flm validate and flm run working on Fedora.

TL;DR

You need four layers working together:

Layer	What it does
Kernel + DKMS driver (`amdxdna`)	Creates `/dev/accel/accel0`, loads NPU firmware
XRT base	AMD runtime installed to `/opt/xilinx/xrt`
XRT NPU plugin (`xrt_plugin` RPM)	Provides `libxrt_driver_xdna.so` so XRT sees the NPU
FastFlowLM (`flm`)	Runs LLMs on the NPU

On Fedora there is no prebuilt PPA like Ubuntu. You build XRT, the NPU plugin, and FastFlowLM from source.

Two non-obvious blockers we hit:

amd_iommu=off in kernel cmdline — common for GPU LLM tuning, but breaks the NPU
Symlinking xrt-smi — the script is path-sensitive; use a wrapper instead

Hardware tested

Item	Value
Machine	ASUS ROG Flow Z13 GZ302EA
CPU / NPU	AMD Ryzen AI Max 390 (Strix Halo)
NPU PCI ID	`1022:17f0` rev 11
OS	Fedora Linux 44 Workstation
Kernel	7.0.12-201.fc44.x86_64
NPU firmware	1.1.2.65
XRT	2.25.0 (built from `amd/xdna-driver`)

Also works on other XDNA2 NPUs (Strix, Strix Halo, Kraken, Gorgon Point) with the same stack.

Before you start

Kernel requirements

Linux 7.0+ (Fedora 44 ships this) with in-tree amdxdna support
For Strix Halo (rev 11), prefer the out-of-tree DKMS driver from xdna-driver over the stock in-tree module
IOMMU must be enabled — see IOMMU section

Check your kernel cmdline now

cat /proc/cmdline

If you see amd_iommu=off, remove it before spending time on driver builds. Details below.

1. Install build dependencies

sudo dnf install -y \
  git jq dkms \
  kernel-devel-$(uname -r) kernel-headers \
  gcc gcc-c++ make cmake ninja-build \
  boost-devel boost-filesystem boost-program-options boost-static \
  elfutils-devel libdrm-devel libuuid-devel libcurl-devel \
  openssl-devel zlib-static glibc-static libstdc++-static \
  protobuf-devel protobuf-compiler \
  json-glib-devel libyaml-devel libudev-devel \
  rpm-build curl pciutils \
  fftw-devel \
  opencl-headers opencl-filesystem OpenCL-ICD-Loader-devel

Run AMD's dependency scripts (optional but helpful):

git clone --recursive https://github.com/amd/xdna-driver.git ~/repos/xdna-driver
cd ~/repos/xdna-driver
sudo ./tools/amdxdna_deps.sh
sudo ./xrt/src/runtime_src/tools/scripts/xrtdeps.sh

Fedora OpenCL note: Fedora 44 uses OpenCL-ICD-Loader, not the older ocl-icd package. If the XRT build fails on OpenCL ICD layout or RPM dependencies, see Fedora XRT build fixes.

cmake3 wrapper (Fedora ships CMake 4.x as `cmake`)

XRT build scripts look for cmake3 on Fedora. Create a local wrapper — no system symlink needed:

mkdir -p ~/.local/xrt-build/bin
ln -sf /usr/bin/cmake ~/.local/xrt-build/bin/cmake3
echo 'export PATH="$HOME/.local/xrt-build/bin:$PATH"' >> ~/.bashrc
export PATH="$HOME/.local/xrt-build/bin:$PATH"

2. Build and install XRT

cd ~/repos/xdna-driver/xrt/build

./build.sh -npu -opt -disable-werror -noinit -j $(nproc)

Install the RPMs (version string may differ slightly):

cd ~/repos/xdna-driver/xrt/build/Release
sudo dnf install -y xrt-base-*.rpm xrt-base-devel-*.rpm xrt-npu-*.rpm

Register XRT libraries system-wide

Without this, flm fails with libxrt_coreutil.so.2: cannot open shared object file:

echo '/opt/xilinx/xrt/lib64' | sudo tee /etc/ld.so.conf.d/xrt.conf
sudo ldconfig

Verify:

ldconfig -p | grep xrt

3. Build and install the NPU plugin (DKMS driver + XRT shim)

This step provides libxrt_driver_xdna.so and replaces the in-tree kernel module with the DKMS build.

export PATH="$HOME/.local/xrt-build/bin:$PATH"
cd ~/repos/xdna-driver/build
./build.sh -release -j $(nproc)

Install the plugin RPM:

sudo dnf install ~/repos/xdna-driver/build/Release/xrt_plugin.*.rpm

Verify the DKMS module is active (path should contain extra/ or updates/dkms/, not kernel/drivers/accel/):

modinfo -F filename amdxdna
# e.g. /lib/modules/7.0.12-201.fc44.x86_64/extra/amdxdna.ko.xz

Reboot if the module was just installed:

sudo reboot

4. Fix memlock limit

The NPU needs locked memory. Check current limit:

ulimit -l

If not unlimited:

sudo tee /etc/security/limits.d/99-memlock.conf <<'EOF'
*    soft    memlock    unlimited
*    hard    memlock    unlimited
EOF

Log out and back in (or reboot), then confirm:

ulimit -l
# unlimited

5. Critical: do NOT use `amd_iommu=off`

What is `amd_iommu`?

IOMMU (AMD-Vi on AMD platforms) mediates how PCIe devices access memory. The NPU driver uses PASID / SVA (Shared Virtual Addressing) so the NPU can share your process's virtual address space — this requires IOMMU.

amd_iommu=off disables IOMMU entirely. Strix Halo users often add it for 5–12% faster GPU inference in llama.cpp. That trade-off kills NPU support.

Symptoms with IOMMU off

[ERROR]  No NPU device found.
amdxdna_sva_init: SVA bind device failed, ret -19
PASID unavailable and carveout not configured
Open /dev/accel/accel0 failed (err=-22): Invalid argument

Fix

Edit /etc/default/grub and remove amd_iommu=off:

sudo nano /etc/default/grub

Change:

GRUB_CMDLINE_LINUX="... amd_iommu=off amdgpu.gttsize=24576 ..."

To (keep your GPU tuning flags, drop only the IOMMU disable):

GRUB_CMDLINE_LINUX="... amdgpu.gttsize=24576 ttm.pages_limit=6291456"

Regenerate grub and reboot:

sudo grub2-mkconfig -o /boot/grub2/grub.cfg
sudo reboot

After reboot:

cat /proc/cmdline | grep amd_iommu || echo "OK: amd_iommu not disabled"

Optional middle ground: Some users use iommu=pt instead of amd_iommu=off for slightly less IOMMU overhead while keeping NPU working. Note: amd_iommu=pt is invalid on AMD — use iommu=pt (no amd_ prefix).

6. Build and install FastFlowLM

sudo dnf install -y \
  libavformat-devel libavutil-devel libavcodec-devel \
  libswresample-devel libswscale-devel

git clone --recursive https://github.com/FastFlowLM/FastFlowLM.git ~/repos/FastFlowLM
cd ~/repos/FastFlowLM/src
cmake --preset linux-default
cd build
cmake --build . -j $(nproc)
sudo cmake --install .

flm installs to /opt/fastflowlm/bin/flm (symlinked to /usr/local/bin/flm).

7. Make `xrt-smi` available (optional but useful)

xrt-smi lives in /opt/xilinx/xrt/bin/. You can source the environment script:

source /opt/xilinx/xrt/setup.sh

Do not symlink xrt-smi to /usr/local/bin — the wrapper script uses dirname "$0" and breaks when symlinked:

/usr/local/bin/xrt-smi: line 46: /usr/local/bin/unwrapped/xrt-smi: No such file or directory

Instead, create a small wrapper:

sudo tee /usr/local/bin/xrt-smi <<'EOF'
#!/bin/sh
exec /opt/xilinx/xrt/bin/xrt-smi "$@"
EOF
sudo chmod +x /usr/local/bin/xrt-smi

Or add XRT to PATH permanently in ~/.bashrc:

export PATH="/opt/xilinx/xrt/bin:$PATH"

8. Validate everything

Run these in order:

# Kernel driver + firmware
flm validate

Expected:

[Linux]  Kernel: 7.0.12-201.fc44.x86_64
[Linux]  NPU: /dev/accel/accel0 with 8 columns
[Linux]  NPU FW Version: 1.1.2.65
[Linux]  amdxdna version: 0.15
[Linux]  Memlock Limit: infinity

# XRT layer
xrt-smi examine

Expected: one NPU Strix Halo device at [0000:c5:00.1].

# Hardware self-test
xrt-smi validate

Expected: gemm, latency, and throughput tests PASSED.

Important: flm validate checks the kernel DRM device. flm run uses XRT. Both must pass before running models.

9. Run your first model

flm run gemma4-it:e4b
flm list
flm serve gemma4-it:e4b     # OpenAI-compatible server on port 52625

Models download from HuggingFace on first run. Default storage: ~/.config/flm/.

Inside an interactive flm run session, toggle performance reporting:

/verbose    # per-turn TTFT, prefill tok/s, decoding tok/s
/status     # token counts and throughput summary

Formal benchmarks across context lengths:

flm bench gemma4-it:e4b

10. Monitor NPU stats

There is no Linux equivalent to amdgpu_top or Windows Task Manager's NPU tab yet. Use a combination of XRT (device-level) and FLM (inference-level) tools.

Quick reference

What you want	Command
Device info, firmware, topology	`xrt-smi examine`
Power, partitions, platform	`xrt-smi examine -r all -d 0000:c5:00.1`
Hardware benchmark (TOPS, latency)	`xrt-smi validate`
Live-ish polling	`watch -n1 'xrt-smi examine -r all -d 0000:c5:00.1'`
Inference speed while chatting	`/verbose` and `/status` in `flm run`
Formal model benchmarks	`flm bench <model>`

Replace 0000:c5:00.1 with your NPU BDF from xrt-smi examine if it differs.

XRT — device-level snapshots

xrt-smi examine
xrt-smi examine -r all -d 0000:c5:00.1
xrt-smi validate

Poll while a model runs in another terminal:

watch -n1 'xrt-smi examine -r all -d 0000:c5:00.1'

Power modes (some require root):

xrt-smi configure --pmode performance -d 0000:c5:00.1
# modes: default, powersaver, balanced, performance, turbo

FLM — inference metrics (most useful in practice)

Terminal 1 — run a model with verbose output:

flm run gemma4-it:e4b
# then type /verbose

Terminal 2 — watch the NPU device:

watch -n1 'xrt-smi examine -r all -d 0000:c5:00.1'

Terminal 3 (optional) — GPU is separate from NPU:

amdgpu_top    # Radeon iGPU only, not the NPU

Kernel debugfs (low-level, requires root)

sudo ls /sys/kernel/debug/accel/
sudo ls /sys/kernel/debug/dri/

# When present (exact path varies by kernel/driver):
sudo cat /sys/kernel/debug/dri/0/telemetry_profiling
sudo cat /sys/kernel/debug/dri/0/powerstate
sudo cat /sys/kernel/debug/dri/0/get_app_health

These are read-on-demand debug interfaces, not a live dashboard.

What does NOT show NPU utilization

htop / top — CPU and RAM only
amdgpu_top / radeontop — GPU only
/sys/class/accel/accel0/ — device node metadata, no utilization graph

11. Real-world benchmark (ROG Flow Z13)

Measured on the same machine as this guide after a successful setup (Fedora 44, Ryzen AI Max 390, NPU firmware 1.1.2.65, IOMMU enabled).

`gemma4-it:e4b`

flm run gemma4-it:e4b
# /verbose enabled during session

Metric	Result
TTFT (time to first token)	1.21 s
Prefill speed	18 tok/s
Decoding speed	11 tok/s

These numbers come from FLM's /verbose output (prefill and decoding tokens/s). Your results will vary with prompt length, context size, power mode, and background load.

xrt-smi validate on the same hardware reported 4.4 TOPS (gemm), 52 µs average latency, and ~76k op/s throughput — useful as a hardware sanity check, not directly comparable to LLM tok/s.

Troubleshooting

Symptom	Cause	Fix
`libxrt_coreutil.so.2: cannot open shared object file`	XRT libs not in loader cache	Add `/opt/xilinx/xrt/lib64` to `ld.so.conf.d`, run `sudo ldconfig`
`No NPU device found` + clean dmesg	IOMMU disabled	Remove `amd_iommu=off`, reboot
`PASID unavailable and carveout not configured`	Same as above	Enable IOMMU
`Memlock limit is too low (8MB)`	Default ulimit too low	`/etc/security/limits.d/99-memlock.conf`, re-login
`xrt-smi: 0 devices found`	Missing NPU plugin	Install `xrt_plugin` RPM from `xdna-driver/build`
`/dev/accel/accel0` exists but open fails (ENODEV)	In-tree driver failed probe	Install DKMS driver via `xrt_plugin` RPM
`flm validate` OK but `flm run` fails with `No such device with index '0'`	XRT can't see NPU	Fix XRT plugin + `xrt-smi examine`
`xrt-smi: unwrapped/xrt-smi: No such file`	Bad symlink	Use wrapper script (see section 7)
`cmake3 is not installed`	Fedora CMake naming	Create `cmake3` wrapper pointing to `/usr/bin/cmake`
XRT build fails on OpenCL ICD	Fedora OpenCL 3.0 layout	See Fedora XRT build fixes below
Link errors for `libfftw3`	Missing dev package	`sudo dnf install fftw-devel`

Useful debug commands

# NPU PCI device
lspci -nn | grep 17f0

# Device node
ls -la /dev/accel/

# Kernel messages
sudo dmesg | grep -iE 'amdxdna|xdna|pasid|firmware|17f0'

# Which driver module is loaded
modinfo -F filename amdxdna
lsmod | grep amdxdna

# Firmware files (rev 11 = 17f0_11)
ls -la /usr/lib/firmware/amdnpu/17f0_11/

# IOMMU status
cat /proc/cmdline

Fedora XRT build fixes

Fedora 44 changed OpenCL packaging. Upstream XRT may fail to build or produce RPMs with wrong dependencies. Symptoms:

Compile error in ocl_icd_bindings.cpp (OpenCL 3.0 ICD struct layout)
RPM dependency conflict between ocl-icd and OpenCL-ICD-Loader

Workarounds applied in our build (track upstream fix):

Patch xrt/src/runtime_src/xocl/api/icd/ocl_icd_bindings.cpp for OpenCL 3.0 ICD compatibility
Patch xrt/src/CMake/cpackLin.cmake to require OpenCL-ICD-Loader >= 3.0 on Fedora

Upstream issue: Xilinx/XRT #9163

Install these before building if xrtdeps.sh fails on OpenCL packages:

sudo dnf install -y opencl-headers opencl-filesystem OpenCL-ICD-Loader-devel

Build flags that helped on Fedora:

./build.sh -npu -opt -disable-werror -noinit -j $(nproc)

Use -j $(nproc) with a space — some build scripts break on -j$(nproc).

Architecture: why so many pieces?

┌─────────────────────────────────────────┐
│  flm run / flm serve                    │  ← FastFlowLM (user-facing)
├─────────────────────────────────────────┤
│  libxrt_driver_xdna.so (XRT plugin)     │  ← xrt_plugin RPM
├─────────────────────────────────────────┤
│  libxrt_core.so (XRT base)              │  ← xrt-base RPM
├─────────────────────────────────────────┤
│  amdxdna.ko (DKMS kernel driver)        │  ← xrt_plugin RPM (postinst)
├─────────────────────────────────────────┤
│  NPU firmware (amdnpu/17f0_11/)       │  ← linux-firmware + plugin
├─────────────────────────────────────────┤
│  /dev/accel/accel0                      │  ← kernel DRM device node
└─────────────────────────────────────────┘
         ▲
         │ requires IOMMU (PASID/SVA)
         │ requires memlock = unlimited

Quick re-setup checklist (future you)

After a fresh Fedora install or kernel upgrade:

# 1. Confirm IOMMU is NOT disabled
grep -q 'amd_iommu=off' /proc/cmdline && echo "FIX GRUB FIRST" || echo "IOMMU OK"

# 2. Rebuild DKMS if kernel changed
sudo dkms autoinstall -k "$(uname -r)"
sudo depmod -a

# 3. Check memlock
ulimit -l

# 4. Validate
flm validate && xrt-smi examine && xrt-smi validate

References

FastFlowLM — NPU-first LLM runtime
FastFlowLM Linux docs
amd/xdna-driver — XRT + NPU plugin source
Lemonade NPU Linux guide
XRT OpenCL Fedora issue #9163
Strix Halo IOMMU discussion — GPU vs NPU trade-off

Written from a working Fedora 44 + ROG Flow Z13 setup. If AMD ships Fedora packages later, prefer those over building from source — but the troubleshooting sections above still apply.

Kriya-Egocentric-100K: Action100M-style Annotations for Real-World Labor Videos

Ankit Khandelwal — Tue, 17 Mar 2026 05:22:36 +0000

Just pushed a new preview dataset to Hugging Face: Kriya-Egocentric-100K.

It contains Action100M-compatible hierarchical action annotations for a small 5-video subset of Build AI’s Egocentric-100K — real first-person footage captured with a monocular head-mounted fisheye camera during manual labor tasks.

What’s inside?

One JSON file per video (f001-w001-0001.json etc.)
Full Action100M-style tree: root → sub-segments with precise start/end timestamps
LLM-generated natural language captions + structured GPT outputs (brief/detailed summaries, action labels, actors)
Everything generated 100 % automatically via the Kriya Full Automated Action Annotation API (early preview)

The videos themselves are not hosted here (you’ll need to pull them directly from Build AI under their license), but the annotations are MIT and drop-in compatible with the Kriya Visualizer — just load the video + matching JSON and explore the timeline instantly.

Why this matters

After the EPIC-KITCHENS preview, this is the next step toward scaling automatic annotation to more diverse egocentric domains. Manual labor footage brings new challenges (occlusions, tool use, unstructured environments) — and the results already look strong for downstream tasks like video world models, VLMs, VLA policies, and embodied robotics.

Visualizer demo, full pipeline details, and the previous Kriya-EPIC-KITCHENS release are all in the original Kriya tools blog post.

This is still an early preview — feedback and collaboration super welcome! Drop a comment or DM if you want to try the API on your own footage or discuss scaling plans.

Excited to keep pushing the boundary of automatic video understanding .

Kriya: Tools for Exploring and Generating Action100M-style Video Annotations

Ankit Khandelwal — Sat, 14 Mar 2026 06:29:49 +0000

After reading the excellent Action100M paper, I became very excited about the potential of fully automated, large-scale video action annotation.

High-quality temporal action hierarchies open doors for training stronger video world models, video-language models (VLMs), vision-language-action models (VLAs), humanoid control policies, and physical reasoning systems.

But two practical problems quickly appeared:

There was no convenient way to visualize these rich, hierarchical annotations together with the video.
Generating such annotations at scale for new/custom video datasets still felt out of reach for many researchers and engineers.

So I built two tools to help move things forward.

1. Kriya Visualizer – See Action100M-style Annotations Come Alive

I created a lightweight, static web-based visualizer specifically designed for Action100M-style temporal action trees.

Features (current version):

Video player synced with the annotation timeline
Hierarchical timeline (one row per level in the action tree)
Nodes highlight at the current timestamp
Side panel with metadata, full transcript, and raw JSON view
Clean, single-screen layout (no installation needed)

It's open source under MIT license → feel free to fork, improve, or use it in your projects.

Access Here: https://ankk98.github.io/kriya-viz/

GitHub repo: https://github.com/Ankk98/kriya-viz

If you're working with Action100M data (or any similar dense temporal action hierarchy), give it a try and let me know what features would make it more useful.

2. Kriya-EPIC-KITCHENS – Automatic Annotations on Egocentric Videos

Next, I wanted to test how well fully automatic annotation works on real, challenging egocentric data.

I ran the Kriya Full Automated Action Annotation API (early preview) on a small subset of videos from the popular EPIC-KITCHENS-100 dataset.

Result: A preview Hugging Face dataset with ~6 videos fully annotated in Action100M style, no human labeling involved.

Temporal segments with hierarchical actions
Natural language captions/descriptions per segment
Ready to download and use

Dataset link: https://huggingface.co/datasets/ankk98/kriya-epic-kitchens

Early results on kitchen egocentric videos look very promising. I'm excited to see if/how these annotations can feed downstream tasks:

Video world models
VLM / VLA fine-tuning
Robotic manipulation from egocentric views
Physical AI reasoning

The current API version deliberately follows the Action100M pipeline closely. An improved version that addresses some limitations is already in the works.

API docs (early preview): https://mindandmotionlabs.com/api-docs.html

(You send videos → get back structured temporal action hierarchies)

Why This Matters

Manual video annotation at scale is expensive and slow. If high-quality automatic annotation becomes reliable, we can:

Train on orders-of-magnitude more grounded video data
Build more general-purpose video understanding and action generation models
Accelerate progress toward capable robotic and embodied AI systems

These two small releases are just early steps. Kriya Visualizer for inspection/debugging, and Kriya-EPIC-KITCHENS as a proof-of-concept dataset.

Feedback, feature requests, collaboration ideas, or even just "I tried it and here's what broke" are very welcome!

What are you building with video action data right now? Drop a comment below 👇

From Perception to Embodied Intelligence: Evolution, Architectures, and the Humanoid Gap

Ankit Khandelwal — Sat, 14 Feb 2026 13:56:12 +0000

Vision-Language-Action (VLA) models represent a paradigm shift from passive multimodal understanding to active embodied control. This brief maps the lineage from foundational Vision-Language Models (VLMs) like CLIP and BLIP to current state-of-the-art VLA systems, revealing critical architectural transitions, data strategies, and failure modes that define the frontier of humanoid manipulation.

The analysis identifies three core evolutionary phases:

(1) VLM pre-training for semantic understanding
(2) action tokenization enabling end-to-end control
(3) hybrid architectures balancing reasoning with real-time execution

For humanoid robotics, fundamental gaps remain in proprioceptive reasoning, long-horizon planning, and physics-aware action generation, challenges that current open-source models address only partially.

1. The Evolutionary Timeline: From VLMs to VLAs

Phase 1: Foundation (2021–2022) – VLMs as Semantic Engines

CLIP (2021) and BLIP (2022) established contrastive learning as the dominant paradigm for aligning vision and language modalities. These models excelled at matching images to text descriptions but lacked any mechanism for action generation. Their legacy persists in modern VLAs: OpenVLA inherits SigLIP's vision encoder, while Pi0 leverages PaliGemma's VLM backbone. hankyukim

Key Limitation: VLMs were fundamentally passive, optimized for retrieval and classification, not sequential decision-making. Early attempts like CLIPort (2022) demonstrated that grafting CLIP representations onto robotic policies via imitation learning could achieve task-specific success but failed to generalize across embodiments or semantic concepts beyond the training distribution. arxiv

Phase 2: Tokenization Breakthrough (2023) – RT-2 and the Birth of VLAs

Google DeepMind's RT-2 (July 2023) catalyzed the field by reconceptualizing robot actions as text tokens. The architecture quantized continuous actions into discrete bins (typically 256 per dimension) and appended them to the vocabulary of a PaLM-E or PaLI-X VLM. This enabled training with standard next-token prediction objectives, unifying web-scale vision-language pre-training with robotic demonstrations. madison-proceedings

Performance Leap: RT-2 achieved 3× improvement in generalization over RT-1, demonstrating emergent capabilities like reasoning about object categories and improvising tools. The model could interpret novel commands ("place the apple on the 3") despite never observing such combinations in robot data. deepmind

Phase 3: Scaling and Open-Source (2024–2025) – OpenVLA, SmolVLA, and Pi0

OpenVLA (2024) democratized access with a 7B-parameter model trained on 970k demonstrations from the Open X-Embodiment dataset. Built on Llama 2 + DINOv2 + SigLIP, it outperformed closed models like RT-2-X (55B parameters) with 7× fewer parameters by leveraging more diverse training data and 27 training epochs (vs. typical 1-2 epochs for VLMs). arxiv

SmolVLA (2025) pioneered efficiency, achieving OpenVLA-level performance with <0.5B parameters by employing a compact VLM backbone, flow matching action expert, and asynchronous inference stack. Its key insight: action generation quality depends more on architectural efficiency than raw parameter count. youtube

Pi0 Series (Physical Intelligence, 2024–2025) introduced hybrid architectures combining autoregressive action tokens with continuous flow matching. Pi0.5 added temporal awareness through timestep conditioning, while Pi0.6 scaled to 5B parameters and incorporated knowledge insulation, training the VLM backbone on FAST tokens while isolating the action expert's gradients. arxiv

2. Thematic Deep Dives: What Worked vs. What Failed

2.1 Key Ideas That Worked

Action Tokenization as Sequence Prediction

Treating actions as discrete tokens enabled direct transfer of LLM training infrastructure to robotics. RT-2's 256-bin quantization scheme remains the default in OpenVLA, providing a simple bridge between continuous control and autoregressive generation. This approach inherits powerful properties from language modeling: in-context learning, few-shot adaptation, and chain-of-thought reasoning. arxiv

Evidence: OpenVLA achieves 95% action token accuracy after 27 training epochs, with performance correlating strongly to robot success rates. The discrete representation also simplifies multi-task training across heterogeneous robot embodiments. arxiv

Flow Matching for Continuous Control

Diffusion-based action heads address the continuity problem inherent in tokenization. Pi0 and SmolVLA use flow matching to predict action chunks as continuous trajectories, avoiding quantization errors. This enables smoother, more precise control, critical for contact-rich manipulation. youtube

Performance Impact: Pi0 outperforms tokenized baselines on action chunking tasks (e.g., folding laundry) where precise force modulation matters. Flow matching also supports variable horizon predictions, unlike fixed-length token sequences. arxiv

Knowledge Insulation and Modularity

VLA-Adapter and Pi0.6 demonstrate that decoupling VLM reasoning from action generation improves training efficiency. By freezing the VLM backbone and training only a lightweight action expert, these models avoid catastrophic forgetting of web-scale knowledge while specializing for robot control. arxiv

Efficiency Gains: VLA-Adapter trains a powerful VLA in 8 hours on a single consumer GPU, while Pi0.6's insulated gradients prevent performance degradation on vision-language benchmarks. website.pi-asset

2.2 Key Ideas That Failed

Naive Proprioception Integration

Feeding raw robot state (joint angles, end-effector poses) directly as additional tokens creates shortcut learning. Policies overfit to state-action memorization rather than visual reasoning, degrading spatial generalization. In testing, models trained with proprioception fail when object positions deviate slightly from training trajectories. arxiv

Failure Mode: A study on visuomotor policies found that proprioceptive states cause "shortcuts where the policy directly associates absolute configurations with actions," leading to 40-60% success rate drops under spatial perturbations. arxiv

Monolithic Scaling Without Architectural Innovation

Simply increasing VLM backbone size (e.g., RT-2-X's 55B parameters) yields diminishing returns for robot control. The computational overhead, 15GB GPU memory for inference at 6Hz, makes real-time deployment impractical. Larger models also struggle with action token accuracy, as the vast parameter space prioritizes language modeling over control precision. arxiv

Empirical Evidence: OpenVLA's 7B model matches RT-2-X's performance despite 7× fewer parameters, suggesting data diversity and training recipe matter more than scale. arxiv

Single-Modality Action Generation

Pure autoregressive or pure diffusion approaches each have blind spots. Autoregressive models struggle with continuous precision (quantization error), while diffusion models lack the reasoning depth of VLMs for long-horizon planning. HybridVLA attempted to combine both but introduced training interference between the two generation paradigms, requiring complex collaborative ensemble mechanisms that increased inference latency. arxiv

3. Open Source Model Comparison: OpenVLA vs. SmolVLA vs. Pi0

Feature	OpenVLA (7B)	SmolVLA (<0.5B)	Pi0.6 (5B)
Backbone	Llama 2 + DINOv2 + SigLIP	Qwen 2.5 0.5B + custom ViT	Gemma3 4B
Action Head	Autoregressive tokens (256 bins)	Flow matching (continuous)	Hybrid: FAST tokens + flow matching website.pi-asset
Training Data	970k demos (OpenX dataset)	Public community datasets	Proprietary large-scale corpus
Inference Speed	6 Hz on RTX 4090 arxiv	12.5 Hz on L40s (2.5× faster than OpenVLA) ai.stanford	5-10 Hz (denoising steps dependent)
Key Innovation	Cross-embodiment generalization	Asynchronous inference stack	Knowledge insulation + RL fine-tuning pi
Simulation Performance	62% on LIBERO-90 ai.stanford	77% on LIBERO-90 (w/ action chunks) ai.stanford	State-of-the-art on LIBERO-5 (96.5%) arxiv
Real-World Strength	Generalization across robots	Deployment on consumer GPUs	Long-horizon tasks (coffee making, laundry) pi
Critical Weakness	Slow inference, quantization error	Limited long-horizon reasoning	Proprietary, computationally intensive

Architectural Deep Dive

OpenVLA follows the RT-2 blueprint faithfully: discretize actions, append to vocabulary, train with cross-entropy loss. Its strength lies in the curated OpenX dataset diversity, enabling zero-shot control of unseen robots. However, the autoregressive generation bottleneck limits real-time performance, 15GB GPU memory and 6Hz inference constrain deployment to high-end hardware. arxiv

SmolVLA challenges the "bigger is better" orthodoxy. By using a compact VLM and flow matching action expert, it achieves comparable performance with 14× fewer parameters. The asynchronous inference stack decouples perception from action generation, allowing new chunks to be predicted while the robot executes previous commands. This is particularly impactful for dynamic environments where reaction time matters. huggingface

Pi0.6 represents the hybrid extreme: it trains the VLM backbone on FAST discrete tokens while the action expert predicts continuous flows. Knowledge insulation prevents gradient interference, and offline RL pre-training (Recap) doubles throughput on complex tasks. The model's hierarchical design supports heterogeneous prompts, enabling high-level task conditioning. The trade-off is accessibility, Pi0.6's training requires proprietary data and substantial compute, limiting reproducibility. pi

4. The Humanoid Gap Report: Missing Capabilities for Hand Manipulation

4.1 Proprioception and Tactile Integration

Current VLAs treat proprioception as auxiliary inputs, leading to shortcut learning and poor spatial generalization. Humanoid hands require fine-grained force feedback and slip detection, capabilities absent in standard VLA pipelines. arxiv

Gap: No open-source VLA integrates tactile sensing end-to-end. ForceVLA and AnyTouch explore Mixture-of-Experts for contact-rich tasks, but these remain research prototypes. The lack of large-scale tactile datasets mirrors the early scarcity of robot demonstrations. themoonlight

Opportunity: Develop a "Tactile VLA" that fuses vision, language, and distributed pressure sensor arrays. The architecture should use tactile tokens analogous to image patches, enabling the VLM backbone to reason about contact forces and friction constraints.

4.2 Long-Horizon Planning and Memory

Humanoid manipulation tasks (e.g., assembling furniture) span 5–20 minutes and require remembering partial progress. Standard VLAs operate with Markovian assumptions and fixed context windows, causing failure when intermediate steps are ambiguous. arxiv

Gap: MemoryVLA demonstrates perceptual-cognitive memory banks for manipulation, but its evaluation is limited to tabletop tasks. Humanoid whole-body control introduces additional complexity: locomotion plans must be retained while hands execute fine manipulations. arxiv

Opportunity: Implement a hierarchical memory system with (1) working memory for immediate action chunks and (2) episodic memory for task-level progress. The hippocampal-inspired consolidation mechanism from MemoryVLA could scale to humanoid tasks by encoding proprioceptive trajectories alongside visual observations. arxiv

4.3 Physics-Aware Action Generation

VLAs hallucinate physically implausible actions, predicting grasps that violate kinematic constraints or object trajectories that ignore gravity. This stems from the VLM backbone's pixel-space reasoning lacking 3D physical grounding.

Gap: GeoVLA and 3D-VLA integrate point clouds and depth maps, but these are add-ons rather than core architectural features. The models still prioritize semantic alignment over physical feasibility. arxiv

Opportunity: Embed a differentiable physics simulator within the VLA training loop. Actions could be penalized for violating Newtonian mechanics, similar to how RL uses physics-based rewards. The "visual foresight" approach in F1-VLA shows promise: predicting next visual states correlates with action reliability, suggesting that generative world models could enforce physical consistency. arxiv

4.4 Sim-to-Real for Humanoid Morphology

Humanoid robots exhibit high-dimensional action spaces (30+ DOF) and complex contact dynamics. Current sim-to-real methods rely on domain randomization, which fails to capture the nuance of bipedal balance and bimanual coordination. pmc.ncbi.nlm.nih

Gap: HumanVLA demonstrates vision-language directed object rearrangement but requires privileged state information and hand-crafted finite state machines. The sim-to-real gap remains 17% failure rate in real-world experiments, primarily due to depth sensing errors and contact estimation delays. arxiv

Opportunity: Leverage human video data as an intermediate domain. EgoVLA extracts wrist and hand actions from egocentric videos, using inverse kinematics to retarget to robot hands. This "human-to-robot" transfer could bootstrap humanoid VLA training without expensive real robot data collection. rchalyang.github

5. Critical Disagreements and Uncertainties

Disagreement 1: Proprioception's Role

Proponents: Proprioception provides compact, accurate state information essential for precise servo control. arxiv
Critics: End-to-end visuomotor policies without explicit state inputs achieve better spatial generalization, as they cannot memorize trajectories. arxiv
Resolution: The consensus is shifting toward conditioned proprioception, using state inputs only for low-level control while keeping high-level reasoning vision-driven, as seen in Helix's dual-system architecture. iotworldtoday

Disagreement 2: Action Representation

Tokenization Camp: Discrete tokens enable direct VLM transfer and chain-of-thought reasoning (OpenVLA, RT-2). arxiv
Diffusion Camp: Continuous flow matching captures action continuity and supports variable horizons (Pi0, SmolVLA). youtube
Resolution: Hybrid approaches (Pi0.6, HybridVLA) are emerging as the synthesis, but training interference remains an open problem. arxiv

Uncertainty: The optimal data mixture ratio for humanoid VLAs is unknown. RT-2 used 10% robotics data, while OpenVLA uses 100%. For humanoids, the scarcer data may require more aggressive web-scale pre-training, but this risks physics misalignment.

6. Conclusion

VLA models have evolved from passive VLMs to active embodied agents, but the leap to reliable humanoid manipulation remains incomplete. The open-source ecosystem (OpenVLA, SmolVLA) has democratized access, yet critical gaps persist in proprioceptive reasoning, long-horizon memory, and physics-aware generation.

Teleoperation Data Quality for Imitation Learning: What Actually Breaks the Model

Ankit Khandelwal — Sun, 08 Feb 2026 13:53:43 +0000

Practical rubric design and failure modes from auditing robot teleop datasets (e.g. LeRobot).

Why this post

We audited teleoperation episodes for an imitation-learning pipeline. Removing poor-quality episodes (about 20–40% in our case) led to clearly better learning; the literature often reports ~10–15% policy improvement from similar filtering. This post covers rubric mistakes that cause inconsistent scores and failure modes we kept seeing.

1. Rubric mistakes and how to fix them

Mistake 1: Metrics that sound clear but aren’t.

Example: “Mistake-to-Recovery-Ratio.” People disagree: Is it (total mistakes)/(total recoveries) or (total mistakes)/(total recovery attempts)? If a pick fails, then fails again, then succeeds, is that one recovery or two attempts?

How it should be: Define one ratio per episode. Count each distinct mistake once (each new failure event). Count a recovery only when the operator successfully got back on track; failed attempts in between don’t add extra recoveries. Write this in the rubric: “Count a recovery only when intended behavior has resumed; don’t count failed attempts as new mistakes unless it’s a new failure (e.g. new drop).” If you also want to penalize messy recoveries, add a separate “recovery attempts per mistake” number.

Mistake 2: No rule for overall quality.

Scorers give High when most dimensions are High but one is Low. Then “high quality” is not strict.

How it should be: Overall = Low if any dimension is Low; High only if all dimensions are High. One bad dimension pulls the episode down.

2. Failure modes we kept seeing

Short name (formal term) with plain-language meaning. One line each; add a screenshot or GIF per item when you publish.

Post-task idle / run-on footage (extra 10–15 s of video after the task is done). Dilutes the signal; policy can learn to linger.
Temporal misalignment (sync issues between cameras or sensors). Bad for multi-view or fusion; causes inconsistent state.
Self-collision / kinematic clash (arm hits itself or the body). Unsafe; don’t let the policy imitate it.
Low contrast / poor observability (white background, same-color object, or bad lighting). Object hard to see; weak visual signal.
Rubric incompleteness (scorers disagree or don’t know how to score). Add explicit rules and examples; flag “undefined” cases and fix the rubric before locking scores.
Repeated failures before success (e.g. 3–5 pick attempts before one works). Noisy trajectory; can teach hesitation.
Over-ideal / low-complexity conditions (too easy, no obstacles). Can bias the dataset; score complexity separately or down-weight.

3. Impact

After fixing the rubric and removing Low-quality episodes (20–40%), retraining gave noticeably better results. Studies on filtering teleop data often report ~10–15% (or more) policy gain. Define metrics and overall quality clearly, then audit before scaling data.

Summary

Rubric: Define “mistake” and “recovery” in writing; one ratio per episode. Overall quality = Low if any dimension is Low.
Failure modes: Post-task idle, sensor sync, arm clashes, poor visibility, rubric gaps, repeated failed attempts, over-ideal setup. Name them, add examples (screenshots/GIFs), score consistently.
Filtering a chunk of bad episodes is high leverage; do it before collecting more

DEV Community: Ankit Khandelwal

Agentic AI That Survives the Enterprise, Part 5: Humans in the Loop Without Burning Out Humans

The rubber-stamp problem

Designing oversight people can actually do

Your action

Agentic AI That Survives the Enterprise, Part 4: Boring Engineering Wins

The production checklist

Your action

Agentic AI That Survives the Enterprise, Part 3: The Agent With Credentials

Two attacks, one root cause

Your action

Agentic AI That Survives the Enterprise, Part 2: You Are Overbuying Intelligence

The math nobody runs

Evals before models

Your action

Agentic AI That Survives the Enterprise, Part 1: Probabilistic Engines, Deterministic Businesses

The uncomfortable truth

Six ways this actually fails

What to do before Part 2

Spec-Driven Development for Agentic Coding

Why Vibes Coding Fails

Make Generation the Last Step

Force the Hard Questions Early

Productize Context

Resolve Unknowns Before You Commit

Keep the Spine When Tools Change

What Effective Agentic Coding Looks Like

How We Scaled Firewall Logs with ClickHouse at MalCare

The Problem

Why We Left MongoDB

What We Learned from ClickHouse's Design

Migrating from MongoDB

Schema and Indexing

The Hidden Disk Hog: system.query_log

Hardware

Rails and MySQL Integration

Read-Only Access and Query Control

Backup and Archival

Closing Notes

References

Running LLMs on AMD NPU with FastFlowLM - Fedora Guide

Running LLMs on AMD NPU with FastFlowLM - Fedora Guide

TL;DR

Hardware tested

Before you start

Kernel requirements

Check your kernel cmdline now

1. Install build dependencies

cmake3 wrapper (Fedora ships CMake 4.x as cmake)

2. Build and install XRT

Register XRT libraries system-wide

3. Build and install the NPU plugin (DKMS driver + XRT shim)

4. Fix memlock limit

5. Critical: do NOT use amd_iommu=off

What is amd_iommu?

Symptoms with IOMMU off

Fix

6. Build and install FastFlowLM

7. Make xrt-smi available (optional but useful)

8. Validate everything

9. Run your first model

10. Monitor NPU stats

Quick reference

XRT — device-level snapshots

FLM — inference metrics (most useful in practice)

Kernel debugfs (low-level, requires root)

What does NOT show NPU utilization

11. Real-world benchmark (ROG Flow Z13)

gemma4-it:e4b

Troubleshooting

Useful debug commands

Fedora XRT build fixes

Architecture: why so many pieces?

Quick re-setup checklist (future you)

References

Kriya-Egocentric-100K: Action100M-style Annotations for Real-World Labor Videos

What’s inside?

Why this matters

Kriya: Tools for Exploring and Generating Action100M-style Video Annotations

1. Kriya Visualizer – See Action100M-style Annotations Come Alive

The Hidden Disk Hog: `system.query_log`

cmake3 wrapper (Fedora ships CMake 4.x as `cmake`)

5. Critical: do NOT use `amd_iommu=off`

What is `amd_iommu`?

7. Make `xrt-smi` available (optional but useful)

`gemma4-it:e4b`