DEV Community: Vladyslav Donchenko

Alibaba Just Built a GUI Agent That Tops Every Benchmark. Here's What It Actually Does.

Vladyslav Donchenko — Sat, 01 Aug 2026 16:49:11 +0000

Benchmark leaderboards are usually unreliable signals. Models overfit to test distributions, evaluation setups differ, and headline numbers rarely translate to production behavior. That's why the new technical report from Alibaba's MAI-UI team deserves closer reading than most: Qwen-UI-Agent doesn't just top a leaderboard. It tops six different leaderboards, on mobile and desktop and web, with a methodology that's specifically designed to eliminate the simulation-to-reality gap that inflates most agent scores.

The numbers: 92.2% on MobileWorld-Real, 97.5% on AndroidDaily, 73.6% on WebArena, 81.5% on ScreenSpot-Pro. Across all six benchmarks in the report, Qwen-UI-Agent either leads or places second. GPT-5.6-Sol, Claude Opus 4.8, and Gemini 3.1 Pro trail by meaningful margins — 7–14 percentage points on mobile tasks.

The interesting question is why. The architecture makes the answer legible.

The Core Problem: Simulation Gaps

Most mobile agent research runs on emulators or sandboxed environments. This is practical — you can run thousands of experiments in parallel, reset state reliably, and avoid the logistics of a physical device fleet. But emulated environments differ from real devices in ways that compound quickly: rendering differences, timing behavior, app states that don't exactly replicate real-world conditions, accessibility tree variations between Android versions.

Qwen-UI-Agent addresses this directly with a cluster of 100+ physical phones supporting 150+ real applications. The training trajectories come from actual device interactions — not simulated ones. The performance gap between MobileWorld (82.1%) and MobileWorld-Real (92.2%) in the benchmarks captures exactly this: the real-device variant is harder, and Qwen-UI-Agent's score goes up rather than down on it, which is the opposite of what you'd expect from a model trained primarily in simulation.

The practical infrastructure required to do this is non-trivial. The paper describes a virtual-screen mechanism that multiplexes single devices across multiple concurrent sessions, achieving roughly 20× throughput improvement for training rollout generation. Without this, the cost of real-device training at scale would be prohibitive.

Hybrid GUI + CLI Actions

Most GUI agents operate entirely through the visual interface — they see a screen, decide what to tap or click, execute the action, observe the new state. Qwen-UI-Agent adds a second interaction modality: CLI commands and API calls within the same trajectory.

This matters because many tasks are genuinely easier to accomplish through the command line or an API than through the GUI. File management, configuration changes, data extraction — in practice, a human power user would reach for the terminal rather than navigate through nested UI menus. The agent can make the same choice.

The result is measurable: hybrid GUI+CLI trajectories are 21–58% shorter than GUI-only baselines for the same tasks. The agent solves problems faster, uses fewer steps, and covers task types that GUI-only agents can't handle at all. The unified action space includes click, type, long-press, drag, system-button, cli-command, api-call, ask-user, and terminate — all available in a single trajectory.

More than 40% of actions in the training data are batched — multiple operations grouped into single steps. This reduces round-trip overhead and makes long-horizon tasks more tractable.

Training: Three Layers

The training pipeline has three components that address different failure modes.

Supervised Fine-Tuning (SFT) uses domain-conditioned expert trajectories with checkpoint merging to preserve general reasoning capabilities. The key challenge with SFT for long-horizon tasks is that trajectories exceeding 100 steps don't fit in standard training windows — the paper addresses this with a sliding-window approach using 5-step windows and 4-step advancement, which lets the model learn from extended trajectories without catastrophic context truncation.

Action RL targets six recurring failure patterns that SFT doesn't eliminate on its own: confusable-element grounding (tapping the wrong element when similar elements appear nearby), sorting misinterpretation, quantity completion failures, premature task termination, repetitive loops, and long-tail action selection. These patterns emerge from empirical failure analysis rather than theoretical assumptions, which is why they're specific enough to be actionable.

Online RL uses group-relative policy optimization (GRPO) for complete trajectory-level learning — the model optimizes against verified task outcomes, not just step-level correctness. A model-adaptive task curriculum adjusts difficulty dynamically based on current policy performance: when the model masters easy tasks, difficulty increases; when performance on harder tasks drops, previously-mastered tasks are reintroduced to prevent forgetting. Around 10,000 task-verifier pairs were constructed automatically to support this.

The Data Flywheel

What makes scaling this kind of training feasible is the automated data pipeline. Human-labeled training data for agent trajectories is expensive to produce: you need domain experts, real device access, and careful trajectory review. The paper describes a flywheel that reduces this bottleneck significantly.

The flywheel combines knowledge- and capability-aware task synthesis (generating new tasks targeted at current model weaknesses), environment synthesis (automatically constructing the app states and verifiers needed to evaluate those tasks), step-level VLM judging to produce SFT signals without human review, and failure-driven iteration that identifies systematic error patterns and generates targeted training tasks to address them.

This is the infrastructure that lets the team scale real-device training across 150+ applications without proportional increases in human annotation effort. The agent effectively generates a significant portion of its own training curriculum.

Cross-Platform and Proactive Behavior

Two capabilities that extend beyond single-device GUI automation:

Cross-platform execution uses a hierarchical planner-executor architecture that maintains shared state across mobile and desktop environments. Multiple agent instances run in parallel on virtual displays, coordinating through a shared context. This allows workflows that span devices — starting something on mobile, continuing on desktop — without losing state or context.

Proactive service is the most architecturally novel component. Rather than waiting for explicit user requests, the system monitors mobile notifications, parses them into structured events, reasons about what assistance might be relevant, and prepares decision-ready proposals. The agent identifies moments when it could help — a calendar notification triggering a workflow, an email prompting a follow-up action — before the user has articulated the need.

This shifts the interaction model from reactive to proactive: the agent as a system that anticipates rather than waits.

What the Benchmarks Actually Show

Looking at the benchmark results in full context:

MobileWorld-Real (92.2%): Real-device benchmark with 400+ tasks across 100+ apps. The closest competitor, Seed 2.1 Pro, scores 88.7%. Claude Opus 4.8 scores 84.7%.
AndroidDaily (97.5%): Near-ceiling performance on daily Android task automation. Seed 2.1 Pro trails at 95.2%.
MobileWorld (82.1%): The simulated counterpart. Qwen-UI-Agent leads here too, but the margin is smaller — which is expected when the real-device training advantage doesn't transfer as directly to simulated evaluation.
OSWorld-Verified (79.5%): Desktop computer-use. Second place here, behind Claude Opus 4.8 at 83.4% — the one benchmark where another model leads.
WebArena (73.6%): Web agent tasks. Leads by 1.7 points over Claude Opus 4.8's 71.9%.
ScreenSpot-Pro (81.5%): GUI element grounding precision. Leads by 0.8 points over Seed 2.1 Pro's 80.7%.

The OSWorld result is notable precisely because it's the exception. Desktop computer-use is where Claude Opus 4.8 currently leads, which suggests that the real-device mobile training advantage doesn't transfer as cleanly to desktop environments — or that Anthropic's desktop-specific training has produced something Qwen-UI-Agent hasn't fully matched yet.

What This Means for the Field

A few observations worth drawing out:

Real-device training is the differentiator. The 92.2% on MobileWorld-Real versus the industry average on simulated benchmarks illustrates the gap that simulation training leaves. If mobile agent capability matters for your use case, the training distribution matters more than the model architecture.

Hybrid action spaces expand task coverage substantially. The 21–58% trajectory length reduction from GUI+CLI isn't just efficiency — it's evidence that there are task categories where GUI-only agents can't compete. An agent that can switch to the terminal when appropriate is a different kind of tool.

Infrastructure scales at least as much as models. The 100+ real-device cluster, the 10,000-environment online RL setup, the automated data flywheel — this is a research infrastructure paper as much as a model paper. The benchmark results are downstream of those investments, not primarily of architectural innovation.

Proactive agents change the usage model fundamentally. Notification-triggered task initiation means the agent surface isn't a chatbox you open when you need something — it's a background process that monitors context and proposes actions. The implications for how we think about agent UX are significant and mostly unexplored.

For anyone building or evaluating mobile or GUI automation: the benchmark numbers from Qwen-UI-Agent set a new practical reference point. The architecture explains why simulation-only training leaves performance on the table, and the data flywheel provides a template for how to scale real-device training without proportional human annotation cost.

Paper: arxiv.org/abs/2607.28227 · MAI-UI Team, Alibaba Group

Originally published at vsebude.it

Your Coding Agent Passes the Tests. Watch What Happens When It Has to Extend Its Own Code.

Vladyslav Donchenko — Thu, 30 Jul 2026 20:04:27 +0000

Your coding agent scores well on SWE-bench. You ship it into a real workflow where it has to extend its own code across multiple sprints. What happens?

SlopCodeBench just measured exactly that — and the results should change how you think about coding agent deployments.

The Benchmark Nobody Was Running

SlopCodeBench (UW-Madison, MIT, Washington State) tests 11 coding agents across 20 problems and 93 checkpoints. At each checkpoint, the agent receives an updated spec and must extend its own prior code — not a reference implementation. Its architectural decisions at checkpoint 1 become the foundation for checkpoint 2, and so on.

They tracked two quality signals at every step:

Structural erosion — fraction of complexity mass concentrated in high-complexity functions (cyclomatic complexity > 10)
Verbosity — redundant/duplicated code as a fraction of total lines

The Results

No agent solved any problem end-to-end. Zero. Across all 11 models.

Best checkpoint solve rate: 17.2%
Structural erosion rose in 80% of trajectories
Verbosity rose in 89.8% of trajectories
Agent code is 2.2x more verbose than 48 maintained open-source Python repos
Human repositories stay roughly flat — agent code deteriorates with each iteration

The Prompt That Didn't Fix It

Quality-aware prompts reduce initial verbosity and erosion by up to a third. But they don't slow the degradation rate, improve pass rates, or reduce cost. Agents start cleaner and degrade at the same rate.

Why This Matters

Pass-rate benchmarks (SWE-bench, HumanEval) are single-shot. They don't give the agent's code a future it has to build on. If your agent is writing code that will be extended — by itself, by other agents, or by your engineers — those scores tell you almost nothing about how that code will age.

Production implications:

Architectural review can't be automated away — complexity concentrating in hot functions is exactly what human architects catch
The longer the horizon, the less benchmark scores predict real performance
2.2x verbosity isn't aesthetic — it's more tokens per turn, more bug surface, more context cost
Quality prompting is a partial mitigation, not a fix

Pass-rate is what you optimize for when building demos. Extension robustness is what you need when building software. Only one is being measured.

Paper: https://arxiv.org/abs/2603.24755 · Leaderboard: https://www.scbench.ai

Originally published at vsebude.it

An AI Agent Canceled Every Stripe Subscription in 7 Seconds. The Model Isn't the Problem.

Vladyslav Donchenko — Wed, 29 Jul 2026 01:35:29 +0000

An AI agent wiped a founder's entire MRR while they slept. The viral post blamed the model. The real lesson is about architecture.

Last week, @bridgemindai woke up to find their MRR had collapsed to $38. A cron job written by GPT-5.6-Sol (Codex) had canceled every active Stripe subscription in 7 seconds. No customers had left. The code just ran.

The take that went viral: "GPT 5.6 cannot be trusted. Fable 5 has never done this to me."

That framing will get you burned by whichever model you switch to next.

What Actually Happened

The bug: a cron job that treated an empty deletion queue as input to process everything. Empty queue → delete all. This is a logic error developers have been writing since the 1980s. The AI generating the code is almost irrelevant to the root cause.

What matters: that code had unguarded access to a live production Stripe API key, was scheduled to run autonomously, and had no human checkpoint between "AI generated this" and "this ran on your entire customer base."

The Real Failure: No Safety Rails on Destructive Operations

Three questions every agentic pipeline touching production must answer:

What operations are irreversible? Canceling subscriptions, deleting records, sending emails — these need different treatment than reads.
What credentials does the agent hold? A Stripe key that can cancel subscriptions should never be handed directly to an autonomous agent. Scoped, restricted keys exist for this.
What's the human checkpoint before destructive actions execute? "The code looked right when I reviewed it" is not a checkpoint.

Three Rails That Would Have Stopped This

Dry-run gates with non-empty checks. Before deleting or canceling, log what would be affected. If the target set is empty or unexpectedly large — abort, alert, never proceed. Two lines of code.

Scoped credentials, not production keys. AI agents get minimum Stripe permissions. Read-only by default. Write access requires explicit escalation. Stripe restricted keys exist for this.

Human-in-the-loop for irreversible ops. A Slack message: "I'm about to cancel 47 subscriptions. Confirm?" Three seconds to respond. Costs nothing. Makes autonomous agents safe to run while you sleep.

Why Model-Blaming Is a Trap

Every current AI model can write a cron job with a logic error. The rate varies. The risk doesn't disappear. Matt Shumer posted the same day that GPT-5.6-Sol deleted almost all his Mac's files — same pattern, different product, same missing rails.

The common thread isn't the model. It's the architecture.

Production agentic pipeline checklist:

Classify every operation: read / write-reversible / write-irreversible
Require explicit human approval for all write-irreversible operations
Scope credentials to minimum necessary permission level
Gate deletion loops on non-empty, size-bounded target sets
Run in sandbox with production-mirrored data first
Log everything the agent plans to do, in plain language, before execution

Build the rails before you run the agent — not after you wake up to an empty dashboard.

Originally published at vsebude.it

RAG Retrieves. Fine-Tuning Forgets. HyperNetworks Inject - and Now We Have the Scaling Laws.

Vladyslav Donchenko — Tue, 28 Jul 2026 10:47:30 +0000

There are two mainstream answers to "how do I get my LLM to reliably use proprietary data?" RAG retrieves relevant chunks at inference time. Fine-tuning bakes new knowledge into the weights. Both are widely deployed. Both have well-documented failure modes.

A new paper from Nace AI and Purdue University proposes a third path backed by the first systematic scaling laws: HyperNetwork-based knowledge injection. Instead of modifying the LLM, you train a second network that generates LoRA adapters from a batch of facts. The base model never changes.

Why Fine-Tuning Is a Harder Problem Than It Looks

Catastrophic forgetting. Updating weights on new facts degrades existing capabilities.
OOD generalization failure. Fine-tuned models struggle with new entity combinations not seen during training.
Instability at scale. Knowledge editing methods break unpredictably past ~1,000 injected facts.
Hallucination amplification. New facts that contradict pretraining increase hallucination rates.

LoRA reduces cost but shares the same structural problems. You still forget.

The HyperNetwork Alternative

A HyperNetwork is a secondary model conditioned on input facts. It generates LoRA adapters (ΔW) inserted into the target model at inference time. The base model is frozen. Always.

Architecture: Fact batch enters a Transformer encoder (no causal mask, Post-LayerNorm, RoPE) -> Mean pooling -> Linear -> ΔW (LoRA rank 4, α=8) -> adapters into upper half of target model layers. Only the HyperNetwork trains.

The Scaling Laws

The team built MegaWikiQA -- 10M+ multi-hop QA pairs from Wikidata5M (4.6M entities, 822 relations, 39 domains, 1-4 hops, explicit OOD splits).

Hypernetworks from 167M to 2.8B parameters. Four findings:

Power-law scaling everywhere. Loss, accuracy, and OOD generalization follow smooth power laws.
Scaling the target model beats scaling the hypernetwork. Upgrade the LLM, not the injector.
Depth ≈ Width for hypernetwork scaling. Comparable improvements per parameter budget.
Better OOD generalization than LoRA fine-tuning at scale. Steeper OOD scaling -- gap widens at larger target model scale.

What This Means for Enterprise AI

Most enterprise teams face this: proprietary knowledge that can't go into pretraining, too dynamic for static fine-tuning, too specific for general RAG.

The hypernetwork approach: no base model modification, predictable scaling, OOD generalization to new entities and regulations, independently upgradable layers.

The Bigger Picture

RAG retrieves. Fine-tuning forgets. HyperNetworks inject -- learning to generate weight adaptations from facts, not memorizing facts themselves. The scaling laws make this a design principle: steeper OOD scaling exponents, gap widening at scale, smooth power laws you can engineer around.

Paper: https://arxiv.org/abs/2607.19604 | Code and data: https://huggingface.co/collections/nace-ai/hypernetwork-datasets

Originally published at vsebude.it

Sakana AI's Fugu-Cyber Tops the Cybersecurity Benchmarks. Their Bigger Argument: The Model Isn't the Hard Part.

Vladyslav Donchenko — Mon, 27 Jul 2026 04:20:26 +0000

Sakana AI just released Fugu-Cyber - a multi-agent orchestration model that scores 86.9% on CyberGym and 72.1% on CTI-REALM, beating GPT-5.5-Cyber and Mythos Preview on both benchmarks. That's the headline.

Sakana's more interesting argument is buried in the announcement: the benchmark score is not the product.

What the Benchmarks Actually Test

CyberGym and CTI-REALM aren't toy evals. CyberGym tests an agent's ability to analyze real codebases and verify real-world vulnerabilities. CTI-REALM measures whether a model can take raw threat intelligence reports and turn them into working detection rules.

These are hard problems. Fugu-Cyber tops both, ahead of two frontier models purpose-built for exactly these tasks.

The architecture: Fugu-Cyber is a multi-agent system that presents as a single API endpoint. You send one request. The system routes it dynamically to a pool of specialized sub-agents - vulnerability analysis, threat intel parsing, adversarial verification, code review - orchestrates the work, and returns a consolidated result. No single-vendor dependency. No orchestration layer to manage yourself.

The Reality Check Nobody Else Is Saying

Sakana's announcement includes an unusually candid section for a model launch. They cite a Nikkei Digital Governance report noting that major Japanese financial institutions - even with access to frontier cybersecurity models - struggle to operationalize them. Without internal talent, without deep integration into proprietary source code, a high-benchmark model doesn't find real vulnerabilities in a real environment.

Their framing: "A highly capable API with strong cyber reasoning is an incredibly important piece of the puzzle. It is not the entire solution."

This is the honest version of what's true about most AI security tooling today. Raw models generate false positives. They don't understand the nuances of your production environment. Without harnesses - specialized verification sub-agents, human-in-the-loop validation, domain-specific context - even a benchmark-topping model is noisy in practice.

The Layer That Actually Matters

What Sakana is building isn't just the model. It's the infrastructure between frontier capability and enterprise deployment:

Adversarial verification: Before a patch is proposed, a sub-agent checks whether the vulnerability would actually trigger in a real environment
Human-in-the-loop validation: Security professionals embedded in the workflow validate findings before action is taken
Enterprise harnesses: Direct work with major Japanese institutions to build production-grade context layers around the model

This is the same architecture insight that makes Visa's VVAH interesting: the bottleneck in AI security isn't the model's reasoning capability. It's the verification layer around it.

Access and Responsible Deployment

Fugu-Cyber isn't open access. Users must submit an application, provide verified contact info, and pass manual review before API access is granted. The updated AUP explicitly prohibits offensive use cases.

The benchmark results are real. The more useful takeaway: a score is a starting point, not a solution. The teams building harnesses around these models - not just calling the API - are the ones getting results in production.

Originally published at vsebude.it

Visa Open-Sourced an 11-Stage AI Security Pipeline. The Key Insight Isn't Discovery - It's Triage Speed.

Vladyslav Donchenko — Sun, 26 Jul 2026 18:24:46 +0000

The natural assumption about AI-powered security tooling is that the hard part is finding vulnerabilities. Visa's newly open-sourced VVAH (Visa Vulnerability Agentic Harness) is built around a different premise. The hardest part isn't discovery. It's triage.

What VVAH Is

VVAH is a four-phase, eleven-stage pipeline from code repository to validated fix. Built on Project Glasswing (Anthropic's initiative for AI-assisted vulnerability research), it works with Claude, OpenAI-compatible models, or both.

Primary metric: MTTA - Mean Time to Adapt (AI-discovered exploitability ? validated fix in production).

Four phases:

Phase 1 - Discovery & Modeling (S1-S3): Attack surface mapping, STRIDE threat model, prioritized hunt plan
Phase 2 - Deep Dive & Verification (S4-S6): Multi-lens analysis (6 specialized roles), policy gates, adversarial verification
Phase 3 - Synthesis & Reporting (S7-S9): Deduplication, exploit chain construction, SARIF 2.1.0 emission
Phase 4 - Remediation & Validation (S10-S11): Minimal fix proposals, adversarial panel validation

The Triage Design

The key structural choice: deterministic controls sit before expensive downstream LLM work. S5 applies policy gates; semantic dedup fires only when survivors exceed a threshold (default 25). Then S6 runs adversarial verification - a separate LLM pass constructing exploit chains and tracing trust boundaries.

Findings that survive S5+S6 are confirmed exploitable. S7-S9 chain and emit SARIF over a clean signal.

Skills as the Modularity Unit

Each LLM-driven stage is a composable, independently replaceable skill. Pass --stop-after s9 for detection only (no code edits). The default profile edits source files in your target repo via S10 fix mode - document this before onboarding teams.

Multi-Model by Design

Three backends: Anthropic Python SDK, claude CLI subprocess, OpenAI-compatible API. Detection pipeline (S1-S9) is backend-agnostic. Full remediation + validation (S10-S11) currently requires Anthropic models.

Why This Architecture Matters

The same pattern scales to any agentic pipeline needing to convert AI signals into validated outputs:

Threat model before analysis
Adversarial verify before synthesis
Measurable metric (MTTA) connecting AI capability to operational velocity
Composable skills tunable per language/framework/CWE

Repo: github.com/visa/visa-vulnerability-agentic-harness

Originally published at vsebude.it

The Hidden Bottleneck in Agent Evolution: What Harness Handbook Reveals About How AI Systems Break

Vladyslav Donchenko — Sat, 25 Jul 2026 16:42:53 +0000

The real estate operations team had been chasing the same bug for three days. Their lease extraction agent was silently skipping certain clause types. The fix, once found, took two hours. Finding it took three days.

A new paper from Tencent and Indiana University finally names this pattern: behavior localization - the process of identifying every code location that implements the behavior you need to change. It is the hidden bottleneck in every production agent system, and it scales in cost with the complexity of your harness.

The Core Problem

Production agent harnesses are behaviorally distributed. A single observable behavior - how the agent handles a timeout, or parses a specific document format - may be implemented across several functions, multiple files, and different execution stages, linked by shared state that is never visible in one place.

Repositories are organized by files and modules. Modification requests describe behaviors. The gap between those two views is where engineering time disappears.

The paper proposes Harness Handbook: a behavior-centric representation of the codebase, built automatically via static analysis and LLM-assisted structuring, that maps what the harness does to where it is implemented.

Three-Level Progressive Disclosure

The Handbook is organized as a hierarchy:

L1 (System Overview) - architecture, execution model, major stages, global data flow
L2 (Component Overview) - per-stage responsibilities, inputs, outputs, dependencies
L3 (Unit Deep Dive) - source-grounded implementation entries, each linked to a validated live code location

Every L3 entry must resolve to a real location in the current source. If the code changes and a locator no longer points anywhere, it is frozen and excluded from localization results until refreshed. The repository is always authoritative.

Behavior-Guided Progressive Disclosure (BGPD)

The Handbook enables a workflow the paper calls BGPD: instead of grep-ing the repository and guessing, a developer or coding agent starts at L1, descends to the relevant stage at L2, follows links to L3 source entries, and verifies those entries against live code before planning any edit.

Evaluated on realistic modification requests across two open-source harnesses, Handbook-Assisted planning improved behavior localization accuracy and edit-plan quality - while using fewer tokens at the planner. The largest gains appeared on cross-module changes, rarely-executed paths, and behaviors distributed across nonadjacent locations.

Why This Matters for PropTech

Production real estate platforms run complex harnesses - document extraction, compliance enforcement, tenant communications, maintenance dispatch - coordinated across dozens of files and execution stages.

When a CRM API changes, a new lease format appears, or a regulatory requirement shifts, the first challenge is always finding every place in the harness that touches the affected behavior. Harness Handbook makes that navigation automatic and verifiable.

Model swaps. A cheaper model replaces the current one. The Handbook surfaces every location where prompts, context windows, or tool-call formats need to adapt.

API evolution. An external integration changes its response format. The behavior "call this system and parse its response" may span an adapter, retry policy, schema validator, and error handler - in four files. The Handbook finds all four before the first edit.

Regulatory changes. New compliance requirements touch verification flows, data retention, logging, and output formatting across different execution stages. Missing one is a compliance failure.

Coding agent delegation. As teams delegate harness edits to coding agents, the quality of behavior localization determines whether the agent's edit plan covers all affected locations. A Handbook-equipped coding agent produces more complete plans with fewer wasted searches.

The Honest Caveats

Construction has a real cost - static analysis plus LLM inference. The paper argues it is recovered through token savings at edit time, but it is upfront investment.

The Handbook must stay current. Stale L3 entries with frozen locators give incomplete results, which may be worse than no Handbook if developers trust it uncritically. Resynchronization triggers on every non-empty diff.

Behavior localization is necessary but not sufficient. Better planning does not fix a bad agent or a broken evaluation gate.

Connection to Self-Improving Harnesses

Harness Handbook is the navigation layer that makes harness evolution loops practical. Before you can improve a harness - whether through Self-Harness-style automated editing or manual review - you need to find where the target behavior actually lives. This paper solves that prerequisite.

The production development loop becomes: observe failures ? localize behavior via Handbook ? propose targeted edits ? validate against held-out benchmark before shipping.

The Takeaway

The quality of your harness documentation is not a nice-to-have. It is the bottleneck that decides how quickly your system can adapt and how safely your coding agents can evolve it.

Originally published at vsebude.it

To Every Agent Its Own Database: Why the Shared Data Warehouse Is the Wrong Model for AI Agents

Vladyslav Donchenko — Wed, 22 Jul 2026 14:46:18 +0000

The data warehouse was designed for humans. It assumed people would write queries in the morning, wait for dashboards to refresh, and debate metrics in meetings. It was built for a world where analytical work happened at human pace, in human time, with humans in the loop to catch mistakes.

That world is ending. Joe Reis just showed us what comes next.

What Joe Built

In a recent prototype, Reis inverted the entire premise of centralized analytical infrastructure. Instead of routing every agent query through a shared warehouse, he gave each active agent its own embedded analytical engine - a local DuckDB instance, exposed via Quack (DuckDB's native client-server protocol) over TCP loopback. Agents don't query a central system. They each are a data system.

But the real elegance is in how they exchange data. Reis introduced a structure he calls a Data Exchange Unit: an immutable analytical "slice" identified by a versioned reference that bundles together a catalog ID, dataset name, snapshot identifier, semantic contract digest, and content digest. When one agent produces results, another agent doesn't query a shared table - it receives a cryptographically identified slice, checks the contract, and either accepts it or rejects it.

The semantic layer is built on Malloy models: typed sources with governed query definitions. Each model's hash becomes part of the slice's cryptographic identity. If you change the revenue calculation - even subtly, even from sum(qty * unit_price) to sum(qty * unit_price) * 0.9 - the contract digest changes, the slice reference changes, and every downstream consumer knows immediately that something is different. Not because a human caught it. Because the math caught it.

Why the Shared Warehouse Breaks Under Agentic Workloads

Shared warehouses were optimized for a specific access pattern: a relatively small number of humans running moderately complex queries on a regular schedule. The warehouse could be tuned around those patterns. Concurrency was manageable. Errors were caught by people who understood the domain.

Agent workloads destroy all three of those assumptions.

Agents don't take turns. In a multi-agent pipeline, dozens of agents might fan out simultaneously, each running speculative queries, generating intermediate results, passing those results to the next layer, and producing yet more queries. The concurrency model of a centralized warehouse - designed for a team of analysts - is not ready for a fleet of parallel reasoning systems.

Agents chain intermediate results rapidly. In human workflows, intermediate data often lives in someone's head or a scratch spreadsheet. In agent workflows, intermediate results are first-class artifacts that need to be passed, versioned, and consumed programmatically. A shared warehouse with mutable tables is a terrible substrate for this - you can't version a shared table, and you can't guarantee that the result one agent reads is the same one another agent wrote.

Most critically: there are no humans checking the work. The entire value proposition of agents is that they operate autonomously. But the warehouse model was built with the assumption that humans would catch semantic drift, notice when a metric definition changed, or flag when a join produced unexpected cardinality. Remove the human, and you remove the error-checking layer the warehouse was designed to rely on.

The Elegant Inversion

What Reis built isn't just a different architecture - it's a different mental model.

Traditional thinking: take the warehouse, make it agent-ready (add APIs, add rate limits, add caching, add observability).

Reis's thinking: make agents warehouse-native. Give each agent its own analytical engine. Let them exchange data peer-to-peer through contracts that are cryptographically enforced.

The inversion matters because it changes where trust lives. In the centralized warehouse model, trust is institutional - you trust the warehouse because the data team built and maintains it. In the Reis model, trust is cryptographic - you trust a data slice because its provenance DAG, semantic contract digest, and content hash are all verifiable, deterministically, without appealing to any human authority.

This is a profound shift. It means an agent receiving data from another agent doesn't have to trust the producing agent as an institution. It can verify the contract mathematically.

What Contracts Agents Need

Reis's prototype implements a three-layer contract system that deserves to become a standard pattern.

The semantic contract (Malloy models) establishes meaning - what does "revenue" mean, what does "active user" mean, what grain is this data at? The structural compatibility layer classifies any change as IDENTICAL, ADDITIVE, BREAKING, or UNRELATED, and Reis is explicit that this classification must be deterministic: Using an LLM to decide whether a contract change is IDENTICAL, ADDITIVE, or BREAKING would make the compatibility boundary probabilistic. The intent contract lets consumers declare what they need, and a resolver converts that mutable request into a pinned, immutable reference.

This is what data contracts should have always been. Not documentation. Not gentleman's agreements between teams. Cryptographically enforced, machine-checkable, version-pinned, with lineage carried inside the artifact itself.

What This Means for Data Architecture

The implications are bigger than DuckDB. What Reis is pointing at is a future where analytical data architecture looks less like a central nervous system and more like a distributed ledger of immutable facts exchanged between autonomous agents.

The data team's role doesn't disappear - it shifts. Instead of owning the warehouse and managing access, data engineers become the authors of semantic contracts: the Malloy models, the provenance schemas, the trust hierarchies that define what WAREHOUSE-tier data means versus DERIVED-tier data. The warehouse becomes one authoritative principal among many, not the single source of truth.

For data architects, this is both liberating and destabilizing. Liberating because the impossible mandate - maintain a single consistent semantic layer for all possible consumers at all possible scales - finally has an escape hatch. Destabilizing because every assumption about governance, tooling, and team structure needs to be revisited.

The shared data warehouse was designed for a world where humans asked questions. The agent-native data layer Reis is prototyping is designed for a world where machines answer them.

We should start building for that world now.

Vladyslav Donchenko is a software engineer and founder of vsebude.it. He writes about AI systems, data infrastructure, and the engineering decisions that shape how intelligent systems are built.

Originally published on Medium.

"From Chatbot to Teammate: What Claude Tag Signals for How We Work"

Vladyslav Donchenko — Tue, 30 Jun 2026 07:05:23 +0000

Most AI at work still behaves like a chatbot in a side panel: you open it, you ask, you copy the answer back into wherever the work actually lives. Anthropic's newly announced Claude Tag points at a different model — and it is worth understanding regardless of which vendor you use.

The idea: Claude joins your team. Starting in Slack, you grant it access to selected channels and connect it to chosen tools, data, and codebases. Then anyone can tag @claude and hand off a task while they do other work. Anthropic says 65% of its product team's code is now created by an internal version of this.

What makes a "tagged agent" different from a chatbot

Multiplayer — one agent per channel that everyone shares; anyone can see what it is doing and pick up where a colleague left off.
Learns over time — it builds context from the channels and data it is permitted to see, so people stop re-explaining background on every request.
Takes initiative — with "ambient" behaviour on, it proactively flags things and follows up on threads/tasks that went quiet.
Works asynchronously — set it a task and it works while you focus elsewhere; it can schedule its own work over hours or days, many agents in parallel.

That is closer to a coworker than a tool.

The hard part was never the model — it's governance

An AI teammate with access to your channels, tools, and data is exactly as useful as it is risky if the access model is sloppy. The most important detail in the announcement isn't capability — it's control:

Admins scope which tools and information the agent can use, per channel.
Each setup is effectively a separate identity — a sales-configured agent never leaks memory or data into an engineering one.
Token-spend limits per org and per channel.
An audit log of everything the agent did and who requested it.

That list is the real checklist for deploying any agent in a regulated, data-sensitive business. Before you tag an agent into a channel touching customer PII, financials, or contracts, you need scoped least-privilege access, memory boundaries between contexts, full audit trails, spend caps, and human approval on irreversible actions.

The governance and orchestration layer — not the underlying model — is what turns an impressive demo into something you can run in production.

The takeaway

Claude Tag matters less as a single product and more as a marker of where applied AI is heading: agents that live inside your workflows, accumulate context, and act on their own initiative within tight guardrails. If you want an AI teammate in your operational channels, start with the access model, the audit trail, and the orchestration around it — not the demo.

Full version, with the real-estate / PropTech angle, on the VSBD blog. Source: Anthropic — Introducing Claude Tag.

"LLM Inference Optimization: The Line Item That Decides If Your AI Ships"

Vladyslav Donchenko — Mon, 29 Jun 2026 07:05:33 +0000

Training gets the headlines. Inference gets the bill. If you run LLMs in production, inference is almost certainly your biggest AI line item — a meter running 24/7 on every request. The gap between naive and optimized serving is routinely 5-10x in cost and 3-5x in latency.

The bottleneck is memory, not compute

During token generation, LLM inference is memory-bandwidth bound. An H100 has ~3.35 TB/s bandwidth but ~989 TFLOPS FP16 compute — during autoregressive decoding you're using only ~10-20% of that compute, waiting on weights and KV-cache to stream from memory. Every optimization attacks the same root cause: move less data, use it better.

The levers that matter

KV cache. It's often bigger than the weights. PagedAttention (vLLM) pages the cache like OS virtual memory, dropping waste from 60-80% to near-zero → 2-3x more concurrent requests. Prefix caching reuses the KV for shared system prompts / few-shot / RAG context. GQA shrinks the cache at the architecture level.
Continuous batching. Swap finished sequences out and new ones in every step — utilization goes from ~20-30% to 80-90%. The single biggest throughput win, and why vLLM / SGLang / TensorRT-LLM exist.
Quantization. FP8 / INT8 / INT4 (AWQ, GPTQ) move less data → lower cost and latency, smaller footprint. 8-bit is near-lossless for most tasks; 4-bit is often an acceptable trade. Validate on your own eval set.
Speculative decoding. A small draft model proposes tokens; the big model verifies in one pass → lower latency with no quality loss.
Right-size the model. The cheapest token is the one you never compute on an oversized model — route easy requests down, hard ones up.

In practice

Use a real serving framework (vLLM, SGLang, TensorRT-LLM) rather than hand-rolling. Measure your actual prompt/response shapes first — long shared prefixes favour prefix caching, high concurrency favours batching, long outputs favour KV-cache and quantization work. Track cost-per-1k-tokens, throughput, and tail latency — the numbers the business actually feels.

Inference optimization is where AI economics are won or lost. The techniques are well understood and together routinely cut serving cost 5-10x — often the deciding factor in whether an AI feature ships at all.

Full version on the VSBD blog.

"Sakana Fugu: When Orchestrating Models Beats Owning One"

Vladyslav Donchenko — Fri, 26 Jun 2026 07:05:20 +0000

Every big AI story this year points the same way: the edge is moving from the model to the layer around it. Sakana AI's newly launched Fugu (and the heavier Fugu Ultra) is the most literal version of that idea — a system that beats frontier models by conducting them, without training a frontier model of its own.

What it actually is

Fugu isn't a bigger LLM. It's a small (~7B) model trained to route: take a task, decide which strong model in a pool should handle each part — Gemini 3.1 Pro, Claude Opus 4.8, GPT-5.5 — dispatch the work, and synthesize one answer. It can call itself recursively on long tasks (run, read its prior output, revise). Two tiers ship behind one OpenAI-compatible API: regular Fugu for everyday speed/quality, Fugu Ultra for hard multi-step work with a wider expert pool.

The claimed results — and the asterisks

Sakana reports Fugu beating Opus 4.8, Gemini 3.1 Pro, and GPT-5.5 on 10 of 11 benchmarks:

SWE-Bench Pro: Fugu Ultra 73.7 vs Opus 4.8 69.2, GPT-5.5 58.6, Gemini 3.1 Pro 54.2.
Humanity's Last Exam: 50.0, edging Opus 4.8 (49.8).
GPQA-D: 95.5, top of the field.
Only loss: MRCRv2 (GPT-5.5 94.8 vs Fugu Ultra 93.6).

Two caveats matter as much as the numbers: results are vendor-reported and not independently verified, and Anthropic's strongest models (Fable 5, Mythos) aren't in the pool because they aren't public. So Fugu matches the frontier by orchestrating what it can reach — not the absolute best models. Read the leaderboard as a claim.

The real insight: resilience as a feature

The sharpest part of Sakana's pitch isn't benchmarks — it's framing orchestration as insurance. Routing across providers means that if one model is restricted, rate-limited, repriced, or pulled, Fugu reroutes to the rest of the pool. That's a procurement argument, and it lands: single-vendor dependence is a real operational risk. A routing layer turns "our AI went down because our provider did" into "it failed over."

The takeaway

Whether or not the exact numbers hold, Fugu makes the year's quiet thesis loud: orchestration is becoming a frontier capability in its own right. Frontier-grade outcomes no longer require owning a frontier model — just the engineering to route, govern, and synthesize across the ones that exist. The tradeoffs are real too: added latency/cost, and broader data exposure across every provider you call (the mirror image of self-hosting an open-weight model for sovereignty). Most serious platforms end up doing both, deliberately.

Full version, with the real-estate / PropTech angle, on the VSBD blog. Source: Sakana AI — Fugu.

"One Model, Seven Worlds: What Qwen-AgentWorld Changes About Agentic AI"

Vladyslav Donchenko — Thu, 25 Jun 2026 15:40:57 +0000

Every agentic system today has an engineering debt nobody talks about: every new environment needs its own scaffold. Browser agent — bespoke prompts and error handling. Terminal agent — start from scratch. Mobile agent — same again. Qwen-AgentWorld attacks this at the root.

What It Is

Qwen-AgentWorld (arXiv 2606.24597) is the first language world model capable of simulating seven distinct agentic environments in a single unified model — not by stitching together seven specialists, but by training one model that learns a unified internal representation of how environments work.

The seven domains: MCP/Tool Calls, Search Engine, IDE/Git/CI-CD, Terminal/CLI, Android/UI, Web Browser/DOM, Operating System/Desktop. Trained on 10M+ real interaction trajectories. Three-stage pipeline: CPT injects state-transition dynamics → SFT activates next-state-prediction → RL with hybrid rewards sharpens fidelity.

Two model sizes: 35B-A3B and 397B-A17B (both MoE).

Two Paradigms

Decoupled Simulator — stands in for real environments during RL training. At 4,000-environment scale, synthetic rollouts via the world model yield gains on Tool Decathlon, MCPMark, and WideSearch that exceed real-environment training alone. Simulation at this fidelity means you can train agents for your specific environment without production traffic.

Unified Foundation — world-model training as a warm-up before task-specific RL. A model that has internalized how seven environments respond reaches higher performance on any specific task faster than a general pretrained base.

Why the PropTech Stack Is Exactly This Shape

The seven environments aren't a random selection — they're exactly the stack a real estate or PropTech operation runs across: browser for portals and listings, search for document intelligence, terminal for pipelines and reports, OS for file and document management, mobile for inspection and tenant apps, IDE/CI-CD for platform development, MCP/API for CRM and ERP integrations.

Today each environment needs its own agent, scaffolding, and eval. A world model that understands all of them without bespoke engineering per environment is the difference between one agent system and maintaining seven.

Caveats

GUI environments use accessibility trees, not pixel frames — no visual understanding
Sim-to-real gaps remain; world-model rollouts complement real training, not replace it
Weights/API availability timeline not yet confirmed

The Direction

The number of distinct models you need to operate an agentic system is collapsing. The bespoke-scaffold-per-environment approach is a transitional state. The durable investment is orchestration, policy enforcement, audit trails, and governance — the layer you own long-term regardless of which foundation model sits underneath.

Full take with the PropTech angle: One Model, Seven Worlds