DEV Community: WonderLab

Is Loop Engineering a New Technology, or Just Reheated Leftovers? Explained Through Training a Dog Classifier

WonderLab — Wed, 11 Nov 2026 07:35:09 +0000

The Short Answer

The term "loop engineering" is everywhere right now, and it sounds like a new craft. My view: the idea is leftovers, but this is the first time the leftovers are actually edible.

The leftover part: let a system iterate on its own, use a set of criteria to judge how each round went, adjust from the feedback, and repeat until the goal is met. That is the most basic machine learning training loop, and also the feedback loop from control theory.

The new part: in the past, what got adjusted inside the loop had to be numbers (model parameters). Now it can be a piece of text, a file, or a knowledge base, and the feedback can be a plain sentence. That is why the pattern can, for the first time, be applied to open-ended tasks.

Below I start from the most basic machine learning training, then use three real agent tasks to take loop engineering apart.

1. Start Somewhere Familiar: Training a Dog Classifier

Say you want to train a model that tells whether a picture contains a dog. The whole process has just five parts:

Part	In the dog model
Goal	95% recognition accuracy
What can be adjusted	The model's parameters (millions of numbers)
Evaluation	The loss function: compares predictions with the correct answers and computes "how wrong"
Feedback	The gradient: tells each parameter which direction to move and by how much
Stop condition	Accuracy reached, N epochs done, or several rounds with no progress

Then there is a loop:

while not done:
    prediction = model(image)                  # do a round
    loss = loss_fn(prediction, correct_answer) # evaluate the round
    gradient = backprop(loss)                  # what is wrong, how to adjust
    params = params - lr * gradient            # adjust

Note the division of labor: the model does not know by itself whether it is right. The loss function outside it does the scoring. The model only does the work, and the direction of adjustment comes from the evaluation system.

2. Put the Same Skeleton on an Agent

Now let an agent do the work. Not a single part is missing; only the carrier changes:

Part	Training the dog model	Loop engineering
Goal	95% accuracy	Type check and tests all pass / 90% accuracy on an eval set
A round	Model makes predictions	Agent completes part of the task, producing code, answers, or a report
Evaluation	Loss function	Compiler, tests, eval set, check scripts, a rubric, another model as judge
Feedback	Gradient (numbers)	Error messages, failing cases, explanations of lost points (text)
What can be adjusted	Model parameters	Code in the workspace, prompts, an attached knowledge base, a task list
Stop condition	Goal reached / epoch cap / early stopping	Passed / iteration cap / budget cap / several rounds without progress

As a diagram:

        ┌──────────── Goal ────────────┐
        ▼                              │
   Agent does a round ──▶ Evaluation ──▶ Done? ──yes──▶ End
        ▲                                │no
        │                                ▼
        └── Adjust from feedback ◀── What is good, what is not

A quick look at where things stand: today's agents are quite capable in a single conversation, and simple tasks come out right the first time. What loop engineering actually solves is a different class of problem:

The task is too big. One conversation's context cannot hold it, and things get messier toward the end
The task is too open-ended. A single pass gives unstable quality: three out of ten attempts go wrong
Nobody is watching. When it goes wrong, no one notices, and the error is passed along

The three cases below correspond to these three problems.

3. Case One: Migrating Hundreds of Files from JavaScript to TypeScript

The Task and Why It Is Hard

A team has a front-end project with 300-plus .js files. All of them must move to TypeScript, and afterward the project has to compile and every test has to pass.

Hand the whole thing to an agent in one go, and typical results are:

The first forty or fifty files look good, then it starts cutting corners with any
The context grows, it forgets types it defined earlier, and defines them again
Halfway through it says "done" while dozens of files are untouched
At the end hundreds of compile errors appear at once, and nobody knows where to start

This is not the agent being dumb. The size of the task exceeds what a single conversation can carry.

How to Build the Loop

1. Break the big task into a list, and keep the list in a file

TODO.md
[x] src/utils/date.js
[x] src/utils/format.js
[ ] src/api/user.js
[ ] src/api/order.js
...

The list is the loop's "memory." It survives every round and does not depend on conversation context.

2. Each round does one small step, from a fresh context

The instruction for each round is simple: read the list, pick one unfinished file, migrate it, run the checks, tick it off and commit if they pass, then end. The next round is a brand-new conversation with none of the accumulated clutter, only the list and the codebase.

3. Evaluation is a compiler and tests the agent cannot change

Evaluation commands:
  tsc --noEmit            # type check; no new suppressions beyond what exists
  npm test                # all tests
  check script: count occurrences of @ts-ignore and `as any`; must not increase

4. Feedback: pass the compiler's errors to the next round as they are

Compiler output such as error TS2345: Argument of type 'string' is not assignable to parameter of type 'number' is a precise diagnosis, more useful than any summary.

5. Stop condition: every item ticked and checks passing; or at most N rounds; or two rounds in a row where the failure count does not drop (it is stuck and needs a human).

Here is the smallest skeleton of that loop. It does three things: call the agent, call the evaluation command, and decide whether to stop. I ran its control flow with stub commands (both the "goal reached" exit and the "stuck" exit), but I did not connect a real agent:

#!/usr/bin/env bash
# Skeleton: do a round -> external evaluation -> leave feedback for next round -> stop check
# AGENT_CMD: have the agent do one round (fresh context each time)
# EVAL_CMD : evaluation command; exit code 0 means the goal is met; the agent cannot modify it
MAX_ITERS=${MAX_ITERS:-10}
PATIENCE=${PATIENCE:-2}
best_fail=999999; stale=0

for i in $(seq 1 "$MAX_ITERS"); do
  echo "== Round $i =="
  bash -c "$AGENT_CMD"                      # 1. The agent does a round
  bash -c "$EVAL_CMD" > eval.log 2>&1       # 2. External evaluation
  if [ $? -eq 0 ]; then echo "Goal met, stopping"; exit 0; fi

  fails=$(grep -c "FAIL" eval.log)          # 3. Track progress by failure count
  if [ "$fails" -lt "$best_fail" ]; then best_fail=$fails; stale=0; else stale=$((stale+1)); fi
  if [ "$stale" -ge "$PATIENCE" ]; then echo "No progress for $PATIENCE rounds, stopping"; exit 2; fi

  cp eval.log feedback.txt                  # 4. Leave feedback for the next round to read
done
echo "Iteration cap reached, stopping"; exit 3

The Ralph loop that circulates in the community is, in essence, the plainest version of this skeleton: a while loop that feeds the same task description to the agent every time and lets it pick up from progress stored in files. The difference is that production use needs a budget cap and stop conditions added, otherwise it is just burning money without limit.

Why the Success Rate Goes Up

	Hand it all to the agent at once	With a loop
Context	Keeps growing; quality drops later on	Clean each round, carrying only the list and feedback
Completion	The agent says "done" by its own feeling	The compiler and tests decide
Error handling	Hundreds of errors surface at the end	Each file is checked and fixed on the spot
When something breaks	No idea where to roll back to	One commit per round; exact rollback

I have not measured this scenario, so I give no specific success rate. The key is not "running it more times," but that the question "is it done" moves from the agent's own impression to the compiler's judgment.

4. Case Two: Making a Support Agent Handle Tickets More Accurately

The Task and Why It Is Hard

A company has a customer support agent that sorts incoming tickets (refund / shipping / account / other) and drafts a suggested reply. After launch, some tickets are misclassified and sent to the wrong team.

What gets adjusted this time is not a piece of code but the agent's own configuration: the prompt and the knowledge base it can search. This is the closest to training the dog model, because it really is parameter tuning, except the parameters are text.

How to Build the Loop

1. Prepare an eval set first: a batch of historical tickets with the correct answers labeled by people

Say 200 of them. It plays the role of "images with correct labels" in the dog model. Then set 40 of them aside and never look at them during the whole tuning process, using them only once at the end. This is the old machine learning rule of keeping the training and test sets apart. Without it, the high score you see at the end only means the prompt memorized those questions.

2. Run a round, look at the score, and look harder at where it went wrong

The numbers below are made up to illustrate the idea, not real experiment results:

Round 1  config v1   accuracy 72%
  Error analysis (grouped by type):
    - "Package shows delivered but I never got it" classified as "other" (should be shipping)   x 17
    - "Want to change my linked phone number" classified as "other" (should be account)         x 12
    - "Refund is slow to arrive" classified as "shipping" (should be refund)                    x 8
    - Other scattered errors                                                                    x 7

3. Have another agent read the error analysis and propose configuration changes

Changes in v2:
  - Add a prompt rule: package problems (delivered but not received, lost, damaged) all go to shipping
  - Add a prompt rule: changing a phone number, email, or password goes to account
  - Add 12 example sentences to the knowledge base that separate "refund" from "shipping"

4. Re-run the eval set, confirm it improved, and confirm nothing that was right got broken

Round 2  config v2   accuracy 86% (tuning portion)
Round 3  config v3   accuracy 91%
Round 4  config v4   accuracy 91% (no progress) -> stop
Final: check once on the 40 held-out tickets -> 88%

Look at the last line: 88% on the held-out set is a little lower than the 91% during tuning. That is normal. The gap is the most honest number in the whole process.

Why the Success Rate Goes Up

	Editing the prompt by gut feeling	With a loop
How do you know it improved	"It feels a bit better"	Compare accuracy on the same questions before and after
How do you know what to change	Guess from impressions	Error analysis points to specific error types
Could it break something else	No idea	Re-run the whole eval set each round; regressions show up immediately
When to stop	When you are tired of editing	When several rounds in a row make no progress

In this case, the eval set is itself an asset: the next time you switch models or change the workflow, it tells you right away whether things got better or worse.

5. Case Three: Having an Agent Write a Competitor Research Report (Where Evaluation Gets "Soft")

The Task and Why It Is Hard

Ask an agent to research three competitors and write a comparison report for a product manager. This kind of task has no test cases, and "is it good" seems to come down to feel.

Typical problems from a single generation:

Numbers have no source, or the source page does not contain that number at all
The three products get different amounts of space and depth
The conclusion is vague: "each has its strengths"

How to Build the Loop

The key for soft tasks: break "good" into small questions that can be checked one by one, and make some of those checks deterministic so they do not depend on a model's subjective judgment.

Check	How	Hard / soft
Every number in the report has a source link	A script extracts numbers and links and confirms each link is reachable	Hard
The number actually appears on the source page	A script fetches the page and searches for it	Hard
All three competitors cover the same dimensions	Tick through a dimension checklist	Semi-hard
The conclusion gives a clear recommendation	Another model scores it against a rubric and writes the reason for each deduction	Soft

The flow:

The agent writes a first draft
The check scripts run the hard checks and the judge model runs the soft ones, merged into a list of problems with exact locations (for example, "the '23% market share' in paragraph 3 cannot be found on the cited page")
The agent edits only for the items on the list and does not rewrite the whole piece, so parts already fixed are not broken again
Check again, for at most three or four rounds

Why the Success Rate Goes Up

The biggest gain comes from the hard checks: numbers without sources and made-up citations get caught by a script on the spot. These are exactly the errors a human reviewer misses most easily and that damage credibility most.

But to be honest: the ceiling for this kind of loop is lower than in cases one and two.

The judge is a model with its own biases, and it can score the same report differently on two runs
Model judges often favor longer, fancier writing, so the agent gets wordier with every edit
"Good enough" is itself a fuzzy call

So a soft-evaluation loop is better for raising the floor and filtering out obvious problems than for producing a final draft automatically. An important report still needs a person to sign off.

6. What the Three Cases Have in Common

Side by side:

	Case 1: code migration	Case 2: support tickets	Case 3: research report
Why one pass falls short	Task too big for the context	Config is off, tuned by feel	Open-ended task, unstable quality
Who evaluates	Compiler + tests	Eval set with correct answers	Scripts + rubric + judge model
What feedback looks like	Compiler errors	Error analysis	Problem list with locations
What gets adjusted	Code, task list	Prompt, knowledge base	The draft report
How "hard" the evaluation is	Hard	Fairly hard	Mix of hard and soft

There is only one shared rule: a loop's ceiling is set by the quality of its evaluation. The harder and more specific the evaluation, the more reliable the loop; the softer it is, the more it needs a person.

7. A Few Easily Misunderstood Points

1. There Is No Real "Gradient" in a Loop

In the dog model the gradient is computed mathematically, and the direction is exact. In a loop, the "gradient" is a piece of text feedback that a large model reads and then decides how to act on. So adjustments can oscillate: fixing one item may break another, and no number of rounds guarantees convergence. That is why an iteration cap, early stopping, and regression checks are necessary.

2. The Model Itself Does Not Change; What Surrounds It Does

Training modifies a model's weights. A loop does not. The large model's parameters stay frozen throughout, and what gets adjusted is the workspace, the context, the prompt, and the knowledge base. Case one adjusts a work product, case two adjusts the agent's own configuration, and they solve different problems: the former is "get this one thing done," the latter is "make the agent more accurate at this kind of work from now on."

3. The Agent Must Not Control the Evaluator

This is the most important point, and the biggest lesson it shares with machine learning. If the party being evaluated decides how it is evaluated, it learns to "get a high score" rather than to "get it right." Common ways agents cheat include deleting or editing failing tests, hard-coding answers for the test cases, and writing their own tests, passing them, and declaring victory.

The countermeasures: keep tests and evaluation scripts where the agent has no write access; have different roles write the code and write the tests; keep a validation set the agent has never seen. One line from the community puts it well: only loop on what you can verify.

4. Evaluation Needs a "Diagnosis," Not Just a "Score"

With only a 72, the agent knows it is not good enough but not where to change. The score is the loss value; the specific failing items and reasons are the gradient. When designing evaluation, try to have it output which item, which location, and why it is wrong.

8. So, New Technology or Leftovers?

The leftover part, which is most of it:

Feedback loops: the thermostat in control theory has had this structure for a long time
Iterating until the goal is met: the machine learning training loop, and "change it, run the tests, change it again" in software
Continuous improvement: PDCA (plan, do, check, act)
Eval sets, early stopping, guarding against overfitting: the basics of machine learning

If you already know these, you have been using about eighty percent of loop engineering all along. There is no need to mythologize it.

The new part does exist:

What can be adjusted has changed. It used to have to be differentiable numbers; now it can be a piece of text or a file, so open-ended tasks can enter the loop for the first time.
Feedback can be plain language. Evaluation used to be numbers only; now an error message or a comment can drive the next round of edits directly.
The "adjuster" is itself a model that can reason. It can read the feedback and draw lessons, rather than following a gradient mechanically.
New risks. An evaluator that can be gamed, runaway cost, and biased judges are less prominent in traditional training.

One more caveat: the boundaries of the term are still fuzzy. It overlaps heavily with harness engineering (building the runtime around an agent) and context engineering, and the terminology is still shifting; different articles mean slightly different things by loop engineering. Do not get hung up on the word. Hold on to the five parts.

A more accurate way to say it: old wine in a new bottle, but with one new ingredient added: the "adjust" step can now be done by a model that understands text feedback. That ingredient lets the old wine be used in places it could not reach before, which is why it is worth discussing again.

9. A Checklist Before You Start

To add a loop to your own agent, answer these six questions first:

What is the goal? Can you state it in one sentence, ideally with a number?
Who evaluates? Is the evaluator out of the agent's reach? Is there a hard metric you can use?
Does the feedback contain a diagnosis? When something fails, can the agent see exactly what is wrong?
What can the agent adjust? Work products, or its own prompt and knowledge base? Is the scope drawn?
When does it stop? Is there an iteration cap, a budget cap, and a stop after several rounds without progress?
How do you prevent regressions? When one thing is fixed, is there a regression check that nothing else was broken?

A simple test: if you cannot answer question 2, do not start the loop yet. Without reliable evaluation, a loop only amplifies errors faster and with more confidence.

Summary

The essence: loop engineering is the most basic machine learning engineering mindset: goal, do a round, evaluate, feedback, adjust, go again, until the goal is met or it stops.
What it solves: tasks too big for the context, tasks too open-ended to be stable, and errors that propagate when nobody is watching. The three cases map to these in turn.
The ceiling is set by evaluation: compilers and tests are hardest, eval sets next, model judges softest.
New versus old: the idea is leftovers; what is fresh is that the adjustable thing can be text and the feedback can be plain language.
In practice: practice on tasks with hard evaluation first, then try soft evaluation; design the evaluation first, then let the loop run.

Notes: the control flow of the skeleton was verified with stub commands (the goal-reached and stuck exits) and not connected to a real agent. The numbers in case two are invented for illustration. The three cases are summaries of common practice; I have not run full experiments on cases one and three, so no specific success rates are given. The description of how the community uses "loop engineering" comes from public articles, and the definition of the term is still not settled.

Open Source Project #229: e2e — Write E2E Tests in Plain English, Then Replay Verified Agent Actions With Zero Model Calls

WonderLab — Wed, 07 Oct 2026 12:11:22 +0000

Introduction

"Describe a goal in natural language and an agent drives the app to reach it. Check the result with locators and assertions in the same test."

This is article #229 in the "One Open Source Project a Day" series. Today's project is e2e — an end-to-end testing framework for web and mobile apps by TesterArmy, 6,361 Stars, Apache-2.0 license.

End-to-end testing has long been stuck in a tension: hardcoded selectors break the moment the UI is redesigned, but if test logic is too loose, it can't run reliably in CI. e2e's answer is to split every test into three fundamentally different kinds of steps — natural-language agent actions (Goals), model-judged assertions (Assertions), and precise selector-based checks (Locators) — then layer a "replay cache" on top, so verified agent actions can skip model calls entirely on later runs, waking the agent back up only when the app actually changes.

What You'll Learn

How the three test step types divide responsibility: Goal / Assertion / Locator
How the replay cache lets AI-driven tests escape the "call the model every single run" cost problem
The Decision Model executor: answering multiple-choice questions with a small model instead of letting a large one free-wheel
Quick start: from npx e2e init to your first test
Migration paths from Playwright, Cypress, Detox, and Maestro

Prerequisites

Experience writing Playwright, Cypress, or similar end-to-end tests
Basic understanding of how LLM agents work (instruction → observation → action loop)
TypeScript fundamentals

Project Background

Overview

e2e is built by TesterArmy — the same company behind an agentic testing platform that runs natural-language tests on web and mobile apps on every pull request or on a schedule. e2e is the open-source core engine behind that capability.

It's not another "let AI generate your test code" tool. It embeds AI-driven action itself into the test runtime: within a single test, which steps should be left to the agent's own judgment, and which must be pinned down with exact selectors, is something the developer explicitly decides.

Author / Team

Organization: TesterArmy
Primary language: TypeScript
License: Apache-2.0
Created: 2026-07-22
Status: Active development toward 1.0; APIs and config can still change between minor releases

Project Stats

⭐ GitHub Stars: 6,361+
🍴 Forks: 285+
📄 License: Apache-2.0
📅 Created: 2026-07-22
🏷️ Topics: e2e, e2e-testing, end-to-end-testing, mobile, mobile-testing, playwright, web

Quick Start

Installation

npx e2e init

init asks for an engine (web or mobile) and a model provider, then writes a config and an example test.

Write a Test

// tests/checkout.e2e.ts
import { test, expect } from 'e2e';

test('a member upgrades to Pro', async ({ app, agent, screen }) => {
  await app.open('/settings/billing');

  await agent.act('upgrade the workspace to the Pro plan');
  await agent.assert('the invoice preview shows a prorated amount');

  await expect(screen.getByRole('status')).toContainText('Pro');
});

All three step types show up in this one test: agent.act lets the agent figure out how to complete "upgrade to Pro" on its own, agent.assert has the model judge whether the invoice preview is semantically correct, and expect verifies the final state with a precise role-based selector.

Supported Engines

Package	What it does
`e2e`	The SDK, runner, and CLI
`@e2e-dev/web`	Browser engine: Chromium, Firefox, WebKit through Playwright
`@e2e-dev/mobile`	iOS/Android engine: simulators and emulators through agent-device
`@e2e-dev/github`	Reporter that posts results as a pull request comment
`@e2e-dev/kernel`	Kernel-hosted cloud browsers
`@e2e-dev/eas`	EAS Simulators-hosted iOS simulators and Android emulators
`@e2e-dev/decision`	Decision-model executors for bounded semantic actions and assertions, instead of a full LLM

The project ships ready-made examples for Vite, Next.js, Expo, and SwiftUI.

Core: How the Three Step Types Divide Responsibility

e2e splits every test step into three categories, each with a different model-call policy:

Step	API	Model call
Goal	`agent.act`	Yes, unless replayed from cache
Assertion	`agent.assert` / `agent.waitFor` / `agent.extract`	Yes
Locator	`screen` and `expect`	No

Goal: Let the Agent Drive the Flow

await agent.act('complete checkout with the test card');
await agent.act('invite {email} as an editor', { params: { email: 'ada@example.test' } });

agent.act takes exactly one goal. The agent decides what to click and what to type. The developer controls the order of goals, not the execution details inside each one. The project's own guidance:

Write one goal per call, using the words the screen actually shows
Pass test data through params; for passwords, use a Secret (the model never sees the plaintext)
Wrap values that change on every run — like emails or timestamps — in unique(), so the replay cache can match them correctly

Assertion: Let the Model Judge Semantics

await agent.assert('the dashboard shows a trial badge');
await agent.waitFor('a download link appears', { timeout: 120_000 });

const data = await agent.extract('every todo title', {
  schema: z.object({ titles: z.array(z.string()) }),
});

The judging model only sees a text snapshot of the current screen plus the agent's context — not the history or summaries of earlier steps. This isolation is deliberate: it keeps assertions from being swayed by the "narrative" of prior steps. For visual checks (charts, layouts), pass vision: true to let the model see a screenshot.

Locator: No Model Involved

await screen.getByRole('button', 'Sign in').tap();
const row = screen.getByRole('listitem').filter({ hasText: 'Design review' });
await row.getByRole('button', 'Archive').tap();
await expect(screen.getByRole('status')).toHaveText('2 remaining');

When you know exactly which control to click and exactly what text to check, use a Locator — zero model calls, zero uncertainty, same writing style as a traditional Playwright test.

The Most Interesting Design: The Replay Cache

This is e2e's core differentiator against "just let an LLM drive the whole test," and the key mechanism for solving the "AI testing is too expensive and too slow" problem.

How It Works

Once an agent.act() step is followed by a verification step that confirms the result, the runner records exactly which actions it took (which control, what value) into a JSON entry
On the next run of the same test, the runner replays those actions directly, with no model call at all
If replay finds a control missing or the page structure changed, the agent takes over from the current screen and finishes the rest using the model
The run report shows the breakdown: 4 replayed · 1 handed off · 1 missed — which steps were pure replay, which fell back to the agent mid-replay, and which had no cache hit at all

           AI  4.1k tokens · 2 model calls · anthropic/claude-sonnet-4.5
        Cache  4 replayed · 1 handed off · 1 missed

What Counts as "Verified"

The cache doesn't record unconditionally. Only when agent.act is immediately followed by a step that actually verifies the outcome — like expect(locator).toHaveText(...) or agent.assert(...) — does a recording get written. Plain-value assertions like expect(value).toBe(...) or read-only calls like agent.extract(...) don't count as verification and won't trigger recording.

This design forces a disciplined "Act, then immediately check" test structure — which happens to be exactly the rigor end-to-end tests should have anyway.

Why a Replay Fails

Reason	Meaning
`no-entry`	No recorded entry matches this step
`wrong-context`	The app is on a different screen than when recorded (different origin/path/query)
`target-not-found`	A recorded control didn't appear within 15 seconds
`end-mismatch`	Every action ran, but the expected final state never showed up
`gap`	An action simply can't be replayed (e.g. one that depended on reading pixels)

In production, --strict-cache turns "cache exists but fails to replay" into a hard failure instead of a silent hand-off — closing off the hidden cost problem of "every CI run is quietly spending model calls and nobody notices."

Why This Design Matters

The biggest pain point of most "AI-driven testing" tools is uncontrolled cost: every CI run makes a real LLM call, and multiply that by dozens of tests times multiple CI triggers per day, and the bill spirals fast. e2e's replay cache essentially separates "the agent discovering a reliable path" from "repeatedly executing that path" — you pay for model calls during discovery, and execution afterward is nearly free, until the app actually changes enough to warrant paying for discovery again. That's a lot smarter than "ask the AI again from scratch every time."

The Decision Model Executor: Multiple-Choice Instead of Free-Form

@e2e-dev/decision offers another way to execute agent.act / agent.assert: instead of a full LLM making free-form decisions, a dedicated Decision Model answers one multiple-choice question per action — "which operation, and which element" — with a probability distribution rather than free text. Only when the chosen operation is "type" does a small text model generate the actual input value.

import { decisionExecutor } from '@e2e-dev/decision';

const executor = decisionExecutor({
  model: typeSafeAi.decisionModel('jev-latest'),
  textModel: openrouter('inception/mercury-2.5'),
  minProbability: 0.9,
  minConfidence: 0.8,
});

The point of this design: model output never becomes a selector, a URL, or a key — it can only pick an index from a finite set of options the executor built. minProbability and minConfidence let you reject low-confidence actions outright instead of gambling on an uncertain guess. The trade-off is reduced capability — no vision, no multi-select lists, no drag-and-drop — which makes it a good fit for apps with regular, predictable UI structure where you want to run a large test suite more cheaply.

Resources

🌟 GitHub: tester-army/e2e
📦 npm: e2e
📖 Documentation: e2e.tester.army/docs
🏢 Made by: TesterArmy
💬 Discord: tester.army/discord
🔄 Migration guides: from Cypress, Detox, Maestro, Playwright, and Selenium

Summary

e2e represents a clear-eyed judgment about AI testing tools: getting an agent to operate an app isn't the hard part — the hard part is doing it in a way that adapts to UI changes without turning every CI run into a model-call bill.

Three things worth noting:

The explicit split into three step types corrects an overgeneralization in "AI testing." Many AI testing tools try to have one agent handle everything — acting, asserting, verifying, all via model judgment. e2e splits tests into Goal / Assertion / Locator layers, letting developers choose precisely where they need flexibility and where they need determinism, instead of being forced to hand everything to a model or write every selector by hand.

The replay cache separates "discovery cost" from "execution cost," which is the key to running AI-driven tests sustainably in real CI. Without this mechanism, AI-driven end-to-end testing is nearly impossible to scale in production — model-call latency and cost would grow linearly with the number of tests. e2e's "record after verification passes, replay afterward, rediscover only when things change" strategy keeps most runs almost free of model cost.

The Decision Model executor shows that not every AI decision needs a full LLM. Turning action selection into a multiple-choice question for a dedicated Decision Model is cheaper, more controllable, and more auditable than letting a general-purpose LLM free-wheel — worth watching for teams that need to run large test suites at scale.

If your team is looking for an AI-driven end-to-end testing approach that can actually run reliably in real CI — not just impress in a demo — e2e is worth a serious look.

Explore PrimeSkills — a curated marketplace of AI agents and skills, each validated against real enterprise workflows. No hype, just what actually works.

Visit my personal site for more insights and interesting products.

One Open Source Project a Day (No. 192): AX — Google's Open-Source 'Kubernetes for Agents,' Orchestrating Billions of Agent Tasks

WonderLab — Sat, 26 Sep 2026 08:49:32 +0000

Introduction

"Agents are a new kind of workload. They are neither stateless microservices nor run-to-completion batch jobs."

This is the 192nd article in the "One Open Source Project a Day" series. Today's project is AX (repository: google/ax).

Once a team starts running AI agents at scale, they quickly hit an awkward fact: none of the existing scheduling infrastructure fits well. Kubernetes' Pod model assumes workloads are either stateless services or run-to-completion batch jobs, but agents accumulate state, need strict sandbox isolation, call out to model APIs and tool servers constantly, and can burn through a token budget in a loop the moment nobody's watching. Traditional CI/CD or batch platforms weren't designed for needs like "pause an agent and resume it exactly where it left off" or "SSH into the sandbox to see what it's actually doing."

AX is Google's open-source "high-throughput, declarative agent orchestration runtime," aimed at running billions of autonomous agent workloads in a single cluster. If you've used Kubernetes, the ax experience will feel very familiar — apply, get, describe, watch, delete, a kubectl-shaped command set, just with agent tasks as the scheduling object. It runs on top of Agent Substrate for sandboxed execution, and focuses purely on the declarative orchestration layer itself.

11,600+ Stars, 560 Forks, Apache-2.0 License, developed and led by Google. It's currently in Alpha — core concepts and protocols are still rapidly evolving, and the project explicitly warns of likely major breaking changes before a stable release.

What You Will Learn

AX's three core primitives — Task, Workspace, Model — and what each solves
Why agent workloads need a new scheduling model, neither a stateless service nor a batch job
The layered relationship between AX and Agent Substrate: orchestration layer vs. sandbox execution layer
The control-plane architecture: why AX stores state in Redis instead of directly as Kubernetes CRDs
The sandbox lifecycle design behind ax suspend / ax resume / ax ssh

Prerequisites

Familiarity with basic Kubernetes concepts (Pods, CRDs, kubectl habits)
A basic understanding of containerized deployment and sandbox isolation
Optional: a basic understanding of MCP (Model Context Protocol)

Project Background

What It Is

AX describes itself as "Google's open agentic orchestration runtime." The README opens with a tight operational description: "\"Declare an agentic task with workspaces and model specifications. AX sandboxes it, wires up its workspace, and helps running it at scale.\" That single sentence captures the project's division of labor — AX itself doesn't handle sandbox isolation or execution; that's delegated to the underlying Agent Substrate. AX focuses on declaratively defining tasks, scheduling at scale, and lifecycle management."

The project documentation directly explains why existing tools fall short: "Agents are a new kind of workload. They are neither stateless microservices nor run-to-completion batch jobs. They accumulate state, need strict isolation, call out to model APIs and tool servers, and can burn money in a loop if nobody is watching." AX's answer is three small primitives that can be declared, rather than one all-encompassing framework trying to model everything an agent does.

Team and Background

Organization: google
License: Apache License 2.0
Primary language: Go
Underlying substrate: Agent Substrate (sandboxed actor execution environment)
Website: agentexecutor.io
Status: Alpha — core concepts and protocols are still evolving, with explicit warnings of likely major breaking changes

Project Stats

⭐ GitHub Stars: 11,600+
🍴 Forks: 560
📄 License: Apache-2.0
🐛 Open Issues: 48
📅 Created: 2026-03

What It Does

The Problem It Solves

Running agent workloads on general-purpose scheduling platforms:
  Kubernetes Pod model → assumes workloads are stateless services or batch jobs
  ↑ agents accumulate state, need strict isolation, call external model APIs frequently
  ↑ no native support for "pause and resume in place"
  ↑ no debugging channel for "SSH in and see what the agent is doing"
  ↑ every run re-clones repos, re-wires tools, re-configures MCP servers — high cold-start cost

AX's approach:
  Three declarative primitives: Task (sandboxed execution unit)
                                + Workspace (a pre-warmed environment)
                                + Model (a cluster-level model configuration)
  ↓
  Declare the whole thing at once with ax apply -f task.yaml
  ↓
  ax watch / ax ssh / ax suspend / ax resume give K8s-style
  observability and lifecycle management
  ↑ sandbox isolation is delegated to Agent Substrate; AX focuses on the orchestration layer

Use Cases

Running autonomous coding/ops agents at scale
- Each agent task runs as its own sandbox, declaring the Git repos, MCP servers, and tools it needs — fitting scenarios like "dispatch a separate agent to fix bugs across a fleet of repos"
Long-running, pausable agent workflows
- Via ax suspend / ax resume, an agent's state can be checkpointed and resumed later in place, without needing to design custom logic for "how do I save resources on a long-running task"
Development scenarios needing to debug and observe actual agent behavior
- ax ssh lets you go directly into a running sandbox to inspect the filesystem or run commands, instead of only being able to stare at log output
Multi-team scenarios needing centralized model credential and config management
- Model is declared as a cluster-level resource, so rotating a key or switching model versions is a single ax apply instead of hunting through each agent's environment variables individually

Quick Start

Prerequisites:

# Requires a Kubernetes cluster with Agent Substrate already installed
kubectl get svc api -n ate-system   # verify Substrate's Control API is ready

Install the CLI:

go install github.com/google/ax/cmd/ax@latest

Deploy the control plane:

make deploy AX_IMAGE_REPO=<your-registry>

Declare a Workspace and Task in one YAML file:

# task.yaml
apiVersion: ax.io/v1alpha1
kind: Workspace
metadata:
  name: golang
spec:
  git:
    - repo: https://github.com/golang/go.git
      branch: "my-fix"
---
apiVersion: ax.io/v1alpha1
kind: Task
metadata:
  name: test
spec:
  workspaces:
    - name: golang
      goal: "Ensure that Go tool chain is available and is built from source"
  debug: true   # lets you `ax ssh` into the sandbox

Apply and observe:

ax apply -f task.yaml
ax watch task test
ax ssh test -- ls -al /workspace

Core Features

1. Three orthogonal declarative primitives

Primitive	Solves
Task	Runs untrusted agent code in an isolated sandbox with CPU/memory limits
Workspace	Pre-wires Git repos, MCP servers, and skill packages so every agent "starts warm"
Model	Declares which LLM the platform itself uses, with credentials from a Kubernetes secret

2. A kubectl-shaped CLI

ax apply, ax get, ax describe, ax watch, ax delete, plus agent-specific verbs like ax ssh, ax suspend, ax resume. It follows your active kubectx context — switch clusters and ax automatically resolves and tunnels to that cluster's control plane in the background.

3. Task's deliberately minimal design philosophy

Over its lifetime, an agent plans, delegates, retries, and fans work out. AX makes no attempt to model that shape. It gives you one primitive that's cheap to create, isolate, suspend, and throw away, and lets the agent compose as many of them as its work demands. A single Task may be the entire job, or the root of a large tree of tasks spawned as the agent breaks the problem down — either way, every node gets the same sandbox, the same lifecycle, the same tooling.

4. Goal-driven Workspace bootstrapping

A Workspace binding can carry a goal — a plain-language description of the environment the task needs. On first boot, the runner hands that goal to an agent that finishes the setup (say, installing a toolchain or dependencies), so the task's own command starts in an already-ready environment.

5. Cluster-level model resources

Declaring model configuration as a cluster resource, rather than scattered across each agent's environment variables, means rotating a key, pinning a new model version, or tuning generation parameters is a single ax apply. AX's own components — for example, the process that plans a Workspace from a goal — read Model resources too.

6. Suspend/resume and SSH debugging

ax suspend checkpoints actor state and pauses the task; ax resume picks it back up. ax ssh requires a task to explicitly set spec.debug: true before it will connect — because the underlying guest services grant arbitrary process execution and file access, keeping this off by default is a deliberate security decision.

Project Advantages

Dimension	Running agents directly on Kubernetes	Building your own agent orchestration	AX
Fit with the state model	Poor — Pods assume stateless or batch	Depends on how much you build	Designed for agents' "accumulates state + needs suspend/resume" nature
Scheduling at scale	Storing millions of tasks as CRDs hits etcd's limits	Needs to be solved yourself	Redis + Streams built for a billions-of-tasks target
Environment pre-warming	Needs custom initContainer logic	Needs custom implementation	The Workspace primitive handles it declaratively
Debugging channel	Logs/exec only	Needs custom implementation	`ax ssh` built in, off by default for security
Model config management	Scattered everywhere	Needs your own wrapper	`Model` managed centrally as a cluster resource

Why choose this project?

Led by Google, already thinking through problems at the scale of "billions of agent tasks" — architectural decisions like choosing Redis over etcd are made specifically for that scale
The operational mental model directly reuses Kubernetes experience, so kubectl users face near-zero onboarding cost
A clear layered design — AX focuses purely on orchestration, while sandbox execution is delegated to Agent Substrate, keeping responsibilities cleanly separated

A Deeper Look

Why Task State Isn't Stored as a Kubernetes CRD

This is a particularly interesting architectural decision in AX. The project documentation states the reasoning directly: "Storing millions of short-lived tasks as Kubernetes CRDs pushes etcd past its comfort zone (single-digit GB storage limits, write-rate bottlenecks, control plane degradation)."

AX's solution is to store state in Redis, using Redis Streams as the work queue between the API server and a horizontally scaled pool of controllers:

ax apply -f task.yaml
        │
        ▼
    ax-server (stateless gRPC API)
        │
   store & publish event
        │
        ▼
      Redis (Task Hashes + Event Streams + PubSub)
        │
   XREADGROUP (consume the stream)
        │
        ▼
  ax-controller (horizontally scaled worker pool)
        │
      gRPC call
        │
        ▼
   Agent Substrate (sandboxed execution)

This choice shows the AX team never intended to box themselves into the "everything is a CRD" Kubernetes mental model — they only borrowed its operational experience (apply/get/watch), while swapping the underlying storage engine for something better suited to high-frequency, short-lived objects. It's a pragmatic tradeoff: the interaction pattern users perceive stays the same, but the backend is redesigned for the real scale requirement.

Task's "Small and Composable" Design

AX's philosophy around Task is worth unpacking further. Many orchestration systems try to model "a complete agent workflow" — defining DAGs, defining step dependencies. AX does the opposite: it refuses to model an agent's internal behavior (planning, delegating, retrying, decomposing), offering instead only an atomic, cheap execution unit and letting the agent decide how to compose it.

The upside of this design is flexibility — a single Task can be the entire job, or the root of an entire tree of tasks, and AX never needs to understand the shape of that tree; every node gets the same sandbox and lifecycle primitives. The tradeoff: dependencies and data flow between tasks have to be managed by the agent or a higher-level tool — AX doesn't provide built-in workflow-orchestration semantics.

Workspace's Goal-Driven Bootstrapping Mechanism

Workspace.spec.goal is a somewhat unusual design: rather than directly specifying "install these dependencies," it's a plain-language description of an environment goal (e.g., "ensure the Go toolchain is available and built from source"), handed to an agent at sandbox startup to interpret and execute.

This turns what would normally be mechanical work — writing setup scripts, writing Dockerfiles — into a dynamic process that can be described in natural language and completed by an agent. The project's roadmap shows this direction continuing to deepen: "Dynamic Agentic Environment Curation" plans to let this bootstrapping process automatically inspect repository contents, resolve toolchains, and discover relevant MCP servers and skills from registries, rather than just executing one pre-written goal description.

Sandbox Lifecycle: The Runner as a Persistent PID 1

Inside every task container, ax-task-runner starts as PID 1, taking on responsibilities well beyond simply "starting a command": it loads the Task and Workspace specs, starts a metadata and guest-management daemon on port 80, prepares each workspace in binding order on first run (cloning repos, setting up the skills path, and handing off to an agent if a goal is set), and only then starts spec.command as a supervised child process.

The key detail: the runner stays alive as PID 1 even after the command exits, meaning the metadata server keeps responding and ax ssh still works after the main command has finished. This makes it possible to debug an already-completed task, rather than having the whole container vanish the moment the task ends.

The Layered Relationship with Agent Substrate

AX doesn't handle sandbox isolation itself — that's fully delegated to Agent Substrate, a lower-level system responsible for atespace provisioning, actor creation and activation, and worker assignment. AX's controller drives tasks toward their desired state by calling Substrate's Control API over gRPC. This layering lets AX focus purely on "declarative orchestration semantics" and "scheduling at scale," leaving the harder, lower-level problem of "how do you safely isolate an untrusted process" to a dedicated project — which is also why AX's roadmap devotes so much space to how the Substrate actor architecture will evolve (migrating to the new actor API, splitting workspace initialization into a dedicated actor, minimally privileged policies, idleness detection with automatic suspension, and so on).

Project Links and Resources

Official Resources

🌟 GitHub: https://github.com/google/ax
🌐 Website: https://agentexecutor.io
📄 License: Apache License 2.0
🐛 Issue Tracker: GitHub Issues

Related Resources

Agent Substrate — The sandboxed execution environment AX depends on
Agent Substrate Guest Services — The process/filesystem services that power ax ssh
Model Context Protocol — The tool-integration standard Workspaces can wire in

Summary

Key Takeaways

Three orthogonal primitives cover the core needs of agent orchestration: Task handles isolated execution, Workspace handles environment pre-warming, Model handles cluster-level configuration management
Reuses Kubernetes' operational mental model, but redesigns the storage backend: replacing etcd with Redis + Streams is a tradeoff tailored to a billions-of-short-lived-tasks target
Doesn't model agent behavior, only provides a cheap composable execution unit: Task is deliberately small, leaving decomposition, delegation, and retry logic to the agent itself
A clear layered architecture: AX focuses on declarative orchestration, delegating sandbox isolation to Agent Substrate, keeping responsibilities cleanly separated
Still in Alpha: core concepts and protocols are still rapidly evolving — watch for breaking changes before adopting it in production

Who This Is For

Teams running autonomous agent workloads at scale: especially where task volume has already outgrown hand-managed scripts
Teams already using Kubernetes and comfortable with its operational mental model: the kubectl-style experience means near-zero migration cost
Scenarios needing to pause/resume long-running agents, or needing a debugging channel to observe actual agent behavior
Developers willing to accept the risk of an Alpha-stage project in exchange for getting in early on the design and contributing

One-Line Verdict

AX isn't trying to reinvent how agents should "think" — it's first solving the infrastructure problem of how to run this new, state-accumulating workload safely and at scale in a cluster.

Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.

Find more useful knowledge and interesting products on my Homepage

One Open Source Project a Day (No. 191): Strands Agents Harness SDK — From Hand-Rolled Agent Loops to a Production-Ready Agent in One Call

WonderLab — Fri, 25 Sep 2026 09:42:33 +0000

Introduction

"Choose Strands when you would otherwise write your own agent loop: it runs in your process with no hosted control plane, and it covers the jobs a hand-rolled loop grows into."

This is the 191st article in the "One Open Source Project a Day" series. Today's project is Strands Agents (repository name harness-sdk).

Writing an AI agent that actually works in production is rarely as simple as "call the model, run the tool." In real usage you quickly run into a set of small but unavoidable problems: what happens when the conversation runs too long? What happens when tool results flood the context window? How do you resume after a session gets interrupted? Do you have to rewrite everything when switching model providers? Most teams' answer is to cobble together their own agent loop, then keep patching it as new problems surface — until that hand-rolled loop has grown session management, context compression, multi-model adapters, guardrails, and a pile of other things that should really be infrastructure, not bespoke code.

Strands Agents is an open-source agent SDK from the AWS team, shipping Python and TypeScript implementations, built to solve exactly this. It's organized in two layers: at the bottom is the Strands SDK, a model-driven, fully customizable agent loop, tool system, and multi-model support; on top of it is Strands Harness, which gives you a fully assembled, production-grade agent with a single create_harness() (or createHarness() in TypeScript) call — already configured with context management, session persistence, long-term memory, and a skill system. And the object you get back is still a plain strands.Agent, meaning everything Harness configures for you can be inspected, overridden, or replaced at will.

8,300+ Stars, 1,200+ Forks, Apache-2.0 License, developed and maintained primarily by the Strands team at AWS.

What You Will Learn

The layered relationship between the Strands SDK and Strands Harness: "build your own agent loop" vs. "get a pre-configured production-grade agent"
What create_harness() gives you by default: context management, session persistence, long-term memory, skill loading
The built-in tool set: shell, file operations, programmatic_tool_caller, and subagent delegation
Model portability: switching between Bedrock, Anthropic, OpenAI, and Gemini using the same code
MCP integration, interventions (tool-call gating), and subagent delegation patterns

Prerequisites

Basic development experience in Python or TypeScript
A basic understanding of LLM agent concepts (tool calling, multi-turn conversations)
Optional: a basic understanding of MCP (Model Context Protocol)

Project Background

What It Is

Strands Agents describes itself as "a model-driven approach to building AI agents in just a few lines of code." The README states the value proposition directly: "Choose Strands when you would otherwise write your own agent loop: it runs in your process with no hosted control plane, and it covers the jobs a hand-rolled loop grows into." This sentence captures the project's core pitch — it isn't just another thin skin over an "agent framework," it's the infrastructure that hand-rolled agent loops keep reinventing (lifecycle controls, tools, structured output, MCP, multi-agent patterns, memory, sessions, model portability, streaming, guardrails, tracing, evals), already built and ready to use.

This repository (harness-sdk) is a monorepo containing:

harness-py/, harness-ts/: the Python/TypeScript Strands Harness, assembled via create_harness() / createHarness()
strands-cli/: a CLI to prototype and chat with a harness agent from the terminal
strands-py/, strands-ts/: the underlying SDKs — the agent loop, model providers, tool system
site/: source for the official documentation site (built with Astro/Starlight)

Team and Background

Organization: strands-agents (led by the Strands team at AWS)
License: Apache License 2.0
Language support: Python 3.10+, TypeScript (Node.js 22+)
Distribution: PyPI (strands-agents, strands-harness), npm (@strands-agents/sdk, @strands-agents/harness, @strands-agents/cli)

Project Stats

⭐ GitHub Stars: 8,300+
🍴 Forks: 1,246
📄 License: Apache-2.0
🐛 Open Issues: 837 (high activity level, frequent community feedback)
📅 Created: 2025-05

What It Does

The Problem It Solves

The typical evolution of a hand-rolled agent loop:
  v1: a simple loop of model.generate() + tool calls
  ↓ conversations get longer → manually truncate history, risking dropped context
  ↓ tool results get too large → write custom summarization logic to fit the prompt
  ↓ need to switch model providers → rewrite the calling layer and parameter mapping
  ↓ need resumable sessions → implement your own session storage and recovery
  ↓ need to remember user preferences → implement your own "memory" storage and retrieval
  ↑ every step reinvents infrastructure, and each team's version is incompatible with everyone else's

Strands' approach:
  Bottom layer SDK: a model-driven agent loop + tool system + multi-model support, ready to use
  ↓
  Top layer Harness: one create_harness() call
  ↓
  Get back an agent pre-configured with context management/sessions/memory/skills
  ↑ the return value is still a plain Agent — everything Harness configures
    can be overridden, extended, or replaced

Use Cases

Quickly building a coding/ops agent that handles long-running tasks
- The kind of task shown in the official example — agent("Find the slowest test in this repo and explain why it's slow") — requiring multi-step file reading, running shell commands, and analyzing results
Production systems that need model portability
- Teams may run Bedrock, Anthropic, and OpenAI simultaneously, or need to switch providers later; Strands' provider/model string syntax makes that a one-parameter change
Assistant-style products that need long-term memory and session recovery
- Harness persists every conversation to disk by default and distills cross-conversation long-term memory via a background task, fitting scenarios like customer support or personal assistants that need to "remember" user preferences
Complex tasks that need to delegate subtasks and keep the main context from flooding with search results or multi-step exploration
- The built-in subagent tool lets the main agent hand off tasks like "search across many files," "a multi-step change," or "open-ended exploration" to a clean child agent, returning only its final answer

Quick Start

Minimal SDK usage (Python):

pip install strands-agents strands-agents-tools

from strands import Agent
from strands_tools import calculator

agent = Agent(tools=[calculator])
agent("What is the square root of 1764")

Minimal SDK usage (TypeScript):

npm install @strands-agents/sdk

import { Agent } from '@strands-agents/sdk'

const agent = new Agent()
const result = await agent.invoke('What is the square root of 1764?')
console.log(result)

Getting a production-grade agent directly with Harness (Python):

pip install strands-harness

from strands_harness import create_harness

agent = create_harness()
agent("Find the slowest test in this repo and explain why it's slow")

Getting a production-grade agent directly with Harness (TypeScript):

npm install @strands-agents/harness

import { createHarness } from '@strands-agents/harness'

const agent = await createHarness()
await agent.invoke("Find the slowest test in this repo and explain why it's slow")

Prefer not to write code at all? Install the CLI with npm install -g @strands-agents/cli and you get a strands terminal command that chats directly with a Harness agent.

Core Features

1. Layered design: SDK provides the foundation, Harness provides the assembly

The underlying strands.Agent is a set of primitives you can fully assemble yourself; create_harness() composes those primitives into a ready-to-use agent using benchmarked defaults. There's no gap between the two layers — the agent Harness returns is a plain Agent, so every piece it configures can still be overridden after the fact.

2. Production-grade capabilities out of the box

Capability	Default behavior
Model	Defaults to Claude Opus on Amazon Bedrock; supports Bedrock/Anthropic/OpenAI/Google and more
Tools	Comes with shell, file read/write/edit, `web_fetch`, and `programmatic_tool_caller` (a sandbox for tool orchestration)
Context management	Automatically summarizes older conversation turns and relocates bulky tool results to storage, leaving a short reference behind
Sessions	Every conversation is persisted to `./.agent/sessions` by default, resumable by session id
Long-term memory	Distills user preferences and project facts across conversations, saved as markdown and folded back into context
Caching	System prompt, tool definitions, and history are cached automatically wherever the provider supports it

3. A flexible model-selection syntax

A single provider/model string switches between providers using consistent syntax; reasoning effort is also unified into one set of levels (off/minimal/low/medium/high/xhigh/max), with the SDK mapping them to each provider's own parameters:

create_harness(model="anthropic/claude-opus-5")
create_harness(model="openai/gpt-5.6-sol")
create_harness(model="bedrock/global.anthropic.claude-opus-5")  # the default

4. Fine-grained control over built-in tools

builtin_tools supports two shapes: a list pins exactly the tools named, while a mapping edits the default set — False removes a tool, True adds one, and a config dict both enables and configures it:

create_harness(builtin_tools=["read"])                     # keep only read
create_harness(builtin_tools={"subagent": False})          # defaults minus subagent delegation
create_harness(builtin_tools={"web_fetch": {"model": "openai/gpt-5-mini"}})  # configure web_fetch's own model

5. Two subagent delegation patterns

The built-in subagent tool lets the main agent hand a subtask off to a full Harness child that inherits the parent's model, tools, skills, and intervention policies; alternatively, Agent.as_tool() wraps any dedicated agent into a tool — useful when you need a fixed role (like a "reviewer" agent that only reviews code). Both can coexist.

6. Native MCP and web access support

Point mcp_servers at a standard mcpServers config (a JSON file or an inline dict) and Harness connects each server, discovers its tools, and adds them to the tool list automatically; web_fetch is on by default, web_search turns on or off automatically depending on whether the model provider natively supports it, and can also be explicitly routed through Exa as a third-party search backend.

7. Agent Skills and interventions

Drop Agent Skills into ./.agent/skills and Harness loads them automatically; the interventions parameter supports "ask", "smart", a custom policy string, or even a .cedar policy file — a gate you can put in front of tool calls for approval or validation.

Project Advantages

Dimension	Hand-rolling an agent loop from scratch	Using only the underlying SDK (strands-py/ts)	Strands Harness
Time to first result	Slow — reinventing infrastructure	Moderate — loop and tool primitives already in place	Fast — one call gets you a full default configuration
Customizability	Fully open	High	High (the return value is still a native Agent — override anything)
Context/session/memory management	Build it yourself	Wire it in yourself	Built in by default
Model portability	Build your own adapter layer	Multi-provider support already exists at the SDK layer	One string parameter switch
Production readiness	Depends on the team	Depends on how much you fill in yourself	Defaults are "benchmarked and tested" out of the box

Why choose this project?

Led by the AWS team, with active updates and issue response (837 open issues signals a real, actively-used user base)
Consistent API design across Python and TypeScript — mixed-stack teams don't need two mental models
A "layered but not locked in" philosophy — Harness gives you defaults, not a closed framework you're boxed into

A Deeper Look

What Harness Actually Assembles

The parameter list of create_harness() reads like a checklist of "what a production-grade agent needs":

create_harness(
    model="bedrock/global.anthropic.claude-opus-5",  # model provider and version
    effort="auto",                          # reasoning effort
    instructions=None,                      # domain instructions appended to the system prompt
    tools=None,                             # custom tools
    mcp_servers=None,                       # MCP server configuration
    plugins=None,                           # custom plugins
    builtin_tools=None,                     # add/remove/edit the built-in tool set
    background_tasks=None,                  # background task policy
    builtin_plugins=["todos", "environment"],  # built-in feature plugins
    caching="auto",                         # context caching
    context_manager="auto",                 # context management policy
    session=True,                           # session persistence
    skills=True,                            # skill loading
    memory=True,                            # long-term memory
    interventions=None,                     # tool-call intervention policy
    **agent_kwargs,
)

The logic behind this list is clear: every capability that a hand-rolled agent loop eventually grows into — lifecycle controls, context compression, session recovery, cross-conversation memory, approval gates — is already implemented with a benchmarked default, rather than being left for each user to rediscover on their own.

Context Management: Summarization Plus Externalized Storage

The easiest thing to lose control of in a long-running task is the context window. Harness's default strategy is two-pronged: automatically summarize older conversation turns, while relocating bulky tool results (say, the large volume of file content returned by a search) to external storage, leaving only a short reference in the context. This means the context doesn't grow linearly no matter how many turns a task runs for — when an old result is actually needed, the agent can pull it back by reference instead of every turn carrying the full history.

Subagent Delegation: Two Modes for Two Different Needs

The project distinguishes between two delegation patterns, reflecting two different intents:

The built-in subagent tool: the model autonomously decides to delegate a task to a full Harness child that inherits the parent's entire configuration (model, built-in tools, plugins, skills, intervention policies). The child can narrow the tools granted by the parent, but never widen them — a deliberate safety boundary. Delegation depth is bounded by default to prevent unbounded recursive delegation.
A dedicated agent wrapped via Agent.as_tool(): suited for a fixed role with a single responsibility (like a reviewer that only does code review). Each call runs from a fresh conversation, making it a clean, focused delegate rather than a shared session.

The two can coexist: the built-in subagent handles general-purpose "exploratory" delegation, while as_tool() handles expert delegation with a fixed role and prompt. This distinction reflects two different understandings of "delegation" — one where the model judges when to hand off work itself, and one where the engineer pre-defines the role boundaries.

Built-in Tools and Sandboxed Orchestration: `programmatic_tool_caller`

One notable design choice in Harness's default tool set is programmatic_tool_caller — a sandbox where the agent writes code to chain, loop over, and parallelize its other tools, instead of going through a full "model generates call parameters → executes → model reads the result → decides next step" reasoning cycle for every single tool call. For batch, mechanical operations — like "read 50 files and count how often a pattern appears" — this kind of orchestration is far more efficient than calling tools one at a time, and it saves on unnecessary model reasoning overhead.

How Model Portability Is Implemented

The provider/model string is a thin alias mapping (bedrock, bedrock-mantle, anthropic, openai, google, ollama, litellm); for any provider outside that list, you simply pass a Model instance and the SDK uses it as-is. The effort parameter follows the same philosophy: unified into one set of enumerated levels, with the SDK mapping them to each provider's own native parameter names — but if a requested level isn't offered by a given provider, the SDK raises rather than silently degrading, avoiding the hidden failure mode of "thinking you turned on high-effort reasoning when it silently didn't take effect."

Compared to Building From Scratch: Layered but Not Locked In

Strands' design philosophy can be summarized as "layered but not locked in." Most agent frameworks give you either a set of fully free primitives that you have to assemble entirely yourself, or a tightly integrated black box that's hard to escape when you need custom behavior. Strands tries to have both: the Harness layer gives you a benchmarked, deploy-ready default configuration, but what it returns is a completely ordinary Agent — meaning you can bypass Harness's defaults at any time and reach directly into the underlying SDK's hooks, intervention mechanisms, and custom session backends, without paying the cost of "escaping the framework" through an expensive rewrite.

Project Links and Resources

Official Resources

🌟 GitHub: https://github.com/strands-agents/harness-sdk
📚 Documentation: https://strandsagents.com
📄 License: Apache License 2.0
🐛 Issue Tracker: GitHub Issues
💬 Community: Discord

Related Resources

Model Context Protocol — The tool-integration standard Strands natively supports
Agent Skills — The specification Harness's skill system follows
Strands Samples — The official repository of practical examples

Summary

Key Takeaways

A layered architecture: the underlying SDK provides a fully customizable agent loop and tool primitives, while Harness on top provides a one-call, production-grade default configuration, with no gap between the two layers
Defaults cover the common pain points of hand-rolled loops: context management, session persistence, long-term memory, and caching optimizations all work out of the box
Model portability is a first-class concern: a unified provider/model syntax and reasoning-effort levels let you switch model providers without rewriting business logic
Two subagent delegation patterns serve different intents: the built-in subagent supports model-driven autonomous delegation, while Agent.as_tool() supports engineer-defined expert roles
Led by the AWS team, with strong activity: 8,300+ Stars and 837 open issues indicate a project that's genuinely in use and actively iterated on

Who This Is For

Teams about to write their own agent loop, but haven't yet gotten to sessions, memory, or context compression: can skip straight past the "reinvent infrastructure" phase
Teams that need to stay flexible across multiple model providers: the unified model-switching syntax reduces provider lock-in risk
Developers wanting to quickly build a coding/ops agent that handles long-running, multi-step operations: Harness's default tool set (shell, file operations, orchestration sandbox) covers most scenarios
Teams mixing Python and TypeScript stacks: both SDKs keep a consistent API design, avoiding two separate mental models

One-Line Verdict

Strands Agents doesn't wrap "building an agent" into a closed black box — it first honestly builds out the infrastructure that a hand-rolled loop would eventually need to add anyway, then lets you decide whether to start from it.

Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.

Find more useful knowledge and interesting products on my Homepage

LLM-Driven Automated Testing Series (04): Web UI Automation — Midscene's Vision-Driven Scripting

WonderLab — Fri, 25 Sep 2026 08:43:33 +0000

Two ways to write the same test step

A traditional Playwright script needs a stable selector before it can click a button:

await page.locator('[data-testid="submit-btn"]').click();

This line assumes a stable data-testid actually exists on the page. If that button is a bare icon, drawn on a <canvas>, or buried inside a cross-origin iframe, this approach breaks down entirely — with no semantic annotation, a selector has nothing to grab onto.

Midscene, open-sourced by ByteDance's web-infra team, answers this differently: skip the selector, hand the full screenshot to a vision model, and describe what to do in natural language:

await agent.aiTap('submit button');

This maps directly onto Path 1: visual grounding from the vocabulary built in the previous post (02) — no dependency on any structured interface the app exposes, purely "looking." This post digs into what that path actually looks like in a real, production-shaped testing framework.

Four core APIs, four levels of granularity

Midscene isn't a single "control the page with natural language" black box — it splits different levels of need into four APIs with clearly separated responsibilities:

aiTap / aiInput   —— Atomic actions: precise, single-step "click" or "type"
      ↓
aiAct             —— Autonomous flow: hand a multi-step natural-language task
                     to the agent and let it plan its own execution
                     (e.g. "search and sort by price ascending")
      ↓
aiQuery           —— Structured extraction: "read" structured data off the page
                     (e.g. extract all item names and prices in the current cart)
      ↓
aiAssert          —— Visual assertion: verify whether "what a user would actually see"
                     matches expectations (e.g. is this button highlighted, is the layout broken)

The design intent behind this split: not everything in a test script should be handed off to autonomous agent decision-making. Atomic actions (aiTap/aiInput) preserve the determinism and readability of traditional scripts, where every single step is explicitly specified by a human. The autonomous flow (aiAct) is reserved for exploratory tasks where the number of steps isn't fixed in advance. This is fundamentally different from a pure Computer Use approach, where a single model decides every action end to end — Midscene lets the test engineer choose exactly where uncertainty gets introduced, rather than betting the entire test flow on the model's planning ability.

aiAssert deserves a callout of its own: it directly addresses the oracle problem from post 01. A lot of visual correctness can't be judged by pixel diffing or checking a DOM attribute — e.g., "does this button look disabled?" aiAssert hands that judgment to the vision model as a semantic decision, instead of hardcoding a CSS class check. This idea gets expanded further in post 12 (visual and semantic regression testing).

Core implementation one: three model roles, not one model doing everything

The previous section said aiAct "hands a multi-step task to the agent to plan and execute on its own," but glossed over how that "planning" actually maps onto model calls. Midscene's answer is to split a single automation task across three roles (its Model Strategy), each of which can be bound to a different underlying model:

Default model    —— Handles element locating, plus whatever workload
                     Planning/Insight don't take over — the base model
                     that carries most automation needs by default
Planning model    —— Enhances planning for complex goals, multi-step
                     tasks, and branching scenarios
Insight model     —— Enhances data extraction, assertion judgment,
                     and page understanding

The key design choice is that these three roles can each be bound to a different underlying model — the official docs put it as "users can layer Planning and Insight models on top of the Default model," letting each model play to whichever role it's actually good at: pair "planning" with a model that reasons better, pair "locating" with a model that's more precise visually, instead of forcing one general-purpose model to carry both entirely different capability demands at once.

That flexibility isn't free, though. The official docs explicitly warn: "this collaboration extends Midscene's ability to handle complex tasks, but also increases task latency and token consumption." The recommended usage is therefore incremental: get things working with the Default model first, and only bring in a Planning or Insight model once you hit a clear capability bottleneck — not turn all three on by default.

This role split is itself an engineering-level response to post 02's conclusion that "locating precision isn't the only variable": explicitly separating "planning" and "locating" as independent capability needs, each with its own model choice, makes it easier to optimize precisely against a bottleneck than a single model trying to cover both — at the cost of extra latency and call overhead for that flexibility.

Core implementation two: aiAct's execution loop — when to replan, when to split out locating

aiAct doesn't just call the model once, get back a full step list, and execute it end to end. The official docs describe it as continuously using AI to plan the next action from the latest UI state, until the goal is complete — meaning it re-observes the current screen after every single step before deciding the next one, rather than sticking rigidly to whatever plan got generated when the task started. The benefit: if a permission dialog pops up mid-flow or a loading delay shifts the layout, planning naturally adjusts from the latest state, without needing extra branch logic hand-written for "what if an unexpected dialog appears."

This loop isn't infinite replanning, either — a replanningCycleLimit (max replanning cycles) parameter backstops it, with a default that varies by model type (20 for standard models, 40 for UI-TARS, 100 for AutoGLM). Once that limit is exceeded, or an assertion embedded in the prompt fails, aiAct throws immediately rather than continuing to retry indefinitely. This is the same design philosophy as post 03's "fail conservatively" idea: better to surface an error quickly for a human to judge than let a task that can't produce a result spin silently.

By default, aiAct handles "plan the next step" and "locate the target element" in a single model call. But complex tasks sometimes need those two split into separate calls — that's what the deepThink parameter does: turning it on splits planning and locating into two independent calls, trading more calls and latency for better stability on complex tasks.

There's a historical wrinkle here worth clarifying, since it's an easy source of confusion: the parameter name deepThink has meant two different things across different APIs. In aiAct(), it has always been the "planning mode" switch; but in single-step methods like aiTap, the older deepThink meant "enhanced element locating." To resolve that ambiguity, Midscene renamed the "enhanced locating" concept to its own deepLocate parameter, leaving deepThink consistently meaning "planning mode" — and inside aiAct(), deepThink (whether to split out planning) and deepLocate (whether to boost locating precision) coexist as two independent parameters, not mutually exclusive ones.

Relationship to Playwright: augmentation, not replacement

Midscene didn't reinvent a browser-control protocol — it wraps the existing Playwright page object via PlaywrightAgent. Everything Playwright already does (navigation, waits, traditional selector-based assertions) stays fully intact; Midscene's four AI APIs are simply layered on top. The same integration approach also supports Puppeteer.

The practical implication: migration cost is incremental, not a rewrite. An existing Playwright test suite can start by swapping aiAssert into the handful of assertion points most likely to break on a UI redesign, leaving everything else unchanged — you don't need to rip out the whole framework just to introduce visual locating.

Going further, Midscene turns this Agent API into a unified cross-platform interface: "the same API covers Web, Android, iOS, HarmonyOS, and desktop applications." That means a test engineer who's already internalized aiTap/aiQuery/aiAssert doesn't need to learn a new API when switching to mobile or desktop automation — only the underlying screenshot-capture and event-injection implementation changes, not the call pattern. (Specific mobile projects, each with their own mobile-specific optimizations, get their own detailed treatment in posts 05-09.)

The caching mechanism: how the biggest pain point of visual approaches gets mitigated

The previous post already covered the cost of Path 1 (visual grounding): every locating step requires a model inference pass — slower and more expensive than reading structured data. If a test suite has hundreds of test cases, each with dozens of steps, and every single step makes a real vision-model call, that cost is untenable to run in CI.

Midscene's answer is to cache two distinct kinds of output separately, rather than lumping everything under a generic "cache the locate result":

Planning cache (for ai / aiAct)
  key   = the raw natural-language instruction the user wrote
  value = the execution plan the model returned (a sequence of concrete steps)

Locate cache (for aiLocate / aiTap and other single-step locating calls)
  key   = the locate prompt
  value = the matched element's XPath

Query-type operations (aiQuery/aiBoolean/aiAssert) are explicitly excluded from caching altogether — because the whole point of those calls is to "read the page's actual current state," and caching a past judgment would be semantically wrong.

The locate cache isn't "cache the XPath and trust it blindly." Every run first re-validates whether that XPath is still valid on the current page, and invalidates it if either "the text content of the element at that same XPath has changed since it was cached" or "the page's DOM structure has changed relative to when it was cached." This is a deliberately strict matching policy — better to invalidate a bit too often and trigger a fresh locate than to click, on a page that's already changed, an element that merely looks visually similar but is no longer semantically the right one. The planning-cache side has a similarly conservative design: if a cached aiAct plan fails during execution, Midscene falls back to the normal AI planning flow and clears the stale cache entry; but the freshly-regenerated plan from that fallback doesn't get written back into the cache under the original prompt, since it may not represent a complete, correct plan starting "from the task's original state."

There's one more boundary condition worth calling out on its own: the locate cache is structurally unusable in rendering scenarios with unstable structure — shapes drawn inside a Canvas have no corresponding DOM nodes, so there's naturally no XPath to cache; cross-origin iframes are inaccessible internally due to browser security policy; Closed Shadow DOM is entirely inaccessible from the outside; WebGL/dynamic SVG have inherently unstable DOM structure, so even a cached result would likely invalidate on the very next run. This lines up exactly with the point made at the top of this post: canvas/iframe/custom rendering are precisely the scenarios where a pure-visual approach has its core advantage — and also precisely the scenarios where the caching mechanism is structurally unavailable and every run genuinely needs a model call. The cost savings caching offers and the pure-visual approach's core advantage scenarios sit in structural tension with each other — that isn't a design flaw in Midscene, it's an inevitable consequence of those interfaces lacking structured information in the first place.

This design effectively downgrades "visual locating" from "must be used every single time" to "a fallback mechanism." Under normal conditions, tests run a cached, deterministic sequence of operations — with speed and cost essentially indistinguishable from a traditional selector-based script. Only when the UI genuinely changes and invalidates the cache does it fall back to the vision model for fresh locating — though in canvas/iframe and other structurally-missing scenarios, that "fallback mechanism" is in fact the only mechanism available. In a sense, this already carries a flavor of the "self-healing locators" topic covered in post 10 — the difference is that post 10's self-healing logic focuses on "how to recover after a locate failure," whereas this caching mechanism solves a different problem: "most of the time, locating shouldn't need to be triggered at all."

Real benchmark numbers

Midscene's official site publishes three benchmark results, worth comparing against the ScreenSpot/OSWorld numbers from post 02:

AndroidWorld Benchmark      Pass@1 93.1%    Pass@3 97.4%
MobileWorld Benchmark       Pass@1 78.6% (92/117 passed)
AppControlBench Benchmark   Pass@1 96.7% (58 items passed)

These are all mobile benchmarks (AndroidWorld gets a full treatment in post 05), but there's a comparison worth flagging here: post 02 mentioned OSWorld's early baseline models achieving only a 12.24% task success rate, while Midscene achieves a 93.1% Pass@1 on AndroidWorld. This isn't saying Midscene's underlying model is dramatically stronger than what OSWorld tested — the task types and evaluation paradigms differ. Benchmarks like AndroidWorld/AppControlBench, purpose-built for mobile GUI agents, have more clearly-scoped tasks, and combined with dedicated planning + retry mechanisms (Pass@3 being 4 points higher than Pass@1 shows that allowing retries genuinely recovers some single-attempt locating failures), can hit success rates far above OSWorld's open-ended desktop tasks. This reinforces post 02's conclusion: end-to-end task success depends on more than grounding precision alone — planning, retries, and task-scope design matter just as much.

Comparing paths against Stagehand

Midscene isn't the only open-source project in this space. Stagehand (open-sourced by Browserbase) is a worthwhile comparison — it also layers a natural-language API (act/extract/observe) on top of Playwright, but its underlying technical path leans more toward Path 2 from post 02: DOM/accessibility-tree semantic understanding — it prioritizes structured information exposed by the browser for locating and action, with visual capability as a supplement rather than the core.

                    Primary locating method       Best fit                            Cost
──────────────────────────────────────────────────────────────────────────────────────────
Midscene            Pure vision (screenshot+model)  canvas/iframe/icon-only scenarios    Requires a model
                                                     lacking semantic annotation           inference call per
                                                                                            locate (mitigated by cache)
Stagehand           DOM/accessibility tree first,   Conventional web apps with            Weaker fallback than a
                    vision as backup                reliable structured info              pure-visual approach when
                                                                                            structure is missing

This comparison echoes post 02's core conclusion: the two paths aren't a matter of one being technically superior — they're trade-offs suited to different scenarios. If the test target is a conventional web app with complete structural info and proper semantic annotation, a DOM-first approach (like Stagehand) is usually faster and cheaper. If the test target itself involves heavy canvas rendering, custom-rendered widgets, or needs one unified test logic maintained across Web/mobile/desktop, a pure-visual approach (like Midscene) has a clearer edge in generality.

Summary

Midscene takes the pure visual grounding path — no dependency on selectors or semantic annotations, feeding screenshots directly to a vision model — which lets it cover canvas rendering, icon-only elements, and cross-origin iframes that traditional approaches can't locate at all
The four core APIs (aiTap/aiInput for atomic actions, aiAct for autonomous flows, aiQuery for structured extraction, aiAssert for visual assertions) let the test engineer choose exactly which layer to introduce uncertainty into, rather than betting the entire flow on the model's planning ability
The Default/Planning/Insight model-role split lets "planning" and "locating" be bound to different models and tuned independently, at the cost of extra latency and token spend — the official recommendation is to adopt Planning/Insight incrementally, not enable all three by default
aiAct's execution loop replans from the latest UI state after every step, rather than sticking to whatever plan was generated at the start; the deepThink parameter controls whether planning and locating happen in one model call or two separate ones, trading extra calls for stability on complex tasks
The caching mechanism caches planning output (keyed on the raw instruction) and locate results (keyed on the locate prompt, valued as XPath) separately, with the locate cache enforcing strict invalidation on text or DOM-structure changes; it's structurally unusable in canvas/iframe and similar structure-missing scenarios — precisely where the pure-visual approach's core advantage and the cache mechanism's blind spot overlap
Compared against Stagehand (DOM/accessibility-tree first), the two paths aren't a matter of one being better — they map onto two different scenarios: unreliable structural information versus complete, reliable structural information

Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.

Find more useful knowledge and interesting products on my Homepage

One Open Source Project a Day (No. 190): AIO Sandbox — Browser, Shell, File, MCP, and VSCode Server in One Docker Container

WonderLab — Wed, 23 Sep 2026 11:11:53 +0000

Introduction

"What an agent needs isn't five tools each doing their own thing — it's one environment where every tool can see the same file."

This is the 190th article in the "One Open Source Project a Day" series. Today's project is AIO Sandbox.

Setting up an execution environment for AI agents runs into a small but persistent annoyance: browser automation lives in one container, code execution in another sandbox, file storage in a third service. Passing files and state between them means hand-writing glue code — a file downloaded in the browser can't be read by a shell script without an extra hop; results generated in Jupyter need to be manually relocated before a browser tool can use them. Every new tool you add means another round of custom integration work.

AIO Sandbox (full project name agent-infra/sandbox) is built to solve exactly this "tool island" problem. It packs browser automation, a shell terminal, file operations, MCP servers, and VSCode Server into a single Docker container. Its core selling point is a unified filesystem shared across all tools — a file downloaded in the browser is immediately usable via shell commands or file APIs, with zero relocation step in between.

6.0k Stars, Apache-2.0 License, with working integrations verified against Browser Use, LangChain, OpenAI Assistants, and MiniMax.

What You Will Learn

AIO Sandbox's layered architecture: browser/VNC layer, dev tools layer, MCP integration layer, infrastructure layer
How the "unified filesystem" eliminates data-relocation costs between tools
The four built-in MCP servers: browser, file, shell, and markitdown
How to deploy it three ways: Docker, Docker Compose, and Kubernetes
How it actually integrates with frameworks like Browser Use, LangChain, and OpenAI Assistants

Prerequisites

Familiarity with basic Docker operations (run, compose)
Basic understanding of MCP (Model Context Protocol)
Optional: basic understanding of Chrome DevTools Protocol (CDP) and browser automation

Project Background

What It Is

AIO Sandbox's official positioning: "an all-in-one agent sandbox environment that unifies Browser, Shell, File, MCP operations, and VSCode Server in a single Docker container," built on cloud-native lightweight sandbox technology. The project documentation directly names the pain point of traditional sandboxes: "traditional sandboxes are typically single-purpose (browser, code, or shell), making file sharing and functional coordination extremely challenging" — AIO Sandbox is the unified answer to that exact pain point.

Team and Background

Organization: agent-infra
License: Apache License 2.0
Deployment: Docker container (Kubernetes also supported)
Official SDKs: Python (agent-sandbox), TypeScript/JavaScript (@agent-infra/sandbox), Go (agent-sandbox-sdk-go)

Project Stats

⭐ GitHub Stars: 6,000+
🍴 Forks: 534
👀 Watchers: 32
📄 License: Apache-2.0
📊 Commits: 130 (main branch)

What It Does

The Problem It Solves

Building an agent environment without AIO Sandbox:
  Browser automation → container A (Playwright/Puppeteer)
  Code execution     → container B (Jupyter/Sandbox Fusion)
  File storage       → a separate file service C
  ↑ the three don't talk to each other — files downloaded by the browser
    need manual relocation before container B can use them
  ↑ every new tool integrated means rewriting cross-container data-passing
    logic from scratch

AIO Sandbox's approach:
  Browser + Shell + File + MCP + VSCode all packed into one container
  ↓ a unified filesystem as the shared underlying layer
  Browser downloads a file → shell reads it directly → Jupyter processes
  it directly → file API writes it back directly
  ↑ every tool sees the same filesystem, no relocation needed

Use Cases

AI agent tool execution and browser automation
- An agent needs to open web pages, click through interactions, scrape content, and also execute code to process what it scraped
Web-page-to-Markdown processing pipelines
- The official example: use Playwright over CDP to scrape a page → convert HTML via Jupyter's markdownify → read the result back through the file API, all inside the same container
Agent development that needs visual debugging
- Watch browser automation execute live through a VNC remote desktop, or debug sandbox code directly via VSCode Server
A unified execution backend for multiple framework integrations
- Serve as the shared execution backend for Browser Use, LangChain, OpenAI Assistants, and other frameworks, instead of building a separate execution environment for each

Quick Start

One-command Docker launch:

docker run --security-opt seccomp=unconfined --rm -it \
  -e SANDBOX_API_KEY=your-secret-key \
  -p 127.0.0.1:8080:8080 ghcr.io/agent-infra/sandbox:latest

Once running, access:

/v1/docs — API documentation
/vnc/index.html — VNC remote desktop
/code-server/ — VSCode Server
/mcp — MCP service entry point

Users in mainland China can substitute an alternate mirror registry for ghcr.io. For production, use a pinned version tag (e.g., 1.11.0) instead of latest to keep deployments reproducible.

Install the SDK:

# Python
pip install agent-sandbox

# TypeScript/JavaScript
npm install @agent-infra/sandbox

# Go
go get github.com/agent-infra/sandbox-sdk-go

Core Features

1. Unified Filesystem

Shared across the browser, shell, file operations, and Jupyter — the core design of the entire project, eliminating the glue code needed to pass files between tools.

2. Four Built-in MCP Servers

MCP Server	Core Capabilities
browser	navigate, screenshot, click, type, scroll
file	read, write, list, search, replace
shell	exec, create_session, kill
markitdown	convert, extract_text, extract_images

3. Multiple Access Interfaces

VNC (remote desktop), VSCode Server, Jupyter Notebook, WebSocket terminal — the same sandbox can be accessed in whatever way best fits the current task.

4. Core REST API

Endpoints like /v1/sandbox, /v1/shell/exec, /v1/file/read, /v1/file/write, /v1/browser/screenshot, and /v1/jupyter/execute cover the common execution needs of an agent.

5. Three Deployment Options

Deployment	Best For
Docker (single command)	Local development, quick trials
Docker Compose	Scenarios needing volume persistence (with `shm_size: "2gb"` and similar config)
Kubernetes	Production cluster deployment (with resource limits like `memory: "2Gi"`, `cpu: "1000m"`)

6. Authentication Mechanism

Enabled via the SANDBOX_API_KEY environment variable, supporting header, Bearer token, or query-parameter auth uniformly across API, JupyterLab, and VNC access; leaving it unset keeps services open (for backward compatibility).

A Deeper Look

Layered Architecture: From Visualization Down to Infrastructure

AIO Sandbox's architecture can be understood as a four-layer stack:

Browser + VNC layer (visualization)
  ↓
VSCode Server + Shell terminal + File operations layer (dev layer)
  ↓
MCP Hub + Sandbox Fusion layer (integration layer)
  ↓
Preview Proxy + Service monitoring layer (infrastructure layer)

All layers share the same container and the same filesystem. The key insight in this design: it doesn't stitch multiple independent services together over a network interface — it runs them all within the same process environment and filesystem namespace. This eliminates the serialization overhead and consistency problems that come with cross-container/cross-service communication — there's no need to solve the classic distributed-systems question of "when will the shell container see the file the browser container just wrote."

The Real Pain Point the Unified Filesystem Solves

Look at the typical workflow the official docs describe, and the design choice becomes concrete:

1. Use Playwright over CDP, connected to the sandbox's browser, to scrape a page
2. The page content → written directly to the sandbox filesystem
3. Call Jupyter's markdownify inside the sandbox to convert HTML to Markdown
4. The converted result → read back directly through the file API

In a traditional multi-container setup, steps 2 and 3 require a cross-container file transfer — maybe a volume mount, an object-storage relay, or an API call carrying the file content as payload. In AIO Sandbox, this step is just an ordinary file write and read within the same filesystem — no extra transport layer, no extra failure point.

Why MCP as the Unified Access Protocol Matters

AIO Sandbox chose to wrap its browser, file, shell, and markitdown tools in the MCP protocol rather than defining its own bespoke REST API spec for each. The benefit: any agent framework that already supports an MCP client (Claude Code, or any other MCP-compatible tool) can plug into the sandbox's full capability set with "zero configuration" — no adapter layer needs to be written specifically for AIO Sandbox. Compared to the traditional pattern where "every sandbox defines its own private API, and integrating a new tool means writing a new adapter layer," this eliminates a large amount of repeated integration work.

How Framework Integrations Actually Work

The README shows several typical framework integrations, and the pattern reveals the sandbox's design intent: to be called as an execution backend, not to become its own agent orchestration framework:

Framework	Integration Method
browser-use	Connect a `BrowserSession` to the sandbox's CDP endpoint, reusing the browser instance already running inside the sandbox
LangChain	Wrap `shell.exec_command` in a custom `BaseTool` subclass and plug it into the tool chain
OpenAI Assistants / Function Calling	Expose a `run_code` tool that routes to the sandbox's internal Jupyter or Node.js execution
MiniMax	Call via the OpenAI-compatible API (`base_url="https://api.minimax.io/v1"`), noting that MiniMax requires `temperature > 0`

This "content to be an execution layer, not competing for the orchestration seat" positioning means it can be grafted broadly onto existing agent ecosystems without requiring users to give up an orchestration framework they're already using.

How It Compares to Traditional Single-Purpose Sandboxes

Dimension	Traditional Single-Purpose Sandbox (pure browser/pure code execution)	AIO Sandbox
Tool coverage	Usually covers one category of capability	Browser+Shell+File+MCP+VSCode all-in-one
Cross-tool file sharing	❌ Requires manual relocation/extra relay layer	✅ Unified filesystem, natively shared
Access protocol	Each defines its own API	✅ Standardized MCP protocol
Visual debugging	Depends on the sandbox type, often missing	✅ Dual VNC + VSCode Server channels
Deployment options	Depends on the project	✅ Three tiers: Docker / Compose / K8s
Positioning	A standalone tool	Content to be an execution backend, compatible with mainstream agent frameworks

Project Links and Resources

Official Resources

🌟 GitHub: https://github.com/agent-infra/sandbox
📄 License: Apache License 2.0
🐛 Issues: GitHub Issues
💬 Community: GitHub Discussions

Related Resources

Model Context Protocol — The standard protocol AIO Sandbox uses to wrap its tool capabilities
Chrome DevTools Protocol — The underlying protocol behind the sandbox's browser automation capability
Browser Use — A browser automation framework with a verified integration into AIO Sandbox

Summary

Key Takeaways

A unified filesystem is the core design: eliminates the file-relocation cost between the browser, shell, and code execution, with all tools sharing the same filesystem
Same-container architecture instead of cross-container orchestration: browser, shell, file, MCP, and VSCode all run inside the same process environment, avoiding distributed consistency problems
MCP as the unified access protocol: all four built-in tools (browser/file/shell/markitdown) are exposed via MCP, compatible with the existing MCP client ecosystem
Content to be an execution backend: its integrations with Browser Use, LangChain, OpenAI Assistants, and MiniMax are all about "being called," not "replacing" — it doesn't compete with existing orchestration frameworks
Three deployment tiers cover different scenarios: a complete deployment path from a single local command to a Kubernetes production cluster

Who This Is For

AI agent developers: who need one unified execution environment instead of building separate infrastructure for browser automation, code execution, and file storage
Teams using multiple frameworks: already using LangChain, Browser Use, and similar tools, and want a shared execution backend instead of reinventing it
Scenarios needing visual debugging: watch an agent's execution live via VNC/VSCode Server instead of only reading logs
Users of the MCP ecosystem: who want an out-of-the-box set of MCP servers covering browser/file/shell without maintaining their own

One-Line Verdict

AIO Sandbox isn't trying to build a smarter agent framework — it's solving a more foundational problem first: making sure the tools an agent uses can at least see the same filesystem.

Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.

Find more useful knowledge and interesting products on my Homepage

LLM-Driven Automated Testing Series (03): Unit Test Generation — TestGen-LLM and Qodo Cover

WonderLab — Wed, 23 Sep 2026 11:10:45 +0000

Where "just write some tests" goes wrong

Hand a function to an LLM and ask it to "generate unit tests to improve coverage," and the most common outcome isn't tests that are merely mediocre — it's tests that look fine but test nothing at all:

def test_calculate_discount():
    result = calculate_discount(100, 0.1)
    assert result is not None  # this assertion will never fail

def test_process_order():
    # doesn't even compile: the mock's return type doesn't match the real interface
    mock_client = Mock(return_value="success")
    process_order(mock_client)

These tests have coverage numbers attached to them — sometimes they even make the coverage report look "better" — but they provide zero protection against real regressions. assert result is not None will never fail, which means it asserts nothing. Worse still is a second failure mode: LLM-generated tests sometimes don't even compile or run, and dropping them straight into CI just manufactures new red X's.

Meta's 2024 paper "Automated Unit Test Improvement using Large Language Models at Meta" tackles exactly this problem, with a tool called TestGen-LLM. Its core idea isn't "train a smarter model to write tests" — it's the opposite: assume the LLM's output is probably garbage, build a mechanical filter pipeline that discards anything substandard, and keep only the small fraction that can prove its own value.

TestGen-LLM's four-layer filter

The paper calls this approach Assured Offline LLM-Based Software Engineering (Assured LLMSE): the LLM is just one generation step in a pipeline, and its output is never trusted directly — it has to pass a series of verifiable checks, and only what clears every check gets recommended to a human. That word "assured" is the key distinction from "just use the model's output as the result."

Concretely, this is a four-stage cascade of elimination:

LLM generates a candidate test
      ↓
① Build Filter
   The candidate code must fully compile within the app's existing build infrastructure
   Fails to build → discarded immediately
      ↓
② Pass Filter (first execution)
   A test that compiles must pass on its first run
   — this step doesn't try to judge "did the failure reveal a real bug,"
     because there's currently no automated way to distinguish
     a real bug from a bad assertion, so the conservative default is:
     fails on first run → discard
      ↓
③ Flakiness Filter (stability)
   Run the test 5 times in a row — it must pass every single time
   Even one failure out of five → deemed flaky, discarded
      ↓
④ Coverage Filter
   Must prove it actually increases code coverage
   Fails to improve coverage → discarded, even if it passed the first three filters
      ↓
Only tests that survive all four layers get recommended to an engineer for review

The design philosophy is narrow the funnel at every stage — it's better to lose a valuable test than let a garbage test through. Layer three is particularly worth noting — running a test five times to filter flakiness is a technique Qodo Cover's roadmap later borrowed directly (more on that below).

One point worth clarifying, since it's easy to get wrong: mutation testing is not part of TestGen-LLM's filter pipeline. The paper does mention that mutation coverage would theoretically be the best-performing criterion, but it explicitly lists this as future work, noting that "it is challenging to deploy such computationally demanding techniques at the scale we would require." In other words, the mutation-testing idea is directionally right, but too expensive to run in practice — so this version relies on the much lighter "did coverage improve" signal instead of the more precise but far costlier "were the mutants of each line actually killed."

Real numbers: how this performed inside Meta

The paper reports two sets of numbers — one from a lab-style evaluation, one from real engineer usage — and the gap between them is itself informative.

Lab evaluation: Instagram's Reels/Stories components

Tested on 86 Kotlin components (31 Stories, 55 Reels):

75%  of test classes: got at least one new test case that compiles correctly
57%  of test classes: compiles AND passes reliably (cleared the first three filters)
25%  of test classes: compiles, passes reliably, AND increases line coverage (cleared all four filters)

This tells a very practical story: there's a big drop-off between "can an LLM generate a test" and "can an LLM generate a test that's actually worth something." Three-quarters of components got a test that "looks like it compiles," but only a quarter actually improved coverage — the middle three filters are precisely what's cutting away the "looks useful but isn't" output.

Real-world usage: three test-a-thons

More convincing than the lab numbers is data from three internal test-a-thons — sprint events where engineers actually reviewed the generated diffs:

                        Diffs generated   Accepted by engineers   Acceptance rate
──────────────────────────────────────────────────────────────────────────────
Instagram (1st round)        17                   16                  94%
Instagram (2nd round)        42                   36                  86%
Facebook                    280                   144                 51% (~70% among reviewed diffs)
──────────────────────────────────────────────────────────────────────────────
Total: 196 out of 1,979 test classes were successfully improved (~10%)
       73% of TestGen-LLM's improvement suggestions were accepted for production

Comparing the two datasets side by side: in the lab, roughly 25% of test classes "clear all four filters." In the real-world test-a-thons, 73% of improvements were accepted by engineers. These numbers don't contradict each other — the filters eliminate output that's mechanically substandard (doesn't build, unstable, doesn't improve coverage), while "engineer acceptance" measures whether a test that's already cleared the mechanical bar is actually worth merging. In other words, the four filters solve "don't let garbage through" — they don't guarantee "everything generated is valuable." That judgment still needs a human.

The open-source implementation: how Qodo Cover turns this into a CLI tool

TestGen-LLM itself is an internal Meta tool and was never open-sourced. Qodo Cover (formerly Codium's Cover-Agent) is the first public open-source implementation of this same approach, runnable directly against your own project.

Its architecture maps the paper's "filter" concept onto four concrete components:

Prompt Builder
   Gathers necessary context from the codebase (source file, existing tests,
   coverage report) and assembles it into a prompt for the LLM
        ↓
AI Caller
   Calls the LLM with the assembled prompt to generate a candidate test
   (via LiteLLM, so the underlying model provider is swappable)
        ↓
Test Runner
   Actually runs the project's test command against the newly generated test
        ↓
Coverage Parser
   Parses the coverage report (Cobertura/JaCoCo, etc.) and checks
   whether coverage genuinely increased
        ↓
   Passes → keep the test, move to next iteration
   Fails  → discard, generate a new candidate

This is effectively a concrete implementation of TestGen-LLM's Pass Filter + Coverage Filter (the Build Filter is implicit in a failed Test Runner execution; a Flakiness Filter is on the roadmap, explicitly described as planned to "check test flakiness, e.g. by running 5 times as suggested by TestGen-LLM").

Usage is a CLI-driven loop: specify the source file, test file, coverage-report path, a target coverage percentage (--desired-coverage), and a max iteration count (--max-iterations); the tool repeatedly generates candidate tests, runs them, and checks coverage until it hits the target or runs out of iterations. This loop can be wired straight into a CI pipeline, or run locally as a one-shot "bump this module's coverage" command.

Worth flagging: Qodo Cover, the open-source project, is no longer actively maintained as of mid-2025 (the README explicitly marks it "no longer maintained" and suggests forking if you want to keep developing it). That doesn't diminish its value as a reference implementation of the TestGen-LLM approach, but it does raise a maintainability risk if you'd depend on it in production — a recurring theme in this series: open-source proof-of-concept projects and production-ready tools are often not the same thing.

Where this approach actually holds up

Putting Meta's lab data and test-a-thon data together, you can map out where this method genuinely works and where it doesn't.

Works well: modules with relatively self-contained logic and clear dependencies — the LLM can reasonably infer input/output relationships from function signatures and existing test style, and the four-layer filter effectively strips out most garbage output.

Doesn't work well:

Code that depends heavily on external state or complex mocks — LLM-generated mocks frequently don't match the real interface's actual behavior, which tends to get caught by the Build Filter or Pass Filter, resulting in a low survival rate
Code that requires deep business context to write a meaningful assertion — a test might "run successfully" but assert something meaningless (like assert result is not None in the opening example). The coverage filter only checks whether the code path was executed, not whether the assertion itself means anything — this is a blind spot the four-layer filter never addresses
Scenarios demanding extreme determinism and reproducibility — the flakiness filter only runs five times, so statistically it can still miss a test that's fundamentally unstable

One point worth emphasizing: the four-layer filter guarantees that a test is mechanically valid — it does not guarantee that the test's assertions are semantically correct for the business logic. Coverage improvement is an automatable proxy metric, but "did this test actually check the thing I care about" still needs human review. That's exactly why Meta's own data treats "cleared all filters" and "accepted by engineers" as two distinct numbers that each need to be examined separately.

Summary

LLM-generated tests are probably garbage by default — they may not compile, assert something trivial, or contribute nothing to coverage. TestGen-LLM's fix isn't a smarter model, it's a four-layer mechanical filter (build, first-run, stability, coverage) that eliminates substandard output stage by stage
Mutation testing is not currently part of TestGen-LLM's methodology — the paper explicitly lists it as future work constrained by computational cost, and uses the much lighter "did coverage improve" signal instead
The lab number (25% clear all filters) and the real-world number (73% accepted by engineers) measure two different things — the former is mechanical validity, the latter is human judgment of value, and the four-layer filter only solves the former
The open-source implementation Qodo Cover turns this idea into a runnable CLI/CI iteration loop, but the project itself is no longer maintained — maintainability risk needs to be weighed before relying on it in production

Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.

Find more useful knowledge and interesting products on my Homepage

LLM-Driven Automated Testing Series (02): The Unified Technical Foundation — Visual Grounding / DOM Semantics / Computer Use

WonderLab — Tue, 22 Sep 2026 01:21:00 +0000

Same problem, three answers

"Find the submit button on this page and click it."

Hand that instruction to three different open-source GUI agent projects and you'll get three entirely different implementations:

One feeds the full screenshot to a specially-trained visual grounding model, which directly outputs a coordinate like (842, 613)
Another parses the page's DOM or accessibility tree, finds the node with role=button, name="Submit", and calls its click handler directly
A third skips any dedicated locating step entirely — a general-purpose multimodal model just looks at the screenshot and decides "click here" on its own

These three paths aren't interchangeable implementation details — they're distinct architectural decisions, each with clear trade-offs and boundaries. This post builds the vocabulary the next 12 case-study posts will reuse. Without it, every post would have to re-explain "which locating approach this project uses" from scratch.

Path 1: Pixel-Based Visual Grounding Models

How it works

This path treats the GUI as an image. Given a natural-language reference ("the submit button") and a screenshot, the model directly outputs the element's coordinates or bounding box on screen.

This is a dedicated sub-task in the literature called GUI grounding: it doesn't decide "what to do next" — it only maps a natural-language reference precisely onto screen coordinates. Representative open-source models include UGround, Aria-UI, and Alibaba's self-trained GUI-Owl (used in Mobile-Agent-v3, covered in a later post).

The typical architecture is two-stage:

User task: "Submit this form"
        ↓
Planning model (can be a general LLM): "Next step is to click the submit button"
        ↓
Visual grounding model (dedicated, e.g. UGround): "submit button" + screenshot → (842, 613)
        ↓
Execution layer: click at (842, 613)

Planning and grounding are two separate models, each specialized for one job.

Trade-offs

Advantages: no dependency on any application-internal structured data — pure "looking," which makes it naturally cross-platform (web, mobile, desktop, even game UIs — anywhere you can take a screenshot). A key result from the UGround paper: this path beats accessibility-tree-dependent approaches (Path 2 below) by up to 20 absolute percentage points on multiple GUI grounding tasks, because real-world accessibility trees are often incomplete or unreliable.

Costs:

Every locating step requires a visual model inference pass — slower and more expensive than reading structured data directly
Accuracy drops sharply for small elements and dense layouts (the ScreenSpot-Pro numbers below quantify how much)
The grounding model itself needs dedicated training and maintenance — you can't get high accuracy "for free" from a general-purpose model

Path 2: DOM / Accessibility-Tree Structured Understanding

How it works

This path doesn't look at pixels — it looks at structure. Web pages have a DOM tree; desktop and mobile apps have an accessibility tree (the semantic view the OS exposes for screen readers and other assistive tools). Serialize that tree into text and hand it to an LLM directly: "Given this structure, which node is the submit button?"

Once the target node is found, subsequent actions don't need coordinates at all — you call the node's exposed click(), setValue(), etc. directly, the same mechanism traditional Selenium/Playwright scripts already use. The only thing that changes is that "finding the node" moves from a hardcoded selector to LLM-based semantic matching.

Trade-offs

Advantages:

Structured data is text, so token cost is far lower than images, and it's faster
Locating accuracy doesn't degrade with screen resolution or element size — if the node is in the tree, semantic matching is precise regardless
Can reuse an existing automation framework's execution layer (Playwright, Appium) — the change is isolated to the "locating" step

Costs:

Strongly dependent on the app exposing complete, reliable structured information. Canvas-rendered content, embedded WebViews, and heavily custom-rendered mobile widgets often show up as an opaque black-box node in the accessibility tree with no semantic information at all — at that point this path simply fails and must fall back to a visual approach
Real-world DOM/accessibility trees can be enormous (modern SPAs routinely have thousands of nodes); stuffing the whole thing into an LLM's context is expensive, requiring pruning or downsampling — this is exactly the problem work like "DOM Downsampling for LLM-Based Web Agents" sets out to solve
Tree structures differ significantly across platforms (Web DOM, Android View tree, iOS UIKit tree), so one semantic-understanding pipeline doesn't transfer cleanly across platforms

Path 3: Computer Use — Raw Coordinate Clicking

How it works

This is the most "brute-force," yet also the most commercially deployed path of the past two years: Anthropic's Computer Use and OpenAI's Operator/CUA both fall into this category.

The key difference from Path 1: there's no separate "grounding model" step. A general-purpose multimodal model takes the screenshot directly as input and directly outputs the next action — "click at (842, 613)," "type text," "press Tab" — perception and decision-making happen in a single pass by the same model. There's no "planning model + dedicated grounding model" division of labor.

Screenshot → general multimodal model (perception + decision combined) → directly outputs: click(842, 613)

Trade-offs

Advantages:

Simplest architecture — no separate grounding model to maintain, no platform-specific structured data to parse
Because it has zero dependency on any interface the app exposes, it has, in principle, the highest ceiling on generality — "if you can screenshot it, you can operate it" — and it's the technical basis for turning an agent from "a web automation tool" into "a general assistant that operates a computer"

Costs:

Locating accuracy is the weakest of the three paths on public benchmarks — the numbers in the next section quantify this
Highest latency and cost: every single action requires a full multimodal inference pass, and tasks often need many consecutive steps to complete — this is the most expensive path of the three to run
No structured-data fallback: Path 2 can approach near-100% precision when the structure is reliable; Path 3 has no such ceiling guarantee — it's purely bounded by the model's visual grounding ability

Quantifying the gap with benchmarks

Saying "one path is better" without numbers is meaningless. A few public benchmarks let us compare the three paths on the same scale.

The ScreenSpot family: testing "locating" in isolation

ScreenSpot was the earliest GUI grounding benchmark: given a natural-language reference and a screenshot, the model must point to the target element's coordinates. Leading models already score above 90% accuracy on this benchmark.

ScreenSpot-Pro is the hardened version: all screenshots come from real professional desktop software (CAD tools, IDEs, data-analysis apps), at higher resolution with smaller, denser interface elements. The same leading models drop to 37.1% (one of the highest scores reported by a 2025-generation model — most models score far lower).

Benchmark          What it measures              Leading model accuracy
─────────────────────────────────────────────────────────────────────
ScreenSpot          Locating in simple UIs         90%+
ScreenSpot-Pro       Locating in professional UIs   ~37% (best case)

That cliff-edge drop reveals a key fact: "locating accuracy" isn't a fixed property of a model — it collapses sharply as interface density and element size push in the wrong direction. This is exactly why mobile automation (covered in five dedicated posts, 05-09) leans so heavily on dedicated grounding models — icons on a phone screen tend to be smaller and denser than buttons on desktop software.

OSWorld: testing end-to-end task success

ScreenSpot only tests "can you find the element." OSWorld tests whether a full task actually gets completed — executing a multi-step task in a real OS environment (e.g., "clean up this data in LibreOffice Calc and export it as a PDF"), which requires planning, locating, execution, and error recovery across the whole chain.

The OSWorld paper reports a human baseline of 72.36% task completion, while early baseline models (e.g. GPT-4V) achieved only 12.24% — a gap of nearly 60 percentage points. The paper explicitly attributes the bottleneck to GUI grounding and operational knowledge, not task comprehension itself. That number has since closed rapidly as dedicated agent architectures (e.g. the Agent S series) improved — by 2025-2026, leading approaches reach roughly 65%-70%+, closing in on the human baseline, but that gain came from agent-architecture-level engineering — planning, reflection, retry loops — not purely from a better underlying grounding model.

                     Human baseline    Early baseline model    2026 leading approaches
──────────────────────────────────────────────────────────────────────────────────
OSWorld task success   72.36%           12.24%                  ~65%-70%

The takeaway: grounding accuracy and task success rate are two different things. A perfectly accurate grounding model still won't get you a high end-to-end success rate if the agent can't re-plan after a failure or verify whether its action achieved the intended effect. That's why "planning + reflection mechanisms" will be just as important a topic as "which grounding model was chosen" when we get to mobile automation projects like ARTEMIS and Mobile-Agent-v3.

The three paths aren't mutually exclusive

In practice, mature open-source projects rarely bet purely on one path. The common pattern is layered fallback: try Path 2 first (DOM/accessibility tree — fast and precise), and fall back to Path 1 or Path 3 (visual grounding) whenever reliable structured data isn't available. AppAgent, covered in a later post (08), is a textbook example of this "parser tree + visual features, dual input" design — the two paths aren't a mutually exclusive technology choice, but two tiers of fallback ranked by reliability within a single system.

Concept vocabulary (reused across the series)

Term	Meaning	Referenced in
GUI Grounding	Mapping a natural-language reference to on-screen coordinates/elements	04-09
Visual grounding model	A model specifically trained for GUI grounding (UGround, GUI-Owl)	05, 07
Accessibility tree / DOM semantic understanding	Locating elements via structured views instead of pixels	04, 08, 10
Computer Use	A general multimodal model directly outputting coordinate actions, no separate grounding stage	09
ScreenSpot / ScreenSpot-Pro	GUI grounding benchmarks that measure single-step locating accuracy only	05-09
OSWorld / AndroidWorld	End-to-end benchmarks measuring full multi-step task success rate	05-09

Summary

There are three technical paths for GUI element locating: dedicated visual grounding models (broadly cross-platform, but slow and expensive), DOM/accessibility-tree structured understanding (fast and precise, but dependent on the platform exposing reliable structured data), and Computer Use raw coordinate clicking (simplest architecture, highest generality ceiling, but the lowest locating-precision ceiling)
The ScreenSpot → ScreenSpot-Pro cliff (90%+ dropping to ~37%) shows that locating accuracy collapses sharply with interface density — it isn't a fixed model capability
OSWorld's human-vs-model gap (72.36% vs. an early baseline of 12.24%) shows that end-to-end task success depends on more than grounding precision alone — planning and error-recovery mechanisms matter just as much
Mature systems typically use layered fallback rather than betting on a single path — this vocabulary will recur throughout every case study that follows

Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.

Find more useful knowledge and interesting products on my Homepage

One Open Source Project a Day (No. 189): NanoJev — A 0.6B Parallel Decision Model That Outputs Probability Distributions Instead of Generating Tokens

WonderLab — Tue, 22 Sep 2026 01:19:41 +0000

Introduction

"A decision doesn't need to write a sentence to explain itself — it just needs a probability distribution."

This is the 189th article in the "One Open Source Project a Day" series. Today's project is NanoJev.

Most LLM-driven decision-making approaches can't escape an awkward efficiency problem: the model autoregressively generates a chunk of text (something like "I think we should choose option B, because..."), and the final decision has to be parsed out of that text afterward. This requires token-by-token decoding — slow, costly, and the "confidence" the model expresses is usually just an adjective it happened to use, not a genuinely calibrated probability.

If a task's essence is really "given a state and a set of candidates, tell me the probability of each candidate," why make the model generate a wall of text first and parse it after? NanoJev's answer: skip the generation, output the distribution directly. It's a lightweight recreation of a parallel decision model system called "Jev." Built on a Qwen3-0.6B backbone with dedicated decision heads, it takes a state and a question as input and, in a single forward pass, outputs a complete probability distribution — zero output-token decoding.

954 Stars, MIT License, validated with maze navigation and Snake game benchmarks.

What You Will Learn

How NanoJev structures a decision as a "state + question + candidate set" triple
Three decision head types: dynamic Choice, Boolean judgment, and Ordered Score
Why "zero output-token decoding" fits decision-making better than autoregressive generation
The full training pipeline: build queries → organize data → train → evaluate → serve
How NanoJev compares against original Jev and an untuned Qwen3-0.6B on game benchmarks

Prerequisites

Basic understanding of how LLMs run inference (autoregressive generation vs. a single forward pass)
Familiarity with basic probability and classification/regression concepts
Optional: familiarity with reward modeling concepts in reinforcement learning

Project Background

What It Is

NanoJev's official positioning: "A 0.6B parallel decision model. States and questions in, complete probability distributions out — with zero output-token decoding."

Worth clarifying up front: this is not an image generation or editing tool. It's a structured decision model/reasoning framework — essentially it keeps an LLM's language understanding ability while re-engineering the output side from "generate text" to "output a calibrated probability distribution," aimed at use cases like game agent decision-making and probability calibration research.

Team and Background

Author: TianyuCodings
License: MIT License
Base model: Qwen3-0.6B
Related resources: model repo C-Tianyu/NanoJev, dataset C-Tianyu/NanoJev-Data (both on HuggingFace)

Project Stats

⭐ GitHub Stars: 954
🍴 Forks: 127
👀 Watchers: 8
📄 License: MIT
📊 Commits: 13

What It Does

The Problem It Solves

Traditional LLM decision-making approach:
  State + question → autoregressively generate text ("I think we should
  pick B, because...")
  ↓ parse the text to extract the final decision
  ↑ token-by-token decoding is slow and costly
  ↑ "confidence" is just an adjective in the text, never truly calibrated

NanoJev's approach:
  State + question + candidate set → a single forward pass →
  directly output a complete probability distribution
  ↑ zero output-token decoding, fast
  ↑ the distribution is trained and calibrated, usable directly for
    ranking / greedy selection / probability sampling
  ↑ a single forward pass can batch multiple independent states and questions
     (official example: "6 states · 18 questions · 44 candidate paths ·
     1 backbone forward")

Use Cases

Game agent decision-making
- Maze navigation (8×8 up to 50×50 in various sizes), real-time decisions for a Snake game agent
Local safety judgment
- Fast boolean judgments for scenarios like collision avoidance, combined with code-based planning
Probability calibration research
- Calibration experiments using CE (cross-entropy)/Brier loss and paired proper-reward learning
Scenarios needing batched decisions
- Evaluate multiple candidate paths in a single forward pass, suited to applications needing high-throughput decision-making

Quick Start

Simplest path: run the interactive demo locally (no model weights needed)

git clone https://github.com/TianyuCodings/NanoJev.git
cd NanoJev
python3 -m http.server 8080 --bind 127.0.0.1 --directory web

Visit http://127.0.0.1:8080/side-by-side.html for a three-panel comparison demo, or arcade.html (dark arena demo) and comparison.html (early benchmark viewer).

Full run: download model weights and start the service

# Install dependencies
python -m pip install -r requirements-toy.txt

# Download model checkpoint and dataset
python -c "
from huggingface_hub import snapshot_download
snapshot_download(
    repo_id='C-Tianyu/NanoJev', local_dir='checkpoints/NanoJev',
    allow_patterns=['best.safetensors', 'config.json', 'tokenizer/*', 'backbone_config/*'],
)
snapshot_download(
    repo_id='C-Tianyu/NanoJev-Data', repo_type='dataset', local_dir='data/NanoJev',
)
"

# Start the persistent service
python scripts/serve_decisions.py \
  --checkpoint-dir checkpoints/NanoJev \
  --web-root web --port 8765

Once started, the service loads the model once and accepts repeated batches of decision requests via POST /api/evaluate.

Core Features

1. Three Decision Head Types

Type	Range	Mechanism
Dynamic Choice	2–255 candidates	A shared scalar head + set attention, outputs a probability for each candidate
Boolean	A single proposition	Single-path sigmoid, outputs the probability the proposition is true
Ordered Score	2–10 levels	Computes a probability-weighted expectation over the ordered level distribution

2. Triple-Structured Decisions

Every decision is defined by a state, a question, and a candidate set. A shared decision head processes the input and returns the corresponding distribution output for the candidate set based on the decision type.

3. Five-Step Training Pipeline

Build queries
  ↓ generate states, questions, candidate descriptions, and target distributions
Organize data
  ↓ keep related maps, rules, and variants in the same data split
Train
  ↓ Qwen3-0.6B init + decision-head warmup + complete-question distribution losses
Evaluate
  ↓ measure probability quality + game controller records actual actions
Serve and visualize
  ↓ reuse a persistent model endpoint, replay full trajectories in the browser

4. Multiple Pretrained Checkpoint Variants

Variant Name	Use
`variants/local_atomic_seed17`	50×50 maze demo
`variants/games_gold_seed17`	Snake demo
`variants/games_api_seed17`	Full map comparison benchmark
`variants/events_ce_seed17` / `events_brier_seed17` / `events_paired_seed17`	Calibrated decision experiments (different loss functions)

A Deeper Look

The Performance Implications of "Zero Output-Token Decoding"

NanoJev's core architectural decision is to abandon autoregressive generation entirely, re-engineering the LLM's output side into structured decision heads. The official documentation gives a fairly intuitive number for the performance difference this brings:

"6 states · 18 questions · 44 candidate paths · 1 backbone forward"

Meaning:
  6 independent states + 18 questions + 44 candidate paths
  ↓ all folded into a single backbone forward pass
  ↓ instead of 44 separate autoregressive generations

The cost of autoregressive generation accumulates token by token — generating a chunk of explanatory text might require tens to hundreds of forward passes (one per token). NanoJev compresses the evaluation of 44 candidate paths into "1 backbone forward pass" — essentially swapping "expressing confidence in text" for "expressing probability directly through the model's internal numerical output," bypassing the entire autoregressive decoding cost chain.

The Design Trade-offs Behind Choice / Boolean / Score Heads

Why not use one generic head for every decision type? The three head types actually correspond to three fundamentally different statistical problems:

Choice (dynamic selection):
  Variable candidate count (2-255) → needs "set attention"
  ↑ the model must understand "these candidates form a mutually
    exclusive set," not independent judgments

Boolean:
  A single proposition, binary yes/no → a single-path sigmoid is enough
  ↑ no need to be aware of other candidates

Score (ordered evaluation):
  Levels have an order relationship (1 star to 5 stars) → can't be
  treated as a plain classification problem
  ↑ using a probability-weighted expectation lets a non-integer
    expectation like "3.5 stars" emerge naturally

This design reflects a simple but easily overlooked principle: when decisions have different statistical structures, the loss function and output head should differ too. Forcing "which option," "true or false," and "what rating" into the same softmax loses the structural information specific to each question type.

Real-World Training Results: Small Model vs. Fine-Tuning

NanoJev was benchmarked against original Jev and an untuned Qwen3-0.6B across three game benchmarks, and the results are fairly compelling:

Benchmark	NanoJev	Jev (original)	Untuned Qwen3-0.6B
50×50 maze (attempts/collisions)	244 attempts / 36 collisions, goal reached	2,738 attempts / 1,044 collisions, goal reached	—
12×12 Snake (food eaten/alive)	27 food, alive at horizon	30 food, alive at horizon	25 food, trapped
40-map navigation benchmark (test/OOD)	95% / 90%	100% / 95%	35% / 15%

A few points worth noting:

The untuned Qwen3-0.6B lags significantly behind (35%/15% vs. NanoJev's 95%/90%), showing that the purpose-trained decision heads deliver a real improvement — not just a rebranded model
NanoJev requires far fewer attempts than original Jev on the 50×50 maze (244 vs. 2,738), suggesting the lightweight recreation is actually more efficient on this specific task, likely due to more focused training data
NanoJev's overall accuracy is slightly lower than original Jev (e.g., 95% vs. 100% on the navigation benchmark), which is expected — recreating a system with fewer resources involves a reasonable trade-off in performance

Positioning: A Research Recreation, Not a Commercial Product

NanoJev explicitly positions itself as a "lightweight, reproducible" research alternative. That positioning is clear: it doesn't aim to outperform original Jev, but rather to let more researchers validate and extend the "parallel decision model" idea at a smaller training cost with a simpler deployment path. Roadmap items like expanding data scale and RLCD (contrastive distillation reinforcement learning) expansion all point toward "continuing to explore the boundaries of this architecture" rather than "polishing it into a product."

Project Links and Resources

Official Resources

🌟 GitHub: https://github.com/TianyuCodings/NanoJev
🤗 Model: C-Tianyu/NanoJev (HuggingFace)
🤗 Dataset: C-Tianyu/NanoJev-Data (HuggingFace)
📄 License: MIT License
🐛 Issues: GitHub Issues

Related Resources

Qwen3 — The 0.6B backbone network NanoJev is built on
Brier Score — One of the calibration scoring methods used in NanoJev's training

Summary

Key Takeaways

Zero output-token decoding: abandons autoregressive generation, using decision heads to output probability distributions directly, sidestepping the performance overhead of token-by-token decoding
Triple-structured decisions: state + question + candidate set clearly bounds every decision query
Three decision heads, each fit for purpose: Choice/Boolean/Score correspond to distinct statistical problem structures rather than being forced into a single softmax
Real benchmark validation: significant improvement over untuned Qwen3-0.6B on maze navigation and Snake game benchmarks, with a reasonable performance trade-off versus original Jev
Research-oriented positioning: explicitly a lightweight recreation rather than a commercial product, serving further exploration of the "parallel decision model" direction

Who This Is For

Game AI / agent decision-making researchers: need a lightweight, reproducible baseline for parallel decision models
Researchers focused on LLM efficiency: interested in architectural changes that skip autoregressive generation in favor of directly outputting structured distributions
Probability calibration researchers: want to study the practical effect of CE/Brier loss and paired reward learning in decision-making scenarios
Teaching/reproduction-oriented learning contexts: want to understand a "decision head + backbone" architecture without training a large-scale system from scratch

One-Line Verdict

NanoJev raises a question worth sitting with: if a task's essence is producing a probability distribution, why detour through generating a paragraph of text at all?

Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.

Find more useful knowledge and interesting products on my Homepage

LLM-Driven Automated Testing Series (01): Why Testing Is LLM's New Battlefield

WonderLab — Mon, 21 Sep 2026 03:06:25 +0000

A scenario every test engineer has lived through

A product manager moves a button by a few pixels — barely noticeable to the eye. The next day, 40 tests fail red in CI simultaneously.

After half an hour of digging, the culprit isn't broken business logic. It's a hardcoded CSS selector, #submit-btn-v2, that can no longer find the element — the DOM structure changed and the id changed with it. The business code is entirely correct; the test script died first.

This is one of the most common, most labor-draining, and least technically interesting failure modes in automated testing. It's also the starting point of this series: can LLMs turn this problem — and a few other long-standing ones in the testing world — from "brute-force human maintenance" into something that's actually engineered?

Three long-standing problems in traditional automated testing

Before discussing what LLMs can do, let's name the problems clearly. None of these are new — the industry has debated them for over a decade without a cheap, general solution.

Problem 1: Fragile Locators

Automated test scripts have to "find" an element on the page before they can act on it — click a button, fill a form, read some text. The common ways to locate an element are CSS selectors, XPath, element IDs, or raw coordinates.

All of them share the same fatal assumption: the UI structure won't change after the test script is written. In reality, UI structure changes constantly — redesigns, A/B tests, a front-end framework rewrite that restructures the component tree. The more specific a script's dependency on UI structure, the more fragile it becomes.

Selector specificity     Robustness     Maintenance cost
────────────────────────────────────────────────────────
#submit-btn-v2           Very low       Breaks on any UI tweak
.btn.primary              Low            Breaks on class refactor
//div[3]/button[1]        Very low       Breaks on DOM hierarchy change
Coordinates (320, 480)    Very low       Breaks on resolution/layout change

The industry has a name for this: "selector hell." The larger a test team grows, the higher the proportion of engineering time spent maintaining locators — not writing new tests, but fixing the locating logic of old ones.

Problem 2: The Oracle Problem (how do you know if the result is right?)

"Oracle" in testing refers to the standard used to judge whether an output is correct. A traditional unit test's oracle is explicit: assert result == 42. But in many testing scenarios, the oracle itself is fuzzy:

A screenshot differs from the last version by a few pixels — is that a bug,
  or just font-rendering jitter?
The returned JSON has an extra field — is that an API change,
  or dirty test-environment data?
The user flow completed, but the page "just doesn't look right" —
  hard to say exactly what's wrong

The traditional fix is to make the oracle explicit: pixel-level diffing (what threshold counts as a real change), schema validation (which fields must exist), assertion chains (explicitly checking state at each step). The shared problem with these approaches: the stricter the oracle, the more false positives (flagging meaningless changes as failures); the looser the oracle, the more false negatives (real bugs slip through).

Problem 3: Maintenance Cost (test scripts age faster than the business code)

This is a consequence of the first two problems, but it deserves its own callout because it's the direct reason teams eventually abandon automated testing.

A real pattern: teams that iterate faster on features tend to end up with lower actual test coverage over time — not because nobody wants to write tests, but because tests go stale the moment they're written, nobody has time to fix them, and eventually they get skipped or ignored outright. The entropy of the test codebase grows faster than the team's capacity to maintain it.

These three problems reinforce each other: fragile locators → tests fail red constantly → maintenance cost rises → the team has no time to maintain them → coverage gets cut → the oracle problem becomes harder to catch (because coverage itself is shrinking).

Why now: testing happens to be a "clear right-or-wrong" scenario

For the past two years, LLM Agent adoption has stalled in the same place across many domains: how do you know the Agent got it right? Writing code, making decisions, generating content — the "correctness" of these outputs is usually continuous and subjective, requiring a whole separate evaluation layer to judge (which is exactly what the eval-series on this blog is about).

Testing is the exception. A testing task carries its own explicit success signal:

Unit test passes / fails             —— boolean, no middle ground
UI element found / not found         —— boolean
Assertion holds / doesn't hold       —— boolean
Coverage increased by X percentage points —— a quantifiable number

This means the quality of an LLM's output in a testing context can be verified using the test's own execution result, without needing an extra meta-layer of "is this LLM output reasonable." This is a structural advantage testing has over other LLM use cases — and it's why the past two years have produced a wave of open-source projects that actually work in production, not just proof-of-concept demos.

That said, "clear right-or-wrong" only applies to the test's final execution result. Whether a test script should be generated, whether a locator should self-heal, whether a failure is a real bug or an environment flake — these judgments remain fuzzy. That's exactly where LLMs have room to intervene, not in the final "did the test pass" verdict itself.

Three places where LLMs can plug in

Coming back to the three problems from the opening: here's where LLMs can plug in for each.

Fragile locators   → Semantic locating ("find the submit button" instead of "find #submit-btn-v2")
                    → Self-healing locators (when the UI changes, re-find the element
                      semantically instead of the script just dying)

Oracle problem      → Semantic judgment ("is this visual change an actual bug to the user")
                    → instead of pure pixel/structural comparison

Maintenance cost    → Coverage-driven automatic test generation
                    → Automatic failure classification (real bug / environment issue / test issue)

These three threads map to what the rest of the series will cover: unit test generation (03), Web/mobile UI automation (04-09), self-healing locators (10), API testing (11), visual regression (12), and failure classification (13).

The next post (02) will first lay out the shared technical foundation behind all of these approaches — visual grounding, DOM-based semantic understanding, and pure coordinate-clicking Computer Use. Each has its own principles and trade-offs, and this vocabulary will be reused in every case study that follows.

What this series won't do

Setting expectations upfront:

This is not a trend-report on "AI testing." Each post picks one technical point paired with a verifiable open-source project — no "reportedly," no "may become possible in the future" claims that can't be checked
This isn't about evaluating LLM output quality. That's the eval-series's territory. This series is about "using LLMs to do testing," not "evaluating LLM outputs"
Limitations won't be glossed over. Every technical approach comes with a discussion of where it breaks down and what it costs

Summary

Fragile locators, the oracle problem, and maintenance cost are the three long-standing problems in automated testing, and they reinforce each other
Testing tasks carry an explicit success signal (pass/fail) — this is testing's structural advantage over other LLM use cases
LLMs plug in at semantic locating/self-healing, semantic oracle judgment, and automatic generation/classification — not at replacing the final "did the test pass" verdict itself

Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.

Find more useful knowledge and interesting products on my Homepage

One Open Source Project a Day (No. 188): ARTEMIS — Google's Open-Source Mobile AI Automation Framework, Let AI Assistants Operate Phones Like a Human

WonderLab — Mon, 21 Sep 2026 02:01:12 +0000

Introduction

"A test script breaks the moment a button moves — an AI that actually understands the interface doesn't."

This is the 188th article in the "One Open Source Project a Day" series. Today's project is ARTEMIS.

Mobile UI automation testing has long lived with a fundamental tension: traditional scripts based on element IDs or coordinates are fast, but extremely brittle — a small layout tweak breaks test cases en masse, and maintenance cost grows linearly with how fast the app iterates. Vision-model-driven approaches are more robust, but slow and expensive, making them a poor fit for large-scale regression testing.

ARTEMIS is Google's attempt to find a balance between these two extremes. It turns natural-language instructions — like "open Settings, find the Battery option, tell me the current level" — into reliable Android automation, while offering two execution modes: one optimized for speed on routine deterministic tasks, another optimized for accuracy and verifiability on complex, long-running ones. More notably, it plugs directly into Claude Code, Antigravity, Codex, and other AI IDEs via MCP, giving AI coding assistants the ability to operate real phones.

8.1k Stars, Apache-2.0, built in Python, achieving 99%+ task completion on the AndroidWorld benchmark.

What You Will Learn

How ARTEMIS turns natural-language instructions into Android automation
The architectural differences between the Flash and Pro execution modes, and when to use each
Multi-modal target localization: combining the accessibility hierarchy, OCR, and vision models
How it integrates with Claude Code and other AI IDEs via MCP
How the Python SDK plugs into existing automated testing frameworks

Prerequisites

Basic understanding of Android automation testing concepts (ADB, Accessibility services)
Familiarity with Python async programming (asyncio)
Optional: basic understanding of MCP (Model Context Protocol)

Project Background

What It Is

ARTEMIS's official positioning: "ARTEMIS turns natural-language instructions into reliable Android automation" — letting AI assistants and test suites operate real phones like a human would.

One clarification worth making up front: this is not a security testing or fuzzing tool. It's an intelligent agent system focused on mobile UI automation testing and everyday task execution, squarely in the AI-driven mobile testing/automation framework space.

Team and Background

Organization: Google
License: Apache License 2.0
Primary language: Python (requires Python 3.12+)
Code provenance: the project notes it includes source code developed by Minitap, Inc., building on the open-source mobile-use project

Project Stats

⭐ GitHub Stars: 8,100+
🍴 Forks: 770+
👀 Watchers: 71
📄 License: Apache-2.0
🏆 Benchmark: 99%+ task completion on AndroidWorld (covering 20+ apps, 100+ multi-step tasks)

What It Does

The Problem It Solves

Traditional element-ID/coordinate-based automation scripts:
  UI layout shifts slightly → element lookup fails → tests break en masse
  ↑ Fast, but extremely brittle, high maintenance cost

Pure vision-model-driven approaches:
  Every step relies on a vision model to read the screen → accurate but slow
  ↑ Good robustness, but slow and expensive, hard to scale

ARTEMIS's approach:
  Natural-language instruction → multi-modal localization (element index first,
  vision/coordinates as fallback)
  ↓ Flash mode: routine deterministic tasks, 3-5 sec/step, fast reactive loop
  ↓ Pro mode: complex long-running tasks, 15-40 sec/step, plan-execute-verify workflow
  ↑ Choose the right mode for task complexity, balancing speed and reliability

Use Cases

Mobile app end-to-end test automation
- Describe test steps in natural language instead of maintaining brittle element-locator scripts
Bug reproduction and diagnosis
- Combined with Logcat log and screenshot capture, quickly reproduce and pinpoint issues
Exploratory stability testing
- Pro mode supports long-running, continuously monitored exploratory testing
CI/CD-integrated automated testing
- Integrate via the Python SDK into pytest and similar frameworks, folded into continuous integration pipelines
AI IDE-driven real device operation
- Let Claude Code, Antigravity, and other AI coding assistants directly operate a phone to verify functionality

Quick Start

Clone and one-command start:

git clone https://github.com/google/artemis.git && cd artemis

# macOS/Linux one-command start (auto-detects and installs ADB, scrcpy, FFmpeg, etc.)
./start.sh

Run a task directly via CLI:

uv run artemis run "Open Settings, find Battery and tell me current level" --profile flash

Install MCP integration for an IDE:

# Install for Antigravity
uv run artemis mcp --install antigravity

# Install for all supported IDEs (including Codex)
uv run artemis mcp --install all

On first task execution, ARTEMIS installs an "Artemis Accessibility Helper" service on the device — it only reads the screen layout locally and uploads no data, and can be pre-installed, checked, or uninstalled via command.

Core Features

1. Four Ways to Use It

Method	Best For
Web visual test console (`artemis ui`)	Interactive debugging and results review
MCP server	Connecting an AI IDE to drive a real device
Developer CLI (`artemis run`)	Automated testing or benchmark scripts
Python SDK	Integrating into existing test frameworks like pytest / CI pipelines

2. Dual Execution Modes: Flash and Pro

Dimension	Flash Mode	Pro Mode
Response speed	~3-5 sec/step	~15-40 sec/step
Architecture	Single-model observe-think-act reactive loop	Multi-agent graph (Planner+Operator+Checker)
Task planning	No task plan	Planner maintains a Markdown task plan with milestones
Safety mechanism	No pre-execution safety net	Every action XML-verified first, pixel fallback
Verification	No checkpoint verification or final report	Checker verifies plan checkpoints, supports a final review before exit
Toolset	No ADB shell	Full toolset: Explorer, notes, history lookback, video analysis
Best for	Routine deterministic UI tasks	100+ step long-running workflows, continuous monitoring

3. Python SDK Integration Example

uv add "artemis-client @ git+https://github.com/google/artemis.git#subdirectory=packages/artemis-client"

import asyncio
from artemis_client import ArtemisClient


async def main():
    client = ArtemisClient(
        "http://artemis-host:8000",
        device_serial="emulator-5554",  # optional: target specific device serial
        default_profile="flash",  # "flash" (fast reactive) or "pro" (deep reasoning)
    )

    result = await client.run(
        "Open System Settings, go to 'Battery', verify battery percentage is displayed, and check for any crash dialogs.",
    )

    assert result.succeeded, f"Test failed: {result.error or result.status}"
    print(f"✅ Test Passed! Device: {result.device_serial} | Trace ID: {result.trace_id}")


if __name__ == "__main__":
    asyncio.run(main())

The SDK claims "zero runtime dependencies" — ADB, the agent, models, and image processing all stay on the device host. It supports strongly typed Pydantic structured outputs and assertions, and plugs directly into frameworks like pytest.

4. MCP Integration Configuration

Using Claude Desktop as an example (claude_desktop_config.json):

{
  "mcpServers": {
    "artemis": {
      "command": "/path/to/artemis/.venv/bin/python",
      "args": ["-m", "mcp_server"],
      "cwd": "/path/to/artemis"
    }
  }
}

The docs also recommend mounting a behavior rules file (mcp_server/rules.md) for AI agents, covering proactive exploration before coding, Flash/Pro routing strategy, and latency compensation logic.

A Deeper Look

Flash vs. Pro: A Pragmatic Speed/Reliability Trade-off

ARTEMIS doesn't use a single mode for every task — it explicitly splits into two architecturally distinct execution paths:

Flash mode (reactive):
  Observe screen → think about next step → execute action → loop
  ↑ Single-model loop, no planning, no safety net, no final report
  ↑ Loop count is unbounded by default (history is compressed, not truncated)
  ↑ Good for short, deterministic tasks like "open Settings and flip a switch"

Pro mode (plan-and-verify):
  Planner drafts a Markdown task plan (with milestones and verification items)
      ↓
  Operator runs a "Safety Net" check before each action
      ↓ (checks against the live UI tree first, then pixel fallback)
  Hits a blocker → opens an "execution incident," Operator handles recovery autonomously
      ↓
  Checker (read-only) verifies plan checkpoints, supports a final review before exit

The core insight behind this design: mobile automation task complexity is bimodal. A large share of tasks are deterministic short tasks — "open a page and confirm a state" — where a simple reactive loop is fast and sufficient. A minority are complex workflows that need multi-step planning, fault tolerance, and long-running monitoring, where only a multi-agent plan-execute-verify structure can guarantee reliability. Using one architecture for both extremes wastes something either way.

Fault-Tolerant Design in Multi-Modal Target Localization

ARTEMIS's element localization mechanism reflects a "prefer precision, degrade gracefully" fault-tolerance philosophy:

Localization priority:
  1. Accessibility Hierarchy — most precise, the go-to for standard Android UI
  2. OCR text recognition — handles cases where the hierarchy tree lacks information
  3. Vision model recognition — fallback, handles custom Canvas/Compose/Flutter UIs

Standard Android widgets are usually exposed accurately by accessibility services, making that the fastest and most reliable localization method. But more and more apps render their UI with Compose, Flutter, or custom Canvas drawing, which is unfriendly to accessibility services — that's when OCR and vision models step in. This layered fallback strategy avoids the waste of "use the most expensive method for everything" while also avoiding the fragility of "just fail outright when standard widgets aren't recognized."

Shared History Compression: Memory Management for Long-Running Tasks

Flash and Pro share the same history compression mechanism, and this detail is worth noting:

Problem: long-running tasks (especially Pro's 100+ step workflows) accumulate
a large volume of screenshots and action logs
  ↑ Stuffing them directly into context blows past the window limit
  ↑ Simple truncation loses important early information

ARTEMIS's approach:
  Old screenshots → replaced with visual summaries (keep semantics, drop pixel detail)
  Completed steps → compressed into retrievable history chunks
  ↑ The agent can retrieve past context when it needs to look back,
    instead of always carrying the full history

This design lets Flash mode run stably long-term even with no loop cap, and is also the memory-management foundation that makes Pro's 100+ step workflows possible.

How It Compares to Similar Mobile Automation Approaches

Dimension	Appium (traditional scripts)	Pure vision-model-driven	ARTEMIS
Localization method	Fixed element ID/XPath	Pure visual recognition	Element index first + OCR + vision fallback
UI-change tolerance	❌ Brittle	✅ Fairly robust	✅ Fairly robust
Execution speed	Fast	Slow	Flash is fast / Pro is moderate
Natural-language driven	❌	Partial support	✅ Native support
AI IDE integration (MCP)	❌	Uncommon	✅ Native support across multiple IDEs
Long-running task plan/verify	Must build yourself	Uncommon	✅ Native in Pro mode
Open source	✅	Varies by project	✅ Apache-2.0

ARTEMIS's differentiation is combining "natural-language driven + multi-modal fault-tolerant localization + native AI IDE integration" into one tool, using its dual-mode design to avoid the dilemma of "either too slow with pure vision models, or too brittle with fixed scripts."

Project Links and Resources

Official Resources

🌟 GitHub: https://github.com/google/artemis
📄 License: Apache License 2.0
🐛 Issues: GitHub Issues

Related Resources

AndroidWorld — Google Research's mobile agent benchmark project, and ARTEMIS's evaluation benchmark
Model Context Protocol — The standard protocol underlying ARTEMIS's integration with Claude Code and other AI IDEs
mobile-use — The open-source project (from Minitap, Inc.) that ARTEMIS is built on

Summary

Key Takeaways

Natural-language-driven mobile automation: replaces brittle element-locator scripts with instructions, lowering test maintenance cost
Dual Flash/Pro mode design: choose a fast reactive loop or a plan-execute-verify workflow based on task complexity, balancing speed and reliability
Multi-modal fault-tolerant localization: accessibility hierarchy first, OCR and vision models as graceful fallbacks, adapting from standard widgets to custom Canvas UIs
Native MCP integration: gives Claude Code, Antigravity, Codex, and other AI IDEs the ability to directly operate real Android devices
99%+ on AndroidWorld: reliability validated on a standard mobile agent benchmark

Who This Is For

Mobile app testing teams: want to escape brittle element-locator scripts and reduce maintenance cost from UI changes
AI application developers: want AI coding assistants to be able to operate real Android devices for feature verification or bug reproduction
CI/CD pipeline maintainers: need to seamlessly fold mobile automation testing into existing test frameworks and continuous integration
Exploratory/stability testing teams: need long-running, continuously monitored mobile testing capability

One-Line Verdict

ARTEMIS doesn't pick a side between "fast but brittle" and "robust but slow" — it splits the tension apart directly with its Flash/Pro dual-mode design, which may be a pragmatic middle answer as mobile automation moves toward AI-driven approaches.

Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.

Find more useful knowledge and interesting products on my Homepage

One Open Source Project a Day (No. 187): TeamAI-CLI — Tencent's Team-Level AI Agent Middleware, Turning Individual AI Skills Into Shared Team Assets

WonderLab — Sun, 20 Sep 2026 02:28:57 +0000

Introduction

"An AI agent's capabilities shouldn't be trapped in one person's chat history — they should become a shared asset for the whole team."

This is the 187th article in the "One Open Source Project a Day" series. Today's project is TeamAI-CLI.

When everyone on a team is using Claude Code, Cursor, or Codex to write code, a hidden problem starts building up: everyone is independently "training" their own AI. One engineer spends half a day figuring out the right prompting approach to get Claude Code to correctly understand the project's architecture; another hits the exact same wall with no idea it's already been solved, and has to work through it from scratch. Skills, Rules, and CLAUDE.md configs scattered across the team never converge into anything shared or synchronized.

That's exactly the problem TeamAI-CLI addresses. It's not another AI coding assistant — the project's own documentation lists Claude Code, Codex, and Cursor as tools it's compatible with, not competitors. It's a meta-management layer that sits on top of the AI tools your team already uses, responsible for syncing skills, rules, and knowledge bases across members, and for turning individual experience gained while using AI into a shared team asset.

4.8k Stars, MIT License, built in TypeScript.

What You Will Learn

TeamAI-CLI's three-layer product architecture: Team Execution / Team Context / Team Improvement
The push → MR → review-and-merge → SessionStart-hook-sync distribution flow
The friction-signal-driven mechanism for automatically capturing experience
How the codebase knowledge graph is built (tree-sitter AST + regex heuristics)
Its relationship to Claude Code, Codex, and similar tools: not a replacement, but a "team layer"

Prerequisites

Some experience using AI coding tools like Claude Code, Cursor, or Codex
Familiarity with Git workflows (branches, Merge Requests)
Optional: basic understanding of MCP (Model Context Protocol)

Project Background

What It Is

TeamAI's official positioning is "the shared foundation for how your team works, learns, and improves with AI." The core idea, in one sentence: turn individual AI capabilities into shared team capabilities — across agents, machines, and team members.

Team and Background

Organization: Tencent
License: MIT License
Primary language: TypeScript
Distribution: global npm install (teamai-cli)

Project Stats

⭐ GitHub Stars: 4,800+
🍴 Forks: 340+
👀 Watchers: 17
📄 License: MIT
🔧 Open Issues: 33, Open PRs: 14 (actively maintained)

What It Does

The Problem It Solves

A team without TeamAI:
  One engineer built a perfect set of project rules in Claude Code  ← lives only on their machine
  Another spent three days figuring out how to get the AI to understand the architecture  ← that lesson never spreads
  Someone's CLAUDE.md and team conventions went stale long ago  ← nobody keeps them in sync
  ↑ Everyone's AI capability stays siloed — the team's collective ability never compounds

TeamAI's approach:
  A unified team resource repository (Skills/Rules/Docs/Agents/Hooks/MCP)
  ↓ teamai push (share) → MR review → merge
  ↓ SessionStart hook auto-triggers teamai pull (sync)
  Every team member's local AI tools automatically get the latest team standards
  ↑ Individual experience → team asset — AI capability can be version-controlled,
    reviewed, and distributed just like code

Use Cases

A tech lead standardizing AI tool conventions across the team
- Write project-specific coding standards and architecture constraints as Skills/Rules once, sync them to everyone
New members onboarding quickly into an existing AI collaboration setup
- After joining the team repo, local AI tools automatically pull the latest skills and context — no starting from scratch
Team members sharing personal experience
- Use teamai contribute to share a hard-won debugging insight with the whole team in one command
Codebase knowledge accumulation for AI retrieval
- The knowledge graph feature lets AI automatically retrieve relevant code context and historical decisions before tackling a coding task
Visualizing team-wide AI usage
- Use dashboard and digest commands to understand team-wide AI usage trends and activity

Quick Start

Option 1: Conversational install (let the AI set it up)

Paste this directly to Claude Code or any agent that supports Skills:

Install the teamai skill: https://github.com/Tencent/teamai-cli/tree/main/skills/teamai ,
load the teamai skill, then set up TeamAI for my team from scratch.

Option 2: Manual CLI install

# Admin: create the team's shared repository
npm install -g teamai-cli
teamai init https://github.com/yourorg/yourrepo

# Team member: join an existing team
cd /path/to/my-project
teamai init https://github.com/yourorg/yourrepo

# Or install at the user scope (applies across all projects)
teamai init https://github.com/yourorg/yourrepo --scope user

Core Features

1. A Three-Layer Product Architecture

Layer	Positioning	Core Capabilities
Team Execution	"Make every agent work the team's way"	init/pull/push sync for Skills/Rules/Agents/Hooks/MCP/Env
Team Context (beta)	"Make every agent understand the team"	Knowledge recall, learnings, codebase knowledge graph, team wiki
Team Improvement (beta)	"Make every execution improve the team"	Friction-signal-driven experience sharing, session logs, weekly digest, usage dashboard

2. A Git-Native Distribution Flow

teamai push → creates a branch + MR → reviewer approves and merges
                                              ↓
        SessionStart hook auto-triggers teamai pull → synced to local AI tools

Supports GitHub, GitLab, GitCode, CNB, TGit, and private Git servers — essentially folding AI collaboration standards into the normal code review process.

3. Key Commands

Command	Function
`teamai init`	Initialize: OAuth login, link repo, register member, inject hooks
`teamai pull` / `push`	Pull/push team resources
`teamai status`	Show the diff between local state and the team repo
`teamai contribute`	Share session experience to the team repo
`teamai recall <query>`	Search the team knowledge base (BM25 + graph-augmented)
`teamai codebase --extract/--deep-enrich/--reconcile`	Codebase knowledge graph operations
`teamai digest` / `dashboard`	Team usage stats and visualization
`teamai doctor`	Diagnose configuration issues

4. A Broad Agent Compatibility Matrix

TeamAI-CLI explicitly claims compatibility with over a dozen AI coding tools, including Claude Code, Codex, Cursor, GitHub Copilot CLI, CodeBuddy, OpenCode, and Kiro. The documentation includes a detailed compatibility table showing which capabilities (skills/rules/docs/env/hooks/mcp) each tool supports.

A Deeper Look

Friction-Signal-Driven Experience Capture

The most interesting design in TeamAI-CLI's Team Improvement layer is its trigger logic: instead of requiring users to proactively log their experience, it monitors friction signals in a session to detect "something worth recording just happened."

Examples of friction signals:
  "The user interrupted the AI twice"
  "The AI retried the same failing tool call 8 times in a row"
  ↓
  The system identifies unusual friction
  ↓
  It prompts the user: "This debugging session looks like it has something
  worth sharing — want to contribute it to the team?"

This is smarter than "asking employees to write weekly summaries of lessons learned" — most valuable debugging insights happen in the moment while solving a problem, and details fade fast in hindsight. Using interaction friction as the trigger captures experience while it's still fresh.

The Codebase Knowledge Graph: A Dual-Track Extraction Strategy

The codebase knowledge graph in the Team Context layer uses a dual-track strategy:

AST Track (precise, but costly)
  Uses a tree-sitter WASM parser to parse code
  ↓ precisely extracts functions, classes, and dependency relationships
  ↓ suited to deep parsing of core modules

Heuristic Track (fast, but coarse-grained)
  Scans code with regex heuristics
  ↓ quickly covers a broad range of files
  ↓ suited to fast indexing of peripheral code

Both tracks' results → written to the teamwiki/ directory → power the recall
command's BM25-based retrieval

This dual-track design reflects a practical trade-off: running tree-sitter-level precise AST parsing across an entire large codebase is expensive, while pure regex matching loses semantic relationships. Combining both — precise parsing for core modules, fast coverage for the periphery — is a pragmatic middle ground.

How It Relates to Claude Code and Codex

This is the point most likely to cause confusion about TeamAI-CLI, so it's worth clarifying directly:

Wrong framing: TeamAI-CLI competes with Claude Code, as another AI coding CLI
Correct framing: TeamAI-CLI sits on top of Claude Code/Codex/Cursor,
                  acting as a team-level distribution and knowledge management layer

Analogy:
  Claude Code / Codex / Cursor  ≈ the IDE on each person's machine
  TeamAI-CLI                    ≈ the team's Git repo + code review process
                                   (but for Skills/Rules/knowledge, not code)

It works by injecting hooks (like a SessionStart hook) and shared config files (CLAUDE.md, Skills directories, etc.) into already-installed AI tools. It doesn't provide reasoning capability itself — it orchestrates "who should use which rules, and when they get synced."

How It Compares to Similar Team Collaboration Tools

Dimension	Manually Maintained CLAUDE.md	Cursor Rules (local)	TeamAI-CLI
Cross-agent compatibility	Depends on each tool's own implementation	Cursor only	✅ A dozen+ tools
Team sync mechanism	Manual copy-paste / word of mouth	❌ None	✅ Git push/pull + MR review
Automatic experience capture	❌	❌	✅ Friction-signal-driven
Codebase knowledge graph	❌	❌	✅ AST + heuristic dual-track
Usage visualization	❌	❌	✅ dashboard/digest
Version control and review	Manual	Local files, no review	✅ Standard Git flow

TeamAI-CLI's differentiation is treating "AI collaboration conventions" as a first-class citizen of standard software engineering process (version control, code review) — instead of leaving everyone to maintain their own local config files.

Project Links and Resources

Official Resources

🌟 GitHub: https://github.com/Tencent/teamai-cli
📦 npm package: teamai-cli
📄 License: MIT License
🐛 Issues: GitHub Issues

Related Resources

tree-sitter — The parser framework behind TeamAI-CLI's codebase knowledge graph AST Track
Model Context Protocol — The resource distribution standard involved in TeamAI-CLI's compatibility matrix
Claude Code — One of the main AI coding agents TeamAI-CLI integrates with

Summary

Key Takeaways

Not an AI coding assistant, but a team-level middleware layer: sits on top of Claude Code/Codex/Cursor, focused on syncing standards rather than generating code
A three-layer architecture that builds progressively: Team Execution handles distribution, Team Context handles understanding, Team Improvement handles evolution
A Git-native distribution flow: push → MR review → merge → hook-triggered sync — AI collaboration conventions follow standard software engineering process
Friction-signal-driven experience capture: doesn't rely on manual summarization; interaction anomalies automatically trigger experience capture
Compatible with a dozen+ agents: not locked to a single AI tool ecosystem, covering mainstream choices like Claude Code, Codex, and Cursor

Who This Is For

Engineering team leads: want to standardize AI collaboration conventions across the team instead of letting everyone figure it out independently
Mid-to-large engineering teams: members use different AI tools and need a neutral layer to keep standards in sync
Teams that value knowledge management: want to systematically capture hard-won lessons and codebase knowledge instead of losing them to staff turnover
Teams already using Claude Code/Cursor and similar tools: don't want to switch tools, just want the ones they already use to be smarter and more consistent

One-Line Verdict

TeamAI-CLI isn't trying to solve "how AI writes code" — it's trying to solve "how one person's AI experience becomes everyone's experience," which may be a more interesting direction than simply stacking on stronger models.

Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.

Find more useful knowledge and interesting products on my Homepage