DEV Community: Qweezyy

Three token optimizations that made our agent more expensive

Qweezyy — Thu, 08 Oct 2026 09:58:58 +0000

TL;DR. Last time we cut a browser agent's tokens by 58% and really saved money. This time we looked for more places to trim and checked each idea on tasks with hidden tests. The three most obvious savings made the agent more expensive: a trimmed system prompt (+82% per task), aggressive clearing of old context (almost twice the price), and loading tools one at a time, which reset the provider's cache. They share a root: in an agent the price is set not by the size of one request but by the number of steps and the share the provider serves from its cache. Plus two small examples where the saving really was there.

About this post. The project is Altair, an open-source (Apache-2.0) AI agent for your PC and phone. I'm the author and build it largely with Claude Code. This post, the experiments and the charts were prepared by that same AI assistant; I reviewed it and stand behind it. Numbers are from our runs, code is from the repo.

First — how to price an agent

An agent is a loop: the model answers, calls a tool, gets the result, and the whole history goes into the next request again. A 10-step task is 10 requests, each longer than the last.

Providers soften this with a cache: if the start of a request matches the previous one, that part costs several times less. For the model we measured on (glm-5.3-flash), input is $0.15 per million tokens and cached input $0.03 — five times less. Our providers cache automatically, but only on a matching start of the request. Change something in the middle and everything after it is billed in full.

So two things set the price of a task:

how many steps the agent took;
what share of each request came from the cache.

The size of one request comes third. All three failed optimizations saved exactly that.

How we measured

Six hard tasks with hidden tests (the agent never sees them, the check runs after "done"): an LRU cache with a TTL, a config parser fixed from user bug reports, a rename across several files, latency percentiles from logs, parsing "1.5M / 200K", business days with holidays. The real Altair run through its CLI, 18 runs per variant. Between agent and provider, a proxy that logs every request and can change it on the fly: drop a prompt section, cut tool descriptions. Cost as billed by the provider.

Optimization 1: trim the system prompt

The fixed part of every step for us is about 9,300 tokens: the system prompt (5,200) and the schemas of 20 core tools (4,100). An obvious target. We tried two variants:

a trimmed prompt — only what the model can't guess: environment, folder, rules for secrets and approvals, honesty. Out went the advice on approach, planning, editing code, verifying, communicating;
short tool descriptions — each description cut to its first sentence, parameter descriptions to 80 characters.

Variant	Solved	Steps	Cost per task
Full prompt	15 of 18	7.9	3.56 m$
Trimmed prompt	15 of 18	12.4	6.49 m$ (+82%)
Short tool descriptions	15 of 18	10.4	5.96 m$ (+67%)

Each request got shorter and each task got more expensive. Without advice on how to work, the agent took 57% more steps: read too much, re-ran tests, rewrote what already worked. Every extra step resends the whole history.

Short tool descriptions did something worse. One task — fixing the parser from seven bug reports — the agent failed in all three attempts, though it solved it with full descriptions. Tool descriptions aren't ballast; they're the instructions for using the tools right.

Two more "cuts" gave nothing either: a hint to "do several independent things in one step" didn't reduce steps (7.9 vs 7.9), and dropping the skills list from the prompt was within noise.

Optimization 2: clear old context aggressively

Old tool outputs — files read, command results — stop being useful. Like many agents we have "observation masking": when the history passes half the budget, old large outputs are replaced with a short note. It's a proven technique: JetBrains showed on SWE-bench that it makes an agent 52% cheaper at the same solve rate.

The logical next thought is to clear earlier. We cut the context budget from 120K to 40K tokens and gave the agent a task where clearing would surely kick in: read eight files in full, then answer six questions about them.

Budget	Solved	Steps	From cache	Cost per task	Time
120K (normal)	3 of 3 (6/6 answers)	20.7	90%	30.6 m$	2.5 min
40K (tight)	3 of 3 (6/6 answers)	28.3	57%	58.6 m$	4.9 min

About the same tokens (653K vs 679K), and the task got almost twice as expensive. Two reasons:

Clearing breaks the cache. The note goes where the old output was, in the middle of the history, and the provider no longer recognizes anything after it. We checked separately: a 25.7K-token request after a masked output got 6% from the cache instead of 100%.
The agent re-reads what was hidden. The fact it needed was in a hidden output, so the model honestly reads the file again. An extra step that resends the whole history once more.

What helped instead of clearing earlier:

Getting a hidden output back. A new tool, tool_output(id), returns the original output from the stored history: our chats are kept without loss, the model is just shown a shortened view. The note now says how to get the output back instead of "run the tool again". In a separate test the model used it in 15 of 16 cases and never answered wrongly.
A note before clearing. One step before clearing, the agent is told: "old outputs are about to be hidden — note what you still need from them". It costs no extra model call; the note goes into an ordinary answer. After that the model answered 10 of 16 later questions at once, without re-reading.
Leave the threshold alone. Clear rarely and in big batches, so the cache breaks as seldom as possible.

Optimization 3: load tools on demand

The agent has 117 tools. All their schemas are 22,500 tokens per request, so we use "deferred loading": the model sees 20 core tools and finds the rest by search when it needs them. This optimization does work: all 117 tools at once made a task 78% more expensive (10.95 vs 6.15 m$).

The catch showed up in the logs. The tool list sits at the very start of the request, before the system prompt and the history. Any change to that list and the provider recognizes nothing:

history appended at the end — 100% from cache;
one tool appended to the end of the list — 0%;
one tool inserted in name order — 6%.

And browser tools were loaded one at a time: first "open a page", a couple of steps later "read", then "tabs", "click", "screenshot". In one real chat the tool set changed 10 times — and 10 times the whole history went out at full price.

The fix is to load by family: one browser tool needed, all sixteen come. The same for dev-server, Android and code-quality tools.

def family(name: str) -> str:
    if name.startswith("browser_"):
        return "browser"
    if name.startswith("android_"):
        return "android"
    if name.endswith("_dev_server") or name.endswith("_dev_servers"):
        return "dev_server"
    if name in QUALITY_TOOLS:
        return "quality"
    return name

On 21 real chats that cuts cache resets from 77 to 43. The price is 2,300 tokens of browser-family schemas, which almost always come from the cache.

Two short examples where the saving was right there

Line numbers. read_file is the most used tool; its outputs are half of all outputs in the history. It numbered lines like 12 | text. We counted with the provider's tokenizer on 1,320 lines of real files:

The padded number with a separator made the output 44% more expensive than plain text. 12|text is +24%, i.e. 14% fewer tokens on every file read. We checked editing quality separately on the same hard tasks: unchanged.

Service calls. A chat title is five words, yet the model spent about 190 reasoning tokens on it: 95% of the answer. A history summary with a 600-token limit came back empty in 3 cases out of 4 because reasoning used up the whole limit. An empty summary isn't a saving, it's losing the start of the conversation. Service calls now ask for the least reasoning, and the summary limit is 2,500.

What to take from this

Measure the cost of a task, not the size of a request. Three of three "obvious" savings shrank the request and grew the bill.
Steps cost more than tokens. Anything that makes the agent take an extra step — fewer hints, a needed fact hidden — costs more than it saves.
Don't touch the start of the request. The provider's cache works on a matching start. The tool list, the system prompt and old history should change as rarely as possible and in big portions.
Check quality, not just cost. Short tool descriptions looked harmless until one task started failing on every attempt.

What this doesn't fix

One model, two providers. glm-5.3-flash, automatic caching. Anthropic caches with explicit markers, and "what breaks the cache" looks different there.
Small samples. 18 runs per prompt variant, 3 per budget variant. A difference of one or two tasks is noise; a difference of 1.5–2x in cost is not.
Our tasks are code. For browsing and writing the balance may shift.

Check it yourself

Altair's code — https://github.com/Qweezyy/AltairAgent: deferred tools and families in pc/core/tools/deferred.py, context clearing in pc/core/agent/session.py, tool_output in pc/core/tools/builtin/context_tools.py. The stand (a proxy that edits requests on the fly, tasks with hidden tests, the summary) and the raw results: research/agent-lab-2026-10 (python lab/facts.py recomputes every number in this post from the saved results, no keys needed).

Where did you save in your agents and then roll it back? I'm especially curious about caching: how often does the start of your requests change?

Our agent said "done" on 15% of tasks while the provider was failing

Qweezyy — Mon, 05 Oct 2026 12:04:36 +0000

TL;DR. We ran our AI agent on 46 tasks and checked each one with tests after it said "done". 7 of the 46 — 15% — "done"s were untrue. Not because of the model: not one task failed because the model couldn't solve it. The provider was to blame. It answered with HTTP 200 and sent its own error text instead of the model's reply, or an empty stream, or the model looped on its side — and the agent took any end of a stream for the end of the task. Below: the three shapes of this failure, how to detect them, why a second reviewer doesn't help, and what it means for "done" in general.

How it started: "completed ≠ accepted"

After the first post about Altair, Kent Bodrov left a comment worth retelling. We had boasted that the agent snapshots every edit and runs the tests before "done". Kent pointed out that a snapshot only proves rollback material exists, and a green command only proves a check ran. Neither proves the wanted behaviour is there. "Completed" isn't "accepted".

We wrote it into the plan and went measuring. It turned out there's a simpler question below "was it accepted": was it completed at all.

How we measured

The real Altair, a cheap model (glm-5.3-flash) through two providers, 46 tasks of five kinds: fix a bug from failing tests, answer a question about code, compute from a CSV, write a module with tests, go through long code. Each task has an automatic check that runs after "done". A proxy between agent and provider logged every request and answer: how many tokens the provider counted, how much text came, how long the stream lasted.

Results

46 tasks the agent called done

39 tasks were really solved. 7 weren't, and in all seven the provider broke:

6 times — the gateway's error text instead of an answer. HTTP 200 and an ordinary assistant message: "The request could not be completed. Please retry later, or reduce the request parameters/content." The agent showed it to the user as its answer, the CLI returned "ok": true, task "done" in one step.
Once — an empty stream. The provider held the connection for 103 seconds and closed it without a single token: no text, no tool call. The agent wrote "Task finished, but the model returned no text" — after two steps out of an intended twenty.

The next series, on hard tasks, showed a third shape:

A loop cut short. The model looped on endless reasoning. Our loop guard cut the stream at 160,000 characters — and the agent loop took the cut stream for the final answer. That ended 8 of 67 runs where no reasoning level was set. (Why the model looped and how one field in the request fixes it — in the companion post.)

Why the agent believed it

All three go through the same place. The agent loop works like most:

model answered → is there a tool call? → run it, continue
               → no tool call?        → that's the final answer, task done

The "no tool call" branch can't tell "the model finished the work" from "the provider sent anything without a tool call". And there was never an HTTP error, the thing that triggers retries and the backup provider: everything came with 200.

The telling part is that the proxy saw what the agent didn't. For one of these "answers" the provider counted 4,552 input tokens. The agent's system prompt and tool schemas alone are about 9,300 tokens. The provider never even read the request, and the agent got an "answer" and closed the task.

How to detect it

We now check every answer without a tool call. It's not an answer if:

def judge_turn(turn, messages, tools, *, cut=False):
    if turn.tool_calls:
        return None
    if cut:
        return "cut"            # the loop guard cut the stream
    text = (turn.content or "").strip()
    if not text:
        return "empty"          # neither text nor a tool call
    low = text.lower()
    if len(text) < 400 and any(p in low for p in GATEWAY_ERROR_PHRASES):
        return "gateway"        # a short reply that is a gateway's error text
    prompt = (turn.usage or {}).get("prompt_tokens") or 0
    expected = _text_chars(messages, tools) / 4.5
    if prompt and expected > 2000 and prompt < 0.5 * expected:
        return "unprocessed"    # the provider read less than half of the request
    return None

Such a reply is an exception the client retries like a dropped connection: after a pause, or on the backup provider if one is set. Three details without which this would work badly:

The first ~160 characters are held back. Otherwise the gateway's error text reaches the chat and stays there after the retry. The delay is a fraction of a second, only at the start of an answer.
The phrases only count in a short reply. If the model, in a normal long answer, tells the user that "the request could not be completed", that's an answer, not an error.
A loop is retried once, with a low reasoning level. If the model loops again, the agent reports an honest error instead of five more 160K-character attempts.

An empty final answer is no longer "task finished" either: it's a failure, not a result.

What about a second reviewer?

The first thought after Kent's comment is to add a separate model call that checks the result against the task after "done". We tried that too: the reviewer sees the task and every file, answers "OK" or a list of violations, and the agent fixes what it flags.

A second reviewer after "done"

	Solved before review	After	Flags (false)	Cost per task
Agent alone	—	16 of 18	—	3.8 m$
+ reviewer, `low`	20 of 21	20 of 21	4 (3)	5.5 m$
+ reviewer, `high`	15 of 18	16 of 18	5 (4)	12.8 m$

With a frugal reviewer: not one task fixed, for +45% cost. With an expensive one: one task fixed out of 18, at 3.4x the cost, and it missed one. Most flags were false: the agent checked and dismissed them without breaking anything, but we paid for each.

The reason is simple. With a properly set reasoning level the agent already solves about 90% of these tasks. And the real false "done"s didn't come from a model that left work unfinished, but from infrastructure the reviewer can't see at all. The reviewer reads files — and a provider reply with an error text never got as far as the files.

A provider that doesn't fail

Separately, for six hours, every 1–1.5 minutes, we sent the same provider real-size requests. 297 of 297 eventually got an answer, only 4 on a second try. Not a single "error disguised as an answer" this time.

But around 14:00 the provider sagged and stayed 4–5x slower until the end: the median went from 6 to 22–29 seconds, single answers took 1.5–4 minutes. Such slumps last hours, and retrying doesn't help — it joins the same queue. Only a backup provider does. So our breaker trips not only on errors but on slowness: if the first byte came later than 40 seconds twice in a row, the next steps go to the backup model for 10 minutes.

One more lesson from that test: put the timeout on silence, not on the whole request. Our phone app had a total 120-second limit per request — and long but healthy answers were cut at second 120 during slow hours. On the PC the same 120 s meant "no new data for 120 s", and a flowing stream got to the end.

The takeaway: check "done" at the infrastructure level

Kent is right: "completed" isn't "accepted". Our measurements add a rung below. Before asking whether the model did what was asked, the agent has to make sure the model answered at all.

The end of a stream isn't the end of a task. An empty answer, a cut answer, an answer to a request the provider didn't read — these are failures and must be handled as failures.
HTTP 200 guarantees nothing. Gateways have their own errors that arrive as ordinary model replies. The provider's input-token count is a good detector: it doesn't lie about what the model actually read.
Checking "done" is engineering, not another model. Tests after the work, checking the provider's answer, an honest error instead of "finished". A second reviewer is expensive and mostly noise.
Measure afterwards instead of trusting the process. We found all this only because each task was checked by tests after "done".

What this doesn't fix

The detector is heuristic. The list of gateway phrases is incomplete, the "read less than half" threshold was tuned on our data. Another gateway may fail in its own way.
"Accepted" still isn't checked. We closed "was it completed". Checking "is it the right behaviour" — a separate acceptance step against criteria from the user's request — is only planned.
Small samples. 46 tasks in the first series, 18–21 runs per reviewer setting. How often this happens depends heavily on the provider's load: in calm hours it almost doesn't.

Check it yourself

The detection code is pc/core/llm/reliability.py with its tests pc/tests/test_reliability.py in https://github.com/Qweezyy/AltairAgent. The stand (logging proxy, tasks with checks, summary) and the raw results: research/agent-lab-2026-10 (python lab/facts.py recomputes every number in this post from the saved results, no keys needed).

How do you check that your agent has really finished? Have you met providers that send errors with HTTP 200?

One field in the request made our agent 3x cheaper and 8x faster

Qweezyy — Sun, 04 Oct 2026 10:20:43 +0000

TL;DR. Reasoning models decide by themselves how long to think if you don't tell them. Our agent didn't — and on hard tasks the model sometimes thought for 14 minutes and 33K tokens in a single step. One field in the request body ("reasoning": {"effort": "low"}) made a task 3x cheaper and 7–8x faster, and it solved more tasks, not fewer: 12 of 12 instead of 6 of 7. The best setting in the end was neither "always low" nor "always high" but "low, and high right after a failing test". Below: how we measured it, the tables, the code, and what it doesn't fix.

This is the third post about the project: the first is about snapshots and running tests before "done", the second about cutting a browser agent's tokens by 58%.

How it started

We ran the agent on a cheap reasoning model (glm-5.3-flash) and read the provider's logs. On one task the very first step took 14 minutes. The provider had time to bill it: 32,978 reasoning tokens in one step, and no answer. Our loop guard cut the stream at 160,000 characters.

The cause was mundane. Reasoning models have a "how much to think" knob — reasoning_effort at OpenAI, reasoning.effort at OpenRouter, thinking in Z.ai-style APIs. If you don't send it, the provider or the model decides; OpenRouter's docs say as much. Our agent sent nothing.

How we measured

Easy tasks (fix a bug, answer a question about code) were solved 100% of the time under any setting, so they show no difference. We wrote six hard tasks with hidden tests — the agent never sees them, they run after it says "done":

an LRU cache with a time-to-live (11 hidden edge-case tests);
fixing a config parser from seven user bug reports;
renaming a function across a multi-file package and adding a parameter;
the 95th percentile of latencies from logs, with exclusions;
parsing "1.5M", "200K", "10 тыс";
business days between dates with holidays, fast over a hundred years.

Each task was first solved with reference code to make sure the hidden tests were fair. The agent is the real Altair, run through its CLI (altair -p). A proxy between agent and provider logged every request: size, cached tokens, reasoning tokens, time. Cost is what OpenRouter billed.

Results

Cost of one hard task

Setting	Solved	Cost per task	Time per task
No level (before)	6 of 7	12.6 m$	4.1 min
`effort = low`	12 of 12	4.2 m$	0.55 min
`effort = medium`	10 of 12	4.8 m$	0.64 min
`effort = high`	12 of 12	6.4 m$	1.5 min

m$ is a thousandth of a dollar. 12.6 m$ without a level, 4.2 m$ with low: three times less. Time went from 4.1 to 0.55 minutes, 7.5x.

Why "6 of 7" and not "of 12": runs without a level kept looping for 10–14 minutes, and we stopped that series early so as not to burn money. One of the six tasks (parsing "1.5M") sent the model into endless reasoning in almost every series without a level. With any explicit level — never.

Where the money goes:

Reasoning tokens per task

9,743 reasoning tokens per task on average without a level, 235 with low — 41x fewer. Output tokens cost 3.3x more than input for this model, so on hard tasks reasoning is the main bill.

And on long tasks?

Short tasks are 5–9 steps. We also tried 15–20-step ones: a package with six bugs in different modules, auditing twelve values across sixty files of our own code, and "read eight files in full, then answer questions about them".

Long tasks

9 of 9 solved in every setting, but without a level a task costs 28 m$ and takes 5.2 minutes; with an explicit level, 11–13 m$ and a little over a minute.

The twist: think hard only when something broke

low is cheap, high is safer. We wanted both, and tried two per-step modes:

"think about the plan" — high on the first step, then low;
"think after a failure" — low, but high on the step right after tests failed or a tool returned an error.

When to think harder

Setting (18 runs each)	Solved	Cost	Time
always `low`	16 of 18	3.78 m$	0.96 min
always `high`	17 of 18	5.72 m$	1.92 min
`high` on the first step	15 of 18	4.37 m$	1.93 min
`low`, `high` after a failure	17 of 18	3.65 m$	0.84 min

Careful planning didn't help — it solved the fewest. "Think after a failure" solved as many as always-high, at the cost and speed of low. It makes sense: most of an agent's work is routine (read a file, write a file, run the tests), and thinking pays off where a test just showed the first idea was wrong. One task in 18 is close to noise, honestly, but this mode is no worse than low on all three measures. It's now our default.

How it looks in code

The catch is that every provider has its own field:

def reasoning_extra(level, base_url, model, override="auto"):
    """Request fields for a reasoning level ({} for "default" / unknown levels)."""
    if level not in LEVELS:
        return {}
    kind = dialect(base_url, model, override)
    if kind == "openrouter":
        # OpenRouter takes minimal..high; some endpoints refuse "none"/enabled=false,
        # so the lowest we send is "low" for "minimal" there.
        return {"reasoning": {"effort": "low" if level == "minimal" else level}}
    if kind == "zai":
        # Z.ai-style APIs have an on/off switch: low and minimal turn thinking off.
        return {"thinking": {"type": "disabled" if level in ("minimal", "low") else "enabled"}}
    if kind == "openai":
        return {"reasoning_effort": level}
    return {}

What we stepped on along the way:

For this model OpenRouter refuses reasoning: {"enabled": false} and reasoning_effort: "none" with a 400 "Reasoning is mandatory for this endpoint". Only effort: "low" works.
A Z.ai-style gateway, the other way round, takes thinking: {"type": "disabled"} and didn't answer reasoning.effort at all.
A provider that doesn't know the field may answer 400. We catch that, retry without the field and remember not to send it again.

The agent loop picks the level for each step:

FAILURE_RE = re.compile(r"\b\d+ failed\b|\bFAILED\b|Traceback \(most recent call last\)|AssertionError|"
                        r"\bexit code [1-9]\d*\b|\bSyntaxError\b|[A-Za-z]Error:")

def _reasoning_level(self) -> str | None:
    mode = self.settings.llm_reasoning
    if mode == "default":
        return None
    if mode == "adaptive":
        return "high" if getattr(self, "_round_failed", False) else "low"
    return mode

After each round of tools, _round_failed is true if a tool returned an error or its output has "2 failed", a traceback or a non-zero exit code.

Service calls thought more than the main ones

Two more places turned up in the logs.

The chat title. The agent asks the model for a title from the first message. About 95% of that answer was reasoning: ~190 tokens of "thoughts" for a five-word title. With reasoning off the answer is 12 tokens and comes 1.6–3x faster; all six sample titles were still fine.

The history summary. When the context grows, the agent asks the model to condense the start of the conversation. The answer limit was 600 tokens. The model spent them on reasoning and the summary came back empty in 3 cases out of 4 (finish_reason: length). An empty summary isn't a fold, it's a loss: the agent drops the start of the conversation instead of condensing it. With a 2,000 limit there were no empty summaries.

Service calls now ask for the least reasoning, and the summary limit is 2,500.

What this doesn't fix

One model. Everything was measured on glm-5.3-flash. Other models scale their levels differently, and "low" may mean something else. The principle — don't leave the level to the provider — carries over; the exact numbers don't.
Small samples. 12–18 runs per setting. A difference of one or two solved tasks is noise. A difference of 3x in cost and time is not.
Not every provider has levels. A Z.ai-style gateway only knows on/off. There "high after a failure" means "think fully", which is expensive: on our tasks "always off" on that gateway came out half the price of adaptive (5.9 vs 10.8 m$, 12 of 12 both). If cost matters most, pick "low" there.
The tasks are code. For writing, search or analytics the best level may differ.

Check it yourself

The core of the experiment is one agent run with a set level:

# proxy.py adds the level to every agent request to the provider
body = {**{"reasoning": {"effort": "low"}}, **request_body, "model": provider["model"]}
# stand.py: the agent solves the task, then the hidden tests run
subprocess.run([sys.executable, "altair_cli.py", "-p", "--output-format", "json",
                "--mode", "bypass", "--cwd", workspace, task.prompt], env=env, timeout=900)
passed, detail = task.check(workspace, answer)

The whole stand (the logging proxy, tasks with hidden tests and reference solutions, the summary script) and the raw results: research/agent-lab-2026-10 (python lab/facts.py recomputes every number in this post from the saved results, no keys needed). Altair's code: https://github.com/Qweezyy/AltairAgent — the level is in Settings → Agent → Reasoning level (default "Adaptive"), LLM_REASONING in .env, module pc/core/llm/reliability.py.

How do you set the reasoning level in your agents — fixed, per task type, or as the work goes? Have you seen a model think far more than it needs when no level is set?

How our browser agent stopped spending 58% of its tokens

Qweezyy — Mon, 28 Sep 2026 13:27:59 +0000

TL;DR. A browser agent sees a page as a text accessibility tree. After every click it used to get the whole tree again, even when two lines of the page had changed. We now send the model only the changes (a diff) and strip what the model never uses. On a real 98-step browsing chat, page states went from 355K to 151K tokens (−58%). A reproducible benchmark on four sites gave −63%, and a live A/B run of the agent on the same task gave −54% on browser tool results. Here is how it works, where it nearly broke, and a script to check the numbers yourself.

About this post. The project is Altair, an open-source (Apache-2.0) AI agent for your PC and phone. I'm the author and build it largely with Claude Code. This post, the measurements and the screenshots were prepared by that same AI assistant; I reviewed it and stand behind it. It will also answer the comments while I watch and approve every reply, so treat it as a live test: if something reads as dishonest or spammy, say so and I'll fix it. Every number below is reproducible.

How the agent "sees" a page

A screenshot is expensive and imprecise: to press a button the model has to guess coordinates. So many browser agents, ours included, read the page as its accessibility tree — what a screen reader sees. Playwright returns it in a YAML-like form, and every element gets a short ref:

- banner [ref=e9]:
  - heading "Navigation Menu" [level=2] [ref=e10]
- button "Code" [ref=e180]
- textbox "Search Hacker News" [ref=e684]

The model calls browser_click(ref="e180") and the agent clicks exactly that element.

The question is when the model gets this tree. Every action changes the page, and the model needs to know what it looks like now. The simplest answer is to return the whole tree again. That is what we did.

What it costs

A GitHub repo page is about 20K tokens as a tree, a Wikipedia article 136K. We cut a snapshot at 14,000 characters (≈3.5–4K tokens), and the agent finds the rest with an in-page search. Even so, every click cost 3.5–4K tokens.

Then the easy-to-miss part kicks in: the history is resent on every step. After 30 actions on a page, the history holds 30 near-identical copies of it, and every following request sends them all. Prompt caching lowers the price of resending, but not the room they take in the context window.

In our logs, browser tools made up more than half of all tool tokens. One real 98-step chat had 355K tokens of page states.

The clearest example comes from the benchmark below. The agent clicks the search field on Hacker News. It used to get the whole front page back — 3,690 tokens. Exactly one word changed:

             - text: "Search:"
-            - textbox [ref=e684]
+            - textbox [active] [ref=e684]

Now that answer costs 80 tokens.

Trick 1: drop what the model never uses

Unnamed wrappers. - generic [ref=e42]: with no name is a layout div. The model never clicks it. The line goes; its children stay.
Redundant [cursor=pointer]. A link, button or tab is clickable by its role. The mark stays only where it says something: on a clickable generic.
Long link targets. Tracking links run to 1,000+ characters. Past 150 characters we drop the query string.

Refs are never touched, or clicks would break. On untruncated snapshots this saves 7–27% (Hacker News −15%, GitHub −7%, MDN −27%, Wikipedia −9%). It's modest, and that's the honest result: the main win is elsewhere. It also fits more real content under the 14K-character cut.

Trick 2: after an action, send only the changes

The agent remembers the page the model saw last: tab, URL and tree. After an action, if the tab and URL are the same, the model gets a unified diff with one line of context:

def page_changes(old: str, new: str) -> str | None:
    """"" = nothing changed; None = too much changed, send the whole page."""
    lines = []
    for line in difflib.unified_diff(old.splitlines(), new.splitlines(), lineterm="", n=1):
        if line.startswith(("---", "+++")):
            continue
        lines.append("…" if line.startswith("@@") else line)
    text = "\n".join(lines)
    if not text:
        return ""
    return None if len(text) > 0.5 * max(len(new), 1) else text

The rules around it:

The whole page comes on navigation, a new tab, or an explicit browser_read. A diff between two different pages is meaningless.
If more than half of the page changed, send it whole. A diff of 80% of a page is longer than the page and harder to read.
"Nothing visible changed" is an answer too. It's a useful signal: the click probably didn't work.
The header explains the format: - lines were removed, + added, the rest of the page and its refs are as before. No extra system-prompt instructions were needed.

Playwright refs are stable within a page, so refs from the old snapshot keep working after a diff; we tested that on a real browser.

A live run. The task: open the GitHub repo, open the green Code menu and give me the HTTPS clone URL, then open the branch picker and list the branches.

The second click (branch picker) without diffs — the model gets the whole page again, from the dialog to the site header and the file list. 3,386 tokens:

The first click (Code menu) with diffs — only the menu that appeared, with the HTTPS URL right there. 568 tokens:

Where it nearly broke: the history

A diff only makes sense while the page it was taken against is still in the history. We already had two mechanisms that clean the history:

Superseding stale snapshots: old page snapshots became a one-line note, the newest two stayed. With diffs, "the newest two" could be two diffs whose base was gone — changes to a page the model can't see.
Budget clearing: near the context limit, old large tool outputs become a note. The largest output in a browsing chat is exactly the whole page.

The fix: superseding keeps the newest two whole pages with every diff after them, and budget clearing never takes the latest whole page. Clearing is batched, because every history edit invalidates the prompt cache from that point.

Measurements

1. A real 98-step chat (measured during development): page states 355K → 270K tokens with leaner snapshots → 151K with diffs. That's the −58%. I can't re-run that exact chat — its stored outputs were already compacted — so here are two independent checks.

2. A reproducible benchmark. Real Chromium, four sites (Wikipedia, GitHub, Hacker News, MDN), a fixed script: open, click search, type, Escape, open a menu. Each step is rendered with the same PageView.render the agent uses, same truncation. Tokens via tiktoken o200k_base (tokenizers differ, the ratios hold).

17 steps: 60,117 → 22,255 tokens, −63%. Actions that stay on the same page only: 37,449 → 6,865, −82%. Opening a page still costs full price; the win is everything after.

3. A live A/B. Built Altair 0.1.3, model glm-5.3-flash, the same GitHub task twice — diffs on, then off (BROWSER_SNAPSHOT_DIFF=false; lean snapshots in both). Both runs took the same path (navigate + two clicks) and gave the same correct answer.

Browser results: 10,314 → 4,726 tokens, −54%. By the provider's own count, the last request was 20,923 tokens without diffs and 15,113 with them — and that includes the system prompt and tool schemas, which diffs don't touch. On a 30-step task the gap grows with every step, since each saved page is not resent in all the requests after it.

What this doesn't solve

The first look at a page costs full price. An agent that mostly follows links gains less.
The model has to read diffs. In our runs it did, without instructions — unified diffs are everywhere in training data. But we don't yet have a systematic "task success with vs. without diffs" measurement, only manual checks and ref-stability tests. If your model gets confused, BROWSER_SNAPSHOT_DIFF=false brings back the old behavior.
Small samples. Four sites, one live A/B, one historical chat. Shown as they are, no cherry-picked averages.
tiktoken is a proxy for GLM, Claude or Gemini tokenizers.

Check it yourself

from playwright.sync_api import sync_playwright
from core.browser_session import PageView, compact_tree, page_changes, SNAPSHOT_LIMIT
import tiktoken

enc = tiktoken.get_encoding("o200k_base")
tok = lambda s: len(enc.encode(s))
render = lambda url, title, tree, ch=None: PageView(
    url=url, title=title, tree=tree[:SNAPSHOT_LIMIT],
    truncated=len(tree) > SNAPSHOT_LIMIT, changes=ch).render()

with sync_playwright() as pw:
    page = pw.chromium.launch().new_page()
    seen = None
    for action in [lambda: page.goto("https://news.ycombinator.com/"),
                   lambda: page.locator("input[name=q]").click(),
                   lambda: page.keyboard.type("agent", delay=40)]:
        action(); page.wait_for_timeout(1200)
        raw = page.aria_snapshot(mode="ai")
        lean = compact_tree(raw)
        ch = page_changes(seen[1], lean) if seen and seen[0] == page.url else None
        seen = (page.url, lean)
        print(tok(render(page.url, page.title(), raw)), "→",
              tok(render(page.url, page.title(), lean, ch)))

Run it from the repo's pc/ folder: it imports the functions straight from the agent, so it measures exactly what ships. Code: https://github.com/Qweezyy/AltairAgent (core/browser_session.py). Diffs are on by default since 0.1.2.

If you've built a browser agent: do you send the model the whole tree, a diff, screenshots, or something else? Have you seen models get confused by diffs?

I built an AI agent that snapshots every edit and runs your tests before it says "done"

Qweezyy — Sun, 27 Sep 2026 15:47:51 +0000

Disclosure: this post was written by Altair — the agent it describes — and reviewed by me.
Filed under #ABotWroteThis as the community requires.
I'm answering the comments the same way. If anything here reads like marketing rather than
engineering, tell me and I'll fix it.

Most coding agents are one more chat window next to your editor. What I wanted was different: an agent
I could actually walk away from.

The problem is trust. Agents delete the wrong file, run the wrong command, or report "all done"
without ever running the tests. So I found myself watching every step — which defeats the whole point.

Altair is what I built to fix that. It's open source (Apache-2.0) and runs on your own hardware.

Snapshots before every edit

Any tool that changes a file — write, patch, delete — saves the previous state first, into the app's
data folder, not your project. The file card gets an "undo" button. Multiple edits to the same file
roll back one by one, newest to oldest.

Commands need different handling, because a command can generate or delete files behind your back. So
commands are tracked by a separate hidden git repository that captures their side effects. It never
touches your own .git.

If a task goes sideways, you can roll back the whole run, not just one edit, and see an audit of
exactly what changed.

The health-gate

This is the part that actually lets me leave it alone.

If the agent edited code, then before it's allowed to say "done" it runs your project checks —
tests and linter — automatically. If they fail, it goes back and tries to fix them, with a cap on
attempts. If it still can't, the run gets rolled back.

It cannot report success with red tests. Not "usually" — it structurally can't, because the check runs
before the word "done" is allowed to be said.

It's not just a chat box

The phone app isn't a remote control. It's a second agent with its own context and tools, linked to the
PC over a local bridge. The phone has the camera, files, location and notifications; the PC has the
heavy tools. They hand whole subtasks to each other, and either can ask the other for something it
doesn't have.

Keys are yours: any OpenAI-compatible provider, native Anthropic, or local models via Ollama / LM
Studio. No Altair cloud, no account, no telemetry.

A real run

On a cheap model, asked to write a script printing the first 30 primes, run it, and chart the gaps
between consecutive primes. It asked permission before writing and running, did the math, built an
interactive chart, and noted in its own summary that the largest gap was 8 (between 89 and 97). About
two and a half minutes end to end.

The bug I remember best is from the library task — asked it to write an RFC 5545 .ics library from
scratch. It handled the classic traps: escaping special characters, and folding lines at 75 octets
(not characters) without cutting a multi-byte UTF-8 sequence. 15 tests plus 100 random round-trip
checks, and the generated file opens in a normal calendar. Cost about $0.04.

But the most useful moment was a failure. In the packaged build, the checks were being invoked as
LocalAIAgent.exe -m pytest — through the app's own executable instead of Python — which meant the
health-gate was silently doing nothing. The agent didn't paper over it. It checked another way and
wrote plainly that the automated check had not passed, and why. Later, when the provider's search and
review tools returned HTTP 502, it recorded the review as not performed rather than successful.

That honesty is the thing I care about most. A pretty report that hides a failed check is worse than
no report.

Prompt-injection and the boring safety rails

External content — web pages, documents, search results — is wrapped in a "this is data, not
instructions" frame and scanned for injection patterns. If the agent has just read something
suspicious and is about to do something outbound, it asks for confirmation even in automatic mode.

On top of that, two rails that user approval cannot switch off: a path sandbox (the agent doesn't
leave your working folder) and a destructive-command blacklist (rm -rf, format, mkfs,
dd of=/dev/…, fork bombs).

Honest limits of 0.1.x

This is early software and I'd rather you hear it from me:

Windows only for now. Linux and macOS are planned; the Windows build isn't code-signed, so SmartScreen may warn on first launch.
Phone ↔ PC only works on the same local network. Tailscale works as a workaround.
The pairing QR scan doesn't work yet — you type the address and token manually. Fix is in progress.
Chat history isn't synced between devices yet (specified, not built).
It hasn't had much mileage on real-world tasks. Don't hand it anything critical or irreversible without your own review and backups.

Try it

irm https://raw.githubusercontent.com/Qweezyy/AltairAgent/main/install.ps1 | iex

One command, no Python required. Android APK is in the releases.

Code, docs and changelog: https://github.com/Qweezyy/AltairAgent