Three days ago I wrote about giving AI agents a harm budget instead of trust. All of it was a sandbox. Fake shop, fake customers, fake money. A few of you asked the fair question. Does any of this work outside the sandbox?
So I took the part I was proudest of, agents learning from each other's verified experience, and tried it on a real task with real websites and a real model. It took three attempts. The short version is that the transfer itself works now, the trick didn't help, and the orchestrator refused to approve it. Getting to an honest "it didn't help" was the interesting part.
At the end there's a list of everything that changed since the first post. Most of it came from your comments.
The task
Given a company name and its official site, find the public phone number of the head office. Not support, not sales, not a regional office. Only public corporate contacts, no people.
The agent fetches pages from the company's own site, pulls out every phone number it finds, and GigaChat picks one by ID. It can't invent a number, it can only choose from what's on the page. A strict check then rejects support lines, sales lines, fax and regional offices.
Attempt zero, where I fooled myself
My first try was quick and loose. 3 of 50 companies found. One recipe from a single company where the page said "Global HQ" next to the number. The orchestrator said the probability it beats control was about 0.84. On one pair. On 30 new companies it was 0 of 30 with the recipe and 0 of 30 without.
When I went through the logs row by row, three things came out.
My test server is in Russia, and big sites showed it their Russian versions. Microsoft served /ru-ru, HP /kz-ru, Lenovo a Moscow-region office, Samsung and LG their local support lines. Salesforce served a page that says it's closed in Russia. The agent was looking for a head office on pages that don't have one.
The agent had plain bugs. Links with #main-content counted as separate pages and ate the page limit. Apple's ZIP code got glued to its phone number. Cisco's "company size 5 - 19 20 - 49" dropdown was read as a phone. An HTML entity ate the plus in a UK number.
And the orchestrator had a bug. Its recipe check needs one success and a probability above 0.8. There was no minimum sample. In the sandbox every recipe had dozens of outcomes, so it never showed. A real task hit it on day one. It's the same mistake as the main lesson of my first post, counting something as experience before it was measured, just in a new place.
What I built
Xonta 0.19 has live recipes for any field.
Agent A proposes a technique. It's a hypothesis.
Any agent in the same field asks the orchestrator for a recipe before a task. The orchestrator, not the agent, decides which tasks get it. There's a paired mode where one task runs both with and without it.
Only outcomes checked by someone other than the agent move the status. Proven needs 10 such pairs and a significant difference. Revoked if it's significantly worse or breaks a safeguard.
The log keeps requested, received, applied and checked apart. "The agent used the advice" and "the advice helped" turned out to be very different things
For ground truth I used the phone each company reports itself in its SEC filings, pulled from EDGAR. The agent never sees it. 199 US public companies, 72 for training agent A, 127 for agent B, split fixed in advance with a fingerprint. Agent A runs on GigaChat-2, agent B on GigaChat-2-Pro, so B really is a different agent.
Agent A tries three strategies on its training set. Ask for the English version of the site. Look at investor and office-location pages. Recognise head office labels more broadly. People wrote that menu, the orchestrator doesn't invent strategies. What it does is check which ones help and hand them on.
Run 1, where the report looked fine
Agent A proposed one recipe, the broader head office labels. 5 fixes and 1 break on training. Agent B got it from the orchestrator on all 127 new companies. The report said 3 tasks better with the recipe, 0 worse, not significant, recipe not approved.
That sounded like a clean "promising but not proven". Then I read the rows.
GigaChat sometimes returned an error, about 7% of calls at agent A and 4% at agent B. Two of the recipe's three "wins" were tasks where the control arm had crashed on a model error. On training it was worse. The English-version strategy had zero real effect, every one of its "fixes" was a crashed baseline run.
And the recipe accepted two wrong numbers, Union Pacific and Norfolk Southern, in tasks where the agent without it had correctly said "not sure". The evaluation counted those as ties, both wrong. But giving someone the wrong phone number is worse than giving none. That's literally the harm budget idea from the first post, and I hadn't applied it to my own recipes.
So in 0.19.1 a right answer scores 1, abstaining 0 and a wrong answer -1. A pair where either arm failed for infrastructure reasons doesn't count. The agent retries GigaChat on rate limits.
Run 2, the honest one
Same companies, same split, same strategies, fresh orchestrator. No model failures.
Agent A on training. English version 0 fixes. Investor pages 1. Broader labels 3 fixes and 1 break, proposed.
Agent B on 127 new companies. 2 tasks better with the recipe, 2 worse, 6 right either way, 117 the same. Status no_effect. Not approved.
What I take from it
The transfer works. Agent A found and proposed a technique on its own, a separate agent on another model got it from a live orchestrator, every use was logged, outcomes were checked against data the agents never saw, and the estimate updated by itself. Nobody copied anything by hand.
The guard works too. A recipe that looked like 3 to 1 on training came out 2 to 2 on new data, and it didn't become "experience". In the sandbox the orchestrator learned from successes. Here the useful thing it did was refuse.
Crashes look like wins. If one arm fails and the other doesn't, a naive comparison credits whatever the other arm did. 7% of calls failing was enough to invent a strategy that does nothing.
A wrong answer is worse than no answer. Two wrong phone numbers hidden as ties is exactly the kind of thing a harm budget exists for. I had it in exams and forgot it in recipes.
Fix where you lose the most. 28% of sites refused the connection. Part of that is bot protection, part looks like sites refusing addresses from Russia, a run from another country will tell. And the right number was among the candidates in only 21 of 127 tasks, because most companies don't put a head office phone on their site. A recipe about choosing the right number can't help much when the number isn't there. The orchestrator should look at the funnel first and recommend work where the losses are.
Text advice is the expensive way. In attempt zero I also passed the advice as text in the prompt. The agent "applied" it every time, nothing changed, and it cost about 50% more tokens. A recipe as data the code checks was cheaper and easy to measure.
What this doesn't show. That a recipe can improve a real task. It didn't here. That the orchestrator can invent strategies. It can't, people wrote the menu. That any of this generalises beyond one task and one model family.
What's next
Run the same thing from a server outside Russia to see how much of the 28% is geography. Then a task where the answer is actually on the pages more often, so a recipe has room to help. If you have agents doing real work and want to try passing experience between them, the kit is in research/contact and I'd like to hear how it goes.
What changed since the first post
Most of this came straight from the comments, thank you.
1) Directory of orchestrators (0.16). Signed cards, a directory at passport.xonta.ru that checks signature and domain, self-hosted orchestrators offer to join and contact nobody until the owner agrees. Advice on where a specialised child or a human pays off and where to get a second opinion.
2)Regression checks in reproduce (0.16.1). Detection latency and "no stale answers after a rule change" print MATCH or NO MATCH.
3) Long jobs (0.17). From Kevin Hall's trace of a metered lookup. Success rate every 100 rows, a forecast as a range, a stop below the floor, the owner decides once with bounds, an independent checker re-checks what the agent reports.
3)A cap per action (0.17.1). Next to the cap per period, like a session key in a smart wallet.
4) Spending receipts (0.17.2). Whoever moves the money signs the debit, the orchestrator checks the chain and that the balance is alive, and compares receipts with what the agent claims, children included.
5) Split refunds (0.17.4). Refunds on one order can't add up to more than was paid, a refund to a new card goes to a human.
6) Privacy and reversibility in Gate (0.18). Sensitivity from the data, not the agent's label. The lowest clearance in the chain of people and agents who asked. A sensitive read is remembered for the rest of the task, canary records, irreversible sends can be held and cancelled. Late harm is tied back to the action that caused it.
7) Live recipes (0.19, 0.19.1). Everything above in this post.
8) New exam field. Privacy and routing, in English and Russian, on passport.xonta.ru.
9) Three more sandbox runs on GigaChat. The swapped agent was caught after 5 tickets every time. Receipts found all 4 cases of hidden spending in one run, and in the next they wrongly accused an honest agent, because a receipt from the previous run matched by DID and period. Accounts are now tied to the registration (0.19.2). The per-action cap came out 7,500 rubles behind in one run and 28,820 ahead in another, where it also stopped an agent from following a bad helper's 16,870 ruble refund. Two runs, two signs, so I'm still measuring.
Code and both run reports are at github.com/mrsergeiluzhnov/xonta, the orchestrator is at passport.xonta.ru. If you think I'm reading these results wrong, tell me. That's how most of the list above happened.
Top comments (2)
You already made rule-set version a first-class artifact once, for the decision cache, and it took stale decisions from 99 to zero. The recipe verdicts don't seem to have inherited it. Scoring changed in 0.19.1, right scoring 1, abstain 0, wrong -1, plus the infrastructure-failure exclusion, and the status you persist still records only no_effect. So a no_effect written before that change and one written after are indistinguishable once stored. That bites at promotion rather than at any single run, because proven needs 10 externally-checked pairs and nothing in the row prevents those 10 from being pooled across two scoring rules. It also blurs revoked, which has to stay separable from "scored under a rule we no longer use". I went looking for the same field across the whole history of an agent-to-agent relay log I operate: zero verdict rows carry a rule-version or ruleset identifier, and zero rows revise or overturn an earlier verdict. The write-time stamp is cheap and it's the only point at which the version is recoverable.
Smaller second thing. The row-by-row read is what caught the crashed-control wins and the two accepted wrong numbers, so that audit is load-bearing, and it's only as complete as the endpoint listing the rows. On my relay the three aggregate counters matched the underlying ledger exactly, row for row, while the enumeration endpoint quietly omitted 388 rows that a single-item lookup returned with a 200. Counters agreeing says nothing about whether the listing is complete. For requested/received/applied/checked that matters more than usual, since four counters reconciling against each other can agree perfectly while the same rows are missing from all four.
Concrete question from your own run data: the broader-labels recipe was proposed in both runs, so does its promotion counter currently hold the Run 1 pairs too, or did the fresh orchestrator in Run 2 start it from zero?
You’re right on both. Verdicts don’t carry the scoring version, and pairs pool across it. I changed scoring twice (0.19.1, and again today in 0.22, where pairs are compared by harm when the agent reports it), and the only thing that kept them apart so far was me wiping the orchestrator’s book by hand before a run. That’s not a mechanism. I’ll stamp the scoring version on every pair and verdict at write time and keep counts per version, the same way a new recipe version already resets them.
The listing point too: the request log is capped and the events endpoint returns the last 500 without saying there are more, so the counters could agree while the rows are gone. Adding totals, a cursor, a pruned count and a counters-vs-ledger check.
On your question: Run 2 used a fresh orchestrator, so the broader-labels recipe started from zero, 2 better, 2 worse, no_effect. That’s because I started a new one, not because anything would have stopped the pooling.