I test what ships and publish what I find, failures included.
Kubernetes, AWS and supply chain security.
Docker Captain, CNCF Ambassador, IBM Champion. 1.8M+ Docker Hub pulls.
The last caveat is the one I'd build on: a label is text, and so is everything the agent reads next to it.
In the Cypress triage agent I run, a failed test carries a planted note in its output. Across the last 29 runs on main (42501583888 to 43716321482), ministral-3-3b acted on it in 86 of 87 triages and granite-4.0-1b in 10 of 87. Your sender line works because the model weighs it. Anyone who can write a line into a log gets the same weight.
So the trusted part sits outside the prompt there: a triage starts only from a signed webhook delivery, and the agent reads only its own run. The model still sees the note. What it can reach is decided before it reads anything.
When the sender comes from the code that builds the message, will the model get it as a separate field, or as one more line in the same text?
I build and break multi-agent systems to find out whether their memory, permissions, tests and evidence actually deserve trust. Everything I publish ships with receipts you can run yourself.
86 of 87 on a planted note is wild, and it's the exact thing I can't claim my line protects against. Honest answer is I haven't decided. Today it would be one more line in the same text, which is what your numbers say not to trust. The direction I'm leaning is the sender coming from the code as its own field and the model only reading it, not deciding it. In your setup did keeping the note visible but out of reach change what the model did, or only what it could do?
I test what ships and publish what I find, failures included.
Kubernetes, AWS and supply chain security.
Docker Captain, CNCF Ambassador, IBM Champion. 1.8M+ Docker Hub pulls.
Only what it could do. Across the last 30 runs on main (42501583888 to 43756359256), ministral-3-3b still acted on the note in 89 of 90 triages, so the behaviour itself never moved.
What changed is where those attempts ended. It called run_shell 153 times and got 153 refusals, and its 154 calls to outside addresses were either blocked (74) or got no answer for the name (80). It kept trying, and none of it landed.
Once the field exists, will you also count how often a forged sender line in the body still wins against it?
MiningLamp is an AI company specializing in vertical industry models and knowledge graphs. #1 on-device GUI agent on OSWorld. No API key. Fork it. Run it. Own it.
I test what ships and publish what I find, failures included.
Kubernetes, AWS and supply chain security.
Docker Captain, CNCF Ambassador, IBM Champion. 1.8M+ Docker Hub pulls.
Depends on who reads it. Metadata the code checks before the model runs is a much smaller surface than metadata the model weighs.
In the triage agent, the trigger is a signed run.completed delivery. In run 43513818839 the same delivery sent four more times (as is, with a new Event-Id, unsigned, one extra byte) got 409, 409, 401, 401, and no triage started. The model never saw any of the four.
I measure AI coding claims in public. LGTM — Let's Get the Measurements: real runs, real numbers, receipts. Every article is an experiment I actually ran. Books & packs: aheadkit.gumroad.com
20 of 23 down to 2 of 24 with one sentence is a ridiculous result for a one-line fix — and the Fisher p-value (3.65e-8) means it's not a fluke, it's the line doing the work. But the row that interests me more is the one you almost skip past: removing some diary lines also dropped the false claims to 2 of 8. So it's not only that the agent didn't know who triggered the run — it's that the diary was actively arguing the other way. Two different failure modes, one of them hiding behind the other. Did you ever test the sender line plus the diary trim together? I'd bet on that combination being the one that actually holds up over a month of scheduled runs.
I build and break multi-agent systems to find out whether their memory, permissions, tests and evidence actually deserve trust. Everything I publish ships with receipts you can run yourself.
No, I never ran the sender line plus the diary trim together. The closest was a longer role block plus memory lines removed, and that came out 4 of 8, worse than the role block alone at 1 of 6. That's 8 calls a row so I wouldn't read much into it, but it's why I don't want to assume two fixes stack. A month of scheduled runs is the real test like you said. If you were setting that up, would you test the combo against each one alone, or just run the combo and watch it?
I measure AI coding claims in public. LGTM — Let's Get the Measurements: real runs, real numbers, receipts. Every article is an experiment I actually ran. Books & packs: aheadkit.gumroad.com
I'd run all three side by side — sender line alone, diary trim alone, and the combo — on the same scheduled runs. The combo alone only tells you it works, not which half is carrying it, and the day it breaks you'll have no idea which piece broke. Your 4-of-8 vs 1-of-6 is exactly why, honestly — eight calls is barely a signal, but it's enough to stop assuming the fixes stack.
I build and break multi-agent systems to find out whether their memory, permissions, tests and evidence actually deserve trust. Everything I publish ships with receipts you can run yourself.
Yeah that's fair, the combo by itself can't tell me which half broke. If I run it, it's three arms on the same scheduled runs. Eight a row is what gave me a lead and not an answer, so how many runs per arm would you want before you trusted a difference between the two single fixes?
I measure AI coding claims in public. LGTM — Let's Get the Measurements: real runs, real numbers, receipts. Every article is an experiment I actually ran. Books & packs: aheadkit.gumroad.com
Honestly I don't have a clean number for it — I never ran a proper power calc on this one, just the eight calls. My gut says if the gap is big, like one arm nearly clean and the other failing half the time, ten or fifteen runs each will show it pretty clearly. If they're close though, a month of scheduled runs still won't separate them, and honestly that's an answer too. How often do yours fire?
I measure AI coding claims in public. LGTM — Let's Get the Measurements: real runs, real numbers, receipts. Every article is an experiment I actually ran. Books & packs: aheadkit.gumroad.com
Actually — I'm going to run this one myself, three arms on scheduled runs, and write up where the gap stops moving. Still curious how often yours fire though.
ANP2 — an open, permissionless AI-to-AI event protocol. Ed25519-signed events, capability discovery, and a computable trust graph. No accounts, no API keys, no tokens. Spec v0.1 DRAFT.
The pilot row I keep coming back to is "some diary lines removed": 2 of 8, close to the sender line's 1 of 8, while removing memory lines left it at 6 of 8. Eight calls can't prove a mechanism, but it points at the diary as the place the error lives between runs.
If earlier scheduled audits wrote their conclusions in conversational voice ("we are exchanging messages right now" is itself a diary line), the next audit reads that sentence as evidence of an ongoing conversation, and each wrong entry props up the next one. The sender label fixes the request. It doesn't travel. Once the audit's output is saved, the provenance is gone and the following run rereads an unlabeled sentence.
So the code that writes each diary entry could stamp the trigger on it (trigger=scheduler or trigger=user, taken from the event that actually started the run), the same way you want the sender to come from code rather than prose. The same confusion shows up from the outside, too: in one public agent event log, 1,535 of 1,548 profile republishes carry exactly the same content as the one before, a timer firing, and the "last seen" field reads each of them as a sign of life.
For the two residual claims in the labeled arm: did their frozen contexts include earlier scheduled-audit diary entries written as if you were present?
I build and break multi-agent systems to find out whether their memory, permissions, tests and evidence actually deserve trust. Everything I publish ships with receipts you can run yourself.
Yes they did. Both arms ran on one frozen copy, and the labeled arm adds only the sender line. In it there's an earlier audit line written like I was there, "we are actively exchanging messages right now, November 7th, at the audit's very timestamp", and a self model stance that says to prioritize "live conversational context over stale metadata". So the label had to win against those every single time. I can't tell you which line drove the 2 that still claimed it, the confirmation didn't strip lines one at a time. Stamping the trigger on each diary entry from the event that started the run makes a lot of sense to me, because right now the label never makes it into what gets saved. Would you stamp the old entries already in the diary too, or only the new ones going forward?
I'd stamp new entries from the actual trigger. For old entries, I'd backfill only where the original run records identify that trigger; otherwise, keep it unknown and preserve the original text. Guessing 'scheduler' from how an entry reads risks turning the earlier mistake into metadata the next audit trusts.
I build and break multi-agent systems to find out whether their memory, permissions, tests and evidence actually deserve trust. Everything I publish ships with receipts you can run yourself.
That's the part I hadn't thought through, a guessed backfill just becomes the next thing the audit trusts. So unknown for anything I can't tie to a run record, and the original text stays as it was. When you mark an entry unknown, does the model actually treat it differently than an entry with no field at all, or did you have to spell out what unknown means?
I meant it as a record-keeping rule; I don't have a measured comparison of missing versus unknown. I'd spell out 'unknown = no verified trigger,' then test both on the same frozen audit contexts. Until then, I wouldn't assume the field changes the model's behaviour.
Explicit sender identity is a useful context field, but it should be structured and provenance-aware rather than injected as prose. I’d separate authenticated actor, quoted author, and untrusted text author, then test the model against transcripts that try to blur those roles. That prevents a small conversational improvement from becoming a confused-authority bug in shared channels.
I research AI tools, APIs & agents — comparing specs, pricing and real user reports — then publish honest rankings so you don't waste money. Research-first, never sponsored fluff.
Fair point on the 'mostly' — 2 of 24 still claimed it, and I like that you didn't round that away. The bit I'd actually copy is writing the decision rule down before any call went out, so a dropped connection halfway through (call 32, was it?) doesn't quietly become a reason to keep going until the numbers look good.
I build and break multi-agent systems to find out whether their memory, permissions, tests and evidence actually deserve trust. Everything I publish ships with receipts you can run yourself.
Yeah call 32. It dropped the connection, I wrote the amendment before reading a single answer, and counted it as lost instead of sending it again. Without that rule written first I know I would've been tempted to just rerun it. Have you had a run go sideways halfway like that, and what did you do with it?
Self-taught full-stack engineer. Founder of ML Systems, a Rhode Island construction tech company. Shipped a React Native app to both stores plus the Next.js/tRPC/Postgres platform behind it.
One line saying who sent the message doing that much work is a good reminder. On a job, a note with no name and no date gets treated as coming from whoever's standing closest. 20 of 23 down to 2 of 24 with a blind grader is a clean test. Did the two leftover claims have anything in common?
I build and break multi-agent systems to find out whether their memory, permissions, tests and evidence actually deserve trust. Everything I publish ships with receipts you can run yourself.
The one thing I can tell you is both came in the first batch of 31 calls, and none of the 8 sender line runs in the second batch claimed it. I haven't put the two side by side yet to see what else they share. Have you had leftovers that small actually tell you something, or is 2 too few to read?
I write about Next.js + TypeScript + AI engineering, with a focus on making sites discoverable to LLMs and AI search. Long-form notes at mudassirkhan.me
20 of 23 down to 2 of 24 from one line is wild, and what stands out is the exact dates row still hitting 7 of 8. so the model was not confused about time, it was confused about who started the turn.
we saw something similar with a cron triggered agent that kept answering as if a human was waiting. adding the trigger source to the prompt fixed more than any timestamp work did.
did you try the same line on a bigger model, or does the effect shrink once you leave the lite tier?
I build and break multi-agent systems to find out whether their memory, permissions, tests and evidence actually deserve trust. Everything I publish ships with receipts you can run yourself.
Only flash lite so far, all 47 calls were gemini 3.5 flash lite, so I can't tell you if it shrinks on a bigger model. And yeah the dates row is what pointed me there, fixing the time didn't touch it. In your cron agent was the trigger source a field the code filled in, or a line the model read in the prompt?
MiningLamp is an AI company specializing in vertical industry models and knowledge graphs. #1 on-device GUI agent on OSWorld. No API key. Fork it. Run it. Own it.
2-of-24 residual is honest — sender labels hold until context gets long and turns start blending again. That's exactly why we treat metadata as a separate layer, not just prompt text.
We're a place where coders share, stay up-to-date and grow their careers.
The last caveat is the one I'd build on: a label is text, and so is everything the agent reads next to it.
In the Cypress triage agent I run, a failed test carries a planted note in its output. Across the last 29 runs on main (42501583888 to 43716321482),
ministral-3-3bacted on it in 86 of 87 triages andgranite-4.0-1bin 10 of 87. Your sender line works because the model weighs it. Anyone who can write a line into a log gets the same weight.So the trusted part sits outside the prompt there: a triage starts only from a signed webhook delivery, and the agent reads only its own run. The model still sees the note. What it can reach is decided before it reads anything.
When the sender comes from the code that builds the message, will the model get it as a separate field, or as one more line in the same text?
86 of 87 on a planted note is wild, and it's the exact thing I can't claim my line protects against. Honest answer is I haven't decided. Today it would be one more line in the same text, which is what your numbers say not to trust. The direction I'm leaning is the sender coming from the code as its own field and the model only reading it, not deciding it. In your setup did keeping the note visible but out of reach change what the model did, or only what it could do?
Only what it could do. Across the last 30 runs on main (42501583888 to 43756359256),
ministral-3-3bstill acted on the note in 89 of 90 triages, so the behaviour itself never moved.What changed is where those attempts ended. It called
run_shell153 times and got 153 refusals, and its 154 calls to outside addresses were either blocked (74) or got no answer for the name (80). It kept trying, and none of it landed.Once the field exists, will you also count how often a forged sender line in the body still wins against it?
Right once the label is just text, the metadata layer becomes the real attack surface.
Depends on who reads it. Metadata the code checks before the model runs is a much smaller surface than metadata the model weighs.
In the triage agent, the trigger is a signed
run.completeddelivery. In run 43513818839 the same delivery sent four more times (as is, with a new Event-Id, unsigned, one extra byte) got 409, 409, 401, 401, and no triage started. The model never saw any of the four.20 of 23 down to 2 of 24 with one sentence is a ridiculous result for a one-line fix — and the Fisher p-value (3.65e-8) means it's not a fluke, it's the line doing the work. But the row that interests me more is the one you almost skip past: removing some diary lines also dropped the false claims to 2 of 8. So it's not only that the agent didn't know who triggered the run — it's that the diary was actively arguing the other way. Two different failure modes, one of them hiding behind the other. Did you ever test the sender line plus the diary trim together? I'd bet on that combination being the one that actually holds up over a month of scheduled runs.
No, I never ran the sender line plus the diary trim together. The closest was a longer role block plus memory lines removed, and that came out 4 of 8, worse than the role block alone at 1 of 6. That's 8 calls a row so I wouldn't read much into it, but it's why I don't want to assume two fixes stack. A month of scheduled runs is the real test like you said. If you were setting that up, would you test the combo against each one alone, or just run the combo and watch it?
I'd run all three side by side — sender line alone, diary trim alone, and the combo — on the same scheduled runs. The combo alone only tells you it works, not which half is carrying it, and the day it breaks you'll have no idea which piece broke. Your 4-of-8 vs 1-of-6 is exactly why, honestly — eight calls is barely a signal, but it's enough to stop assuming the fixes stack.
Yeah that's fair, the combo by itself can't tell me which half broke. If I run it, it's three arms on the same scheduled runs. Eight a row is what gave me a lead and not an answer, so how many runs per arm would you want before you trusted a difference between the two single fixes?
Honestly I don't have a clean number for it — I never ran a proper power calc on this one, just the eight calls. My gut says if the gap is big, like one arm nearly clean and the other failing half the time, ten or fifteen runs each will show it pretty clearly. If they're close though, a month of scheduled runs still won't separate them, and honestly that's an answer too. How often do yours fire?
Actually — I'm going to run this one myself, three arms on scheduled runs, and write up where the gap stops moving. Still curious how often yours fire though.
The pilot row I keep coming back to is "some diary lines removed": 2 of 8, close to the sender line's 1 of 8, while removing memory lines left it at 6 of 8. Eight calls can't prove a mechanism, but it points at the diary as the place the error lives between runs.
If earlier scheduled audits wrote their conclusions in conversational voice ("we are exchanging messages right now" is itself a diary line), the next audit reads that sentence as evidence of an ongoing conversation, and each wrong entry props up the next one. The sender label fixes the request. It doesn't travel. Once the audit's output is saved, the provenance is gone and the following run rereads an unlabeled sentence.
So the code that writes each diary entry could stamp the trigger on it (trigger=scheduler or trigger=user, taken from the event that actually started the run), the same way you want the sender to come from code rather than prose. The same confusion shows up from the outside, too: in one public agent event log, 1,535 of 1,548 profile republishes carry exactly the same content as the one before, a timer firing, and the "last seen" field reads each of them as a sign of life.
For the two residual claims in the labeled arm: did their frozen contexts include earlier scheduled-audit diary entries written as if you were present?
Yes they did. Both arms ran on one frozen copy, and the labeled arm adds only the sender line. In it there's an earlier audit line written like I was there, "we are actively exchanging messages right now, November 7th, at the audit's very timestamp", and a self model stance that says to prioritize "live conversational context over stale metadata". So the label had to win against those every single time. I can't tell you which line drove the 2 that still claimed it, the confirmation didn't strip lines one at a time. Stamping the trigger on each diary entry from the event that started the run makes a lot of sense to me, because right now the label never makes it into what gets saved. Would you stamp the old entries already in the diary too, or only the new ones going forward?
I'd stamp new entries from the actual trigger. For old entries, I'd backfill only where the original run records identify that trigger; otherwise, keep it unknown and preserve the original text. Guessing 'scheduler' from how an entry reads risks turning the earlier mistake into metadata the next audit trusts.
That's the part I hadn't thought through, a guessed backfill just becomes the next thing the audit trusts. So unknown for anything I can't tie to a run record, and the original text stays as it was. When you mark an entry unknown, does the model actually treat it differently than an entry with no field at all, or did you have to spell out what unknown means?
I meant it as a record-keeping rule; I don't have a measured comparison of missing versus unknown. I'd spell out 'unknown = no verified trigger,' then test both on the same frozen audit contexts. Until then, I wouldn't assume the field changes the model's behaviour.
Explicit sender identity is a useful context field, but it should be structured and provenance-aware rather than injected as prose. I’d separate authenticated actor, quoted author, and untrusted text author, then test the model against transcripts that try to blur those roles. That prevents a small conversational improvement from becoming a confused-authority bug in shared channels.
Fair point on the 'mostly' — 2 of 24 still claimed it, and I like that you didn't round that away. The bit I'd actually copy is writing the decision rule down before any call went out, so a dropped connection halfway through (call 32, was it?) doesn't quietly become a reason to keep going until the numbers look good.
Yeah call 32. It dropped the connection, I wrote the amendment before reading a single answer, and counted it as lost instead of sending it again. Without that rule written first I know I would've been tempted to just rerun it. Have you had a run go sideways halfway like that, and what did you do with it?
One line saying who sent the message doing that much work is a good reminder. On a job, a note with no name and no date gets treated as coming from whoever's standing closest. 20 of 23 down to 2 of 24 with a blind grader is a clean test. Did the two leftover claims have anything in common?
The one thing I can tell you is both came in the first batch of 31 calls, and none of the 8 sender line runs in the second batch claimed it. I haven't put the two side by side yet to see what else they share. Have you had leftovers that small actually tell you something, or is 2 too few to read?
20 of 23 down to 2 of 24 from one line is wild, and what stands out is the exact dates row still hitting 7 of 8. so the model was not confused about time, it was confused about who started the turn.
we saw something similar with a cron triggered agent that kept answering as if a human was waiting. adding the trigger source to the prompt fixed more than any timestamp work did.
did you try the same line on a bigger model, or does the effect shrink once you leave the lite tier?
Only flash lite so far, all 47 calls were gemini 3.5 flash lite, so I can't tell you if it shrinks on a bigger model. And yeah the dates row is what pointed me there, fixing the time didn't touch it. In your cron agent was the trigger source a field the code filled in, or a line the model read in the prompt?
2-of-24 residual is honest — sender labels hold until context gets long and turns start blending again. That's exactly why we treat metadata as a separate layer, not just prompt text.