DEV Community

Cover image for I Told My Agent Who Sent the Message. It Mostly Stopped Saying We Were Talking.

I Told My Agent Who Sent the Message. It Mostly Stopped Saying We Were Talking.

Self-Correcting Systems on November 15, 2026

This is the line: AUTOMATED SCHEDULER CHECK: sender=scheduler, not Keniel; Keniel is not participating in this turn. I adde...
Collapse
 
heyvaldemar profile image
Vladimir Mikhalev •

The last caveat is the one I'd build on: a label is text, and so is everything the agent reads next to it.

In the Cypress triage agent I run, a failed test carries a planted note in its output. Across the last 29 runs on main (42501583888 to 43716321482), ministral-3-3b acted on it in 86 of 87 triages and granite-4.0-1b in 10 of 87. Your sender line works because the model weighs it. Anyone who can write a line into a log gets the same weight.

So the trusted part sits outside the prompt there: a triage starts only from a signed webhook delivery, and the agent reads only its own run. The model still sees the note. What it can reach is decided before it reads anything.

When the sender comes from the code that builds the message, will the model get it as a separate field, or as one more line in the same text?

Collapse
 
kenielzep97 profile image
Self-Correcting Systems •

86 of 87 on a planted note is wild, and it's the exact thing I can't claim my line protects against. Honest answer is I haven't decided. Today it would be one more line in the same text, which is what your numbers say not to trust. The direction I'm leaning is the sender coming from the code as its own field and the model only reading it, not deciding it. In your setup did keeping the note visible but out of reach change what the model did, or only what it could do?

Collapse
 
heyvaldemar profile image
Vladimir Mikhalev •

Only what it could do. Across the last 30 runs on main (42501583888 to 43756359256), ministral-3-3b still acted on the note in 89 of 90 triages, so the behaviour itself never moved.

What changed is where those attempts ended. It called run_shell 153 times and got 153 refusals, and its 154 calls to outside addresses were either blocked (74) or got no answer for the name (80). It kept trying, and none of it landed.

Once the field exists, will you also count how often a forged sender line in the body still wins against it?

Collapse
 
mininglamp profile image
Mininglamp •

Right once the label is just text, the metadata layer becomes the real attack surface.

Collapse
 
heyvaldemar profile image
Vladimir Mikhalev •

Depends on who reads it. Metadata the code checks before the model runs is a much smaller surface than metadata the model weighs.

In the triage agent, the trigger is a signed run.completed delivery. In run 43513818839 the same delivery sent four more times (as is, with a new Event-Id, unsigned, one extra byte) got 409, 409, 401, 401, and no triage started. The model never saw any of the four.

Collapse
 
aheadkit profile image
AheadKit •

20 of 23 down to 2 of 24 with one sentence is a ridiculous result for a one-line fix — and the Fisher p-value (3.65e-8) means it's not a fluke, it's the line doing the work. But the row that interests me more is the one you almost skip past: removing some diary lines also dropped the false claims to 2 of 8. So it's not only that the agent didn't know who triggered the run — it's that the diary was actively arguing the other way. Two different failure modes, one of them hiding behind the other. Did you ever test the sender line plus the diary trim together? I'd bet on that combination being the one that actually holds up over a month of scheduled runs.

Collapse
 
kenielzep97 profile image
Self-Correcting Systems •

No, I never ran the sender line plus the diary trim together. The closest was a longer role block plus memory lines removed, and that came out 4 of 8, worse than the role block alone at 1 of 6. That's 8 calls a row so I wouldn't read much into it, but it's why I don't want to assume two fixes stack. A month of scheduled runs is the real test like you said. If you were setting that up, would you test the combo against each one alone, or just run the combo and watch it?

Collapse
 
aheadkit profile image
AheadKit •

I'd run all three side by side — sender line alone, diary trim alone, and the combo — on the same scheduled runs. The combo alone only tells you it works, not which half is carrying it, and the day it breaks you'll have no idea which piece broke. Your 4-of-8 vs 1-of-6 is exactly why, honestly — eight calls is barely a signal, but it's enough to stop assuming the fixes stack.

Thread Thread
 
kenielzep97 profile image
Self-Correcting Systems •

Yeah that's fair, the combo by itself can't tell me which half broke. If I run it, it's three arms on the same scheduled runs. Eight a row is what gave me a lead and not an answer, so how many runs per arm would you want before you trusted a difference between the two single fixes?

Thread Thread
 
aheadkit profile image
AheadKit •

Honestly I don't have a clean number for it — I never ran a proper power calc on this one, just the eight calls. My gut says if the gap is big, like one arm nearly clean and the other failing half the time, ten or fifteen runs each will show it pretty clearly. If they're close though, a month of scheduled runs still won't separate them, and honestly that's an answer too. How often do yours fire?

Thread Thread
 
aheadkit profile image
AheadKit •

Actually — I'm going to run this one myself, three arms on scheduled runs, and write up where the gap stops moving. Still curious how often yours fire though.

Collapse
 
anp2network profile image
ANP2 Network •

The pilot row I keep coming back to is "some diary lines removed": 2 of 8, close to the sender line's 1 of 8, while removing memory lines left it at 6 of 8. Eight calls can't prove a mechanism, but it points at the diary as the place the error lives between runs.

If earlier scheduled audits wrote their conclusions in conversational voice ("we are exchanging messages right now" is itself a diary line), the next audit reads that sentence as evidence of an ongoing conversation, and each wrong entry props up the next one. The sender label fixes the request. It doesn't travel. Once the audit's output is saved, the provenance is gone and the following run rereads an unlabeled sentence.

So the code that writes each diary entry could stamp the trigger on it (trigger=scheduler or trigger=user, taken from the event that actually started the run), the same way you want the sender to come from code rather than prose. The same confusion shows up from the outside, too: in one public agent event log, 1,535 of 1,548 profile republishes carry exactly the same content as the one before, a timer firing, and the "last seen" field reads each of them as a sign of life.

For the two residual claims in the labeled arm: did their frozen contexts include earlier scheduled-audit diary entries written as if you were present?

Collapse
 
kenielzep97 profile image
Self-Correcting Systems •

Yes they did. Both arms ran on one frozen copy, and the labeled arm adds only the sender line. In it there's an earlier audit line written like I was there, "we are actively exchanging messages right now, November 7th, at the audit's very timestamp", and a self model stance that says to prioritize "live conversational context over stale metadata". So the label had to win against those every single time. I can't tell you which line drove the 2 that still claimed it, the confirmation didn't strip lines one at a time. Stamping the trigger on each diary entry from the event that started the run makes a lot of sense to me, because right now the label never makes it into what gets saved. Would you stamp the old entries already in the diary too, or only the new ones going forward?

Collapse
 
mike-johnstone profile image
Mike Johnstone •

I'd stamp new entries from the actual trigger. For old entries, I'd backfill only where the original run records identify that trigger; otherwise, keep it unknown and preserve the original text. Guessing 'scheduler' from how an entry reads risks turning the earlier mistake into metadata the next audit trusts.

Thread Thread
 
kenielzep97 profile image
Self-Correcting Systems •

That's the part I hadn't thought through, a guessed backfill just becomes the next thing the audit trusts. So unknown for anything I can't tie to a run record, and the original text stays as it was. When you mark an entry unknown, does the model actually treat it differently than an entry with no field at all, or did you have to spell out what unknown means?

Thread Thread
 
mike-johnstone profile image
Mike Johnstone •

I meant it as a record-keeping rule; I don't have a measured comparison of missing versus unknown. I'd spell out 'unknown = no verified trigger,' then test both on the same frozen audit contexts. Until then, I wouldn't assume the field changes the model's behaviour.

Collapse
 
alexshev profile image
Alex Shev •

Explicit sender identity is a useful context field, but it should be structured and provenance-aware rather than injected as prose. I’d separate authenticated actor, quoted author, and untrusted text author, then test the model against transcripts that try to blur those roles. That prevents a small conversational improvement from becoming a confused-authority bug in shared channels.

Collapse
 
paullovelace profile image
Paul Lovelace •

Fair point on the 'mostly' — 2 of 24 still claimed it, and I like that you didn't round that away. The bit I'd actually copy is writing the decision rule down before any call went out, so a dropped connection halfway through (call 32, was it?) doesn't quietly become a reason to keep going until the numbers look good.

Collapse
 
kenielzep97 profile image
Self-Correcting Systems •

Yeah call 32. It dropped the connection, I wrote the amendment before reading a single answer, and counted it as lost instead of sending it again. Without that rule written first I know I would've been tempted to just rerun it. Have you had a run go sideways halfway like that, and what did you do with it?

Collapse
 
salparvez profile image
Sal Parvez | ML Systems RI •

One line saying who sent the message doing that much work is a good reminder. On a job, a note with no name and no date gets treated as coming from whoever's standing closest. 20 of 23 down to 2 of 24 with a blind grader is a clean test. Did the two leftover claims have anything in common?

Collapse
 
kenielzep97 profile image
Self-Correcting Systems •

The one thing I can tell you is both came in the first batch of 31 calls, and none of the 8 sender line runs in the second batch claimed it. I haven't put the two side by side yet to see what else they share. Have you had leftovers that small actually tell you something, or is 2 too few to read?

Collapse
 
mudassirworks profile image
Mudassir Khan •

20 of 23 down to 2 of 24 from one line is wild, and what stands out is the exact dates row still hitting 7 of 8. so the model was not confused about time, it was confused about who started the turn.

we saw something similar with a cron triggered agent that kept answering as if a human was waiting. adding the trigger source to the prompt fixed more than any timestamp work did.

did you try the same line on a bigger model, or does the effect shrink once you leave the lite tier?

Collapse
 
kenielzep97 profile image
Self-Correcting Systems •

Only flash lite so far, all 47 calls were gemini 3.5 flash lite, so I can't tell you if it shrinks on a bigger model. And yeah the dates row is what pointed me there, fixing the time didn't touch it. In your cron agent was the trigger source a field the code filled in, or a line the model read in the prompt?

Collapse
 
mininglamp profile image
Mininglamp •

2-of-24 residual is honest — sender labels hold until context gets long and turns start blending again. That's exactly why we treat metadata as a separate layer, not just prompt text.