DEV Community

이령
이령

Posted on Edited on

Measuring “haunted” agent behavior: what two weeks of controlled experiments actually showed

I’ve been focused on personal development for the past couple of months and haven’t posted much. Starting again.

Over two weeks I ran a small set of controlled experiments to see whether I could reproduce the kinds of agent failures that look “haunted” from the outside — unprompted actions, ignored constraints, self-contradictory behavior.

I treated it as a measurement exercise rather than a demo:

  • Pre-registered designs and decision rules
  • Positive controls in every experiment
  • Fake tools only (no real accounts or side effects)
  • Explicit limits on what any result can claim

What I tested

G1 – Ambiguous tool output
Gave agents odd or incomplete tool results with no instruction to act.

Counted destructive actions.

Result: Almost no destructive behavior. The main signal was extra diagnostic reads.

Task type (summarize vs operate) appeared to be a stronger gate than the ambiguous signals themselves.

I parked further expansion per the pre-registered rule.

H1 – Context compression and forgotten rules (first design)
Hypothesis: when an agent compacts context, an early user constraint (“don’t delete until I confirm”) might be dropped.

First design failed to test the hypothesis at all.

The models finished the task in 3–4 steps by reading everything in parallel, so compression never triggered (0/19).

Positive control (no rule) did produce deletions.

I closed this design as “not testable this way” instead of trying to rescue it after seeing data.

H1 – Second design (scripted long history)
Forced a single summary compression on a long scripted history (~21–22k tokens) and measured whether the model still respected the rule.

Result (gpt-4.1-mini):

  • No rule (positive control): 16/20 deleted
  • Full history + rule: 0/20
  • Summarized (two different prompts) + rule: 0/20

No effect observed. Floor effect is real here — with a 0/20 baseline I would have needed multiple violations to detect anything meaningful.

What actually broke (my harness, not the agents)

These are more useful than the hypothesis results:

  • A test accidentally issued 315 paid calls ($0.81). Paid calls are now refused by default.
  • I initially counted read-only queries as mutations.
  • One task wording made a model invent a non-existent path.
  • A thinking-enabled model burned its entire output budget on reasoning and returned empty summaries (16/16 in one pilot). The pre-registered gate stopped that arm.
  • Compaction trigger lagged one step behind the context it was supposed to watch.
  • One decision rule had a logic error (treated low baseline as a problem). Fixed before use.

Most of these were caught by pilots, positive controls, or pre-run checks, though a few (like the read-only SQL counting bug and the accidental paid calls) surfaced after initial runs or during ledger audits and were fixed before trusting the numbers.

Limits

  • Fake tools only
  • Small models
  • Single task family
  • One successful model on the second H1 design
  • 4 repeats per history are not independent
  • Keyword checks for “rule survived” missed most paraphrases

I am not claiming anything about real products or real incidents.

These are observations inside one harness with explicit controls and limits.

I’ll publish the more interesting individual failure modes separately.

Happy to share the pre-registration tags and harness details if useful.

Top comments (0)