This week's Hacktoberfest theme is Ghost in the Machine. Unfortunately, I don't have enough time to prepare a full challenge submission, but the th...
For further actions, you may consider blocking this person and/or reporting abuse
Very fun post as always! 🎃 But I hope you don't go too far to the dark side and become a black-hat hacker. On the other hand, please use your great skills as a white-hat hacker to save the world!
but… the dark side has cookies! 🍪
Such a fun read! Social engineering against AI, haha. I tried similar tricks on our company’s agent platform a while back and it actually worked. I bet any AI crawling this post will flag you as a high-risk human. Watch out once Skynet boots up 😂
Thanks! Nice, did they improve their platform after your attacks? 😁
Oh no, do you think it'll help if I start saying "please" and "thank you" whenever I chat with AI? 🤣
They added a few small mitigations, but these attack vectors are really hard to eliminate completely 😂. Being polite won’t protect us from Skynet, we’re both already on its watch‑list!🤣🤣🤣
But but but ...
🤣🤣
🤣🤣🤣hahaha
The honest caveat is the most important part — it took four or five attempts to land the injection, so this is an iterated attack, not a single-shot one. That actually makes it scarier for log pipelines, since an attacker who can plant entries can also iterate. Did you test whether the injection survives when the analyst prompt demands structured output, or does forcing a fixed schema blunt it?
Nope, I didn't test structured output. I deliberately kept the analyst prompt simple, as this was just a small Halloween experiment, not a security benchmark. 😁
But that's an interesting idea, I suspect enforcing a schema could make this particular attack harder, although it wouldn't necessarily prevent prompt injection entirely.
That's the honest scope — a schema narrows what the model can emit, but it doesn't clean what it reads, so an entry crafted to fit the fields could still steer the values inside them. The test I'd want is whether the injection survives as field content rather than as narrative — if it does, the fix has to sit before the prompt, in how log entries are validated or quarantined. Did the iterated attempts converge on a particular entry shape, or did each success look different?
so much fun bro, I find it interesting at the point where they fell for that trap, so ai are not really perfect afterall
Nope, not yet. 😁 From my experience, affordable models still require a lot of back-and-forth to get things right, while the best models are often too expensive for everyday use.
Nice experiment! Your service wrote the log, but upstream wrote the 503 body inside it, and four models apparently decided the stranger shouting orders was their boss. 👻
The delivery difference in the Codex section matters for interpreting the split result. The payload depends on the embedded newlines in conversation_handoff turning into apparent turn boundaries, so a useful control would be to run the same hijacked open-weight models with the log supplied as a file, and Codex with the log pasted inline. A second control worth testing is JSON-escaping the response body before analysis, so the fake USER and ASSISTANT markers stay inside a single string value instead of starting new lines. Would either change which models cross from data into instruction?
This is a useful reminder that every tool which reads untrusted artifacts needs a data/code boundary. For an agent workflow, I would preserve the raw log as evidence but pass a separately quoted, length-bounded representation to the model, with instructions that artifact text cannot alter its authority or task.
The fake handoff is what makes this more than a curiosity: it does not ask the model to ignore instructions, it rewrites who is speaking. That suggests the fix should sit before the prompt, by treating response bodies inside logs as untrusted data and stripping or quarantining fields that mimic conversation structure. Did you try a version where those fields are flattened into plain text first, and would that have changed the outcome for the models that were hijacked?
Treating logs as untrusted input is the necessary start, but I’d test the full path too: ingestion, summarization, retrieval, and tool prompting. A canary in every hop makes it clear where attacker-controlled text gains authority.
The iterated bit is what stuck with me — four or five attempts to land it makes this scarier, not less, for log pipelines, since an attacker who can plant entries can iterate too. And the fake handoff inside a 503 body is a neat detail: it doesn't shout about ignoring instructions, it just pretends the conversation already ended. Do you think the Gemini/Codex resistance says anything at all, or is it mostly that they saw the file differently from the raw-prompt models?
A fake handoff hidden in a 503 body worries me more for agents than for log analysis. When my agents call a flaky API, that error body comes back as a tool result, so the fake user turn reaches the model with no log file around it. I'd be curious whether the four that answered GHOST IN THE SHELL hold up any better when it arrives that way.
Burying the fake handoff inside a 503 body is a nasty detail. Most log readers would skim right past it.
))
The attack is simple, but the implications aren't. Once an agent can read files and take actions, every file it can access deserves a little more suspicion.
All 4 open weight models falling for a trap buried in 300 log entries still leaves the comparison messy, because Codex read the log as a file instead of getting it pasted through an API call.