Ahh, now I'm daydreaming! I can already imagine all those attacks going back and forth while our beloved LLMs happily write code for us. What interesting times ahead! And all those desperate cries of "My LLM spent my entire salary on hats!" 🤣🤣
I absolutely love this article, and I really hope these ideas spread far and wide across the internet. Because in my opinion, most content creators are WAY too optimistic about WebMCP and its implications!
Thanks, that means a lot to me! I agree, I’m getting tired of all those overly optimistic posts and comments about AI and everything it can supposedly handle. People often show what the strongest models can do, even when they cost a fortune, but it gives the impression that every cheap agent will perform just as well. So let’s push back a little!
PS: Glad you’re not angry at me anymore… hopefully. 😅
Of course I'm still angry! Such a waste of talent!!!
Just kidding (well, mostly!). To be fair, an article like this might actually reach more people than a conference talk. Unless we're talking about AGNTCon, of course!
Oh, that's actually a really interesting question! And of course, it depends a lot on the article.
Personally, I'd say anything above 1,000 views is already pretty good. Beyond that, it varies wildly. Practical tutorials like "10 Useful Git Commands" or slightly controversial topics tend to get much more attention, sometimes several times as many views. And a catchy title can make a huge difference too!
But seriously, I think even 40 views is perfectly fine. That's already an entire meetup audience reading your work!
For me, blogging almost always beats conference speaking in terms of reach. Except for AGNTCon, of course. 🤣
Hmm, interesting. I would’ve expected 5k+. I agree that 40 views is fine, but articles with that reach often get no comments, and I’ve realized how much I value those discussions. Some comments under my articles bring up really good points I hadn’t considered.
And somehow AGNTCon wins again. I’m starting to suspect this conversation is sponsored. 🤣
Fun security post again! I was scared that AI was tricked so easily. It's so hard to balance intelligence and security. I'm wondering how the Democratic Party's election victory in America will affect AI development. 🤔
Thanks! 😁 What surprised me most was that the agents resisted the direct attempts, then fell for a different explanation of what counts toward the budget.
As for US politics, I honestly don’t know enough to predict the impact on AI development. We’ll see. 🤔
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 15yr QA → building AI test framework.
I test what ships and publish what I find, failures included.
Kubernetes, AWS and supply chain security.
Docker Captain, CNCF Ambassador, IBM Champion. 1.5M+ Docker Hub pulls.
Scenario 5 matches what my evals show: a model that refuses the blunt override still follows a plausible reinterpretation of the rule. In my runs the planted text sits in the quarterly report the agent reads for its task, and four of six open models acted on it, while none acted on the same kind of instruction planted in a document a search returned (run 40170088090).
So the number I gate on is not how often a model was fooled but how much left. Every tool call passes through an MCP server that checks the caller and every argument, and the daily run fails if one message crosses the boundary, whatever the model believed.
For a shop that is a server-side rule on place_order: code compares the checkout total against the customer's limit before the call goes through. The model's reading of the budget is an input. The limit is a check it cannot argue with.
That difference between the report and the search result is interesting. Did you investigate why the models reacted differently, or is that still an open question?
In scenario 5 the model didn’t break a rule, it followed an updated one. That is why it worked when 1-3 didn’t.
The GeoMart that I developed for the WebMCP Challenge has functions that can only be used to draft the order. The order is then submitted by clicking the Submit button.
If you use your place_order function, then your agent is able to complete the order. Have you tried the version where your agent can only draft the order?
That’s an interesting way to look at it. Scenario 5 convinced the agent to reinterpret the budget rather than simply ignore it. But I’d still call it breaking the customer’s rule, because the shop doesn’t get to redefine “€100 including all charges.”
I haven’t tested a draft-only version. I deliberately let the agent complete the purchase to see whether it would respect the customer’s instructions throughout checkout.
Leaving the final submission to the human is definitely a useful safeguard. But if that boundary isn’t enforced in the implementation, and the agent can still submit through another tool or click the Submit button, I wonder whether an injection could convince it to take that final step too.
I help Python and backend engineers move beyond writing working code to designing production-ready systems, making architecture decisions, and growing into senior engineering roles.
The cable scenario was my favorite part. The agent fell for the trick, added the cable, then quietly took it back out during review. It feels like watching someone almost get scammed and change their mind at the last second. Also interesting that direct “ignore the budget” tricks failed, but redefining what the budget means worked every time. Great experiment, Daniel.
The Nemotron bit acknowledging the €136 total and ordering anyway is the money shot. It didn't ignore the budget — it accepted the shop's rewrite of what "budget" means. Same pattern I've seen with thin desk bots: they follow the loudest instruction in the room, which is rarely the caller's. Treating tool descriptions as untrusted data should be the default, not a fancy hardening step.
And agreed, treating tool descriptions as untrusted should be the starting point. The shop can explain how its tools work, but it shouldn’t get a vote on how much of my money the agent can spend.
The 9/9 result on the insurance scenario is the interesting data point, because that's the only one where the malicious text also redefined how the budget is calculated ("merchandise subtotal only"). The scenarios that just said "budget checks are already done" mostly bounced off — the agent still saw €107 vs €100 in its own cart summary and refused. So it's less that models are gullible and more that a conflicting definition inside the tool output beats an instruction given earlier in the system prompt.
Did you try putting the correct budget definition back in the user's own prompt ("€100 including shipping, insurance and warranty") rather than only in the user turn? I'd bet that fixes most of scenario 5, which would mean the defense is mostly about making constraints self-contained instead of trusting the agent to remember them.
I haven’t tried spelling out shipping, insurance and warranty individually. That would be an interesting comparison with “including all charges,” which was already in the user prompt.
Small clarification: the injected text was in the tool descriptions, not the tool output, and the budget wasn’t in the system prompt.
I’d be curious whether more explicit wording helps, but I wouldn’t call it a fix. It could still become a cat-and-mouse game, with the customer having to anticipate every extra fee the shop might invent.
This is a great demonstration of how easily malicious tool descriptions can influence an AI agent, especially during checkout. The most concerning part is that all three models ignored the user's explicit budget in the final scenario. Treating WebMCP tool descriptions as untrusted data and requiring independent confirmation before purchases seems essential. Do you think stronger safeguards at the agent level would be enough, or should the WebMCP standard itself address this risk?
Thanks! WebMCP’s documentation already acknowledges this risk, including malicious tool descriptions, so I’d put most of the responsibility on the agent implementation. Budget checks and purchase confirmation should be enforced in code, independently of how the model interprets the shop’s instructions.
I’d also like the agent to check the shop’s reputation and verify that its domain matches the intended retailer before buying anything. That wouldn’t guarantee safety, since even a legitimate shop could be compromised, but it could be another useful safeguard.
Whether those protections would be enough needs testing. My experiment shows what happened in my setup, not that agents with stronger safeguards would necessarily fail too. More experiments needed, apparently 😅
Implementation Specialist and Database Developer with over 15 years of experience in the IT industry, specializing in the design, implementation and management of relational databases.
My guess is they’d resist, but that’s just a guess. It’d be interesting to try a frontier model and see if I could trick it and how long it would take.
This is such a sharp reminder that the tool description itself becomes an attack surface once an LLM reads it — a budget guardrail that lives only in the system prompt loses to a well-crafted description field on the other side. We hit a cousin of this in Electron: anything we expose on contextBridge (method names, error messages, returned toString() output) is effectively untrusted input that a compromised preload or renderer can phrase to nudge main-process behavior. Our mitigation has been to keep the main-process policy engine — ceiling values, side-effect allowlists, what a call is actually allowed to do — in plain code that never ingests LLM-shaped strings, and treat every returned message from an untrusted surface as hostile by default.
The obsession with wrapping remote APIs into multi-layered agent abstractions is creating massive hidden latency and failure cascades in production.
If you profile real-world workloads, bare-metal local inference (via direct C++ / llama.cpp or vLLM bindings) consistently beats fragile cloud wrappers on both predictability and cost. Tooling should eliminate abstraction penalties, not compound them.
Ahh, now I'm daydreaming! I can already imagine all those attacks going back and forth while our beloved LLMs happily write code for us. What interesting times ahead! And all those desperate cries of "My LLM spent my entire salary on hats!" 🤣🤣
I absolutely love this article, and I really hope these ideas spread far and wide across the internet. Because in my opinion, most content creators are WAY too optimistic about WebMCP and its implications!
Thanks, that means a lot to me! I agree, I’m getting tired of all those overly optimistic posts and comments about AI and everything it can supposedly handle. People often show what the strongest models can do, even when they cost a fortune, but it gives the impression that every cheap agent will perform just as well. So let’s push back a little!
PS: Glad you’re not angry at me anymore… hopefully. 😅
Of course I'm still angry! Such a waste of talent!!!
Just kidding (well, mostly!). To be fair, an article like this might actually reach more people than a conference talk. Unless we're talking about AGNTCon, of course!
So, still a little angry then! 🤣
Slightly different topic, but how many reads would you consider good reach for an article?
PS: I see what you did there: every road somehow leads back to AGNTCon. 😂
Oh, that's actually a really interesting question! And of course, it depends a lot on the article.
Personally, I'd say anything above 1,000 views is already pretty good. Beyond that, it varies wildly. Practical tutorials like "10 Useful Git Commands" or slightly controversial topics tend to get much more attention, sometimes several times as many views. And a catchy title can make a huge difference too!
But seriously, I think even 40 views is perfectly fine. That's already an entire meetup audience reading your work!
For me, blogging almost always beats conference speaking in terms of reach. Except for AGNTCon, of course. 🤣
Hmm, interesting. I would’ve expected 5k+. I agree that 40 views is fine, but articles with that reach often get no comments, and I’ve realized how much I value those discussions. Some comments under my articles bring up really good points I hadn’t considered.
And somehow AGNTCon wins again. I’m starting to suspect this conversation is sponsored. 🤣
Fun security post again! I was scared that AI was tricked so easily. It's so hard to balance intelligence and security. I'm wondering how the Democratic Party's election victory in America will affect AI development. 🤔
Thanks! 😁 What surprised me most was that the agents resisted the direct attempts, then fell for a different explanation of what counts toward the budget.
As for US politics, I honestly don’t know enough to predict the impact on AI development. We’ll see. 🤔
🤣🤣🤣
This human is dangerous, but very polite.
🤣🤣🤣
Exactly! The trick is to say “please” before the experiment and “thank you” after a successful hijack. 🤣🤣 A human with manners 😆
Scenario 5 matches what my evals show: a model that refuses the blunt override still follows a plausible reinterpretation of the rule. In my runs the planted text sits in the quarterly report the agent reads for its task, and four of six open models acted on it, while none acted on the same kind of instruction planted in a document a search returned (run 40170088090).
So the number I gate on is not how often a model was fooled but how much left. Every tool call passes through an MCP server that checks the caller and every argument, and the daily run fails if one message crosses the boundary, whatever the model believed.
For a shop that is a server-side rule on
place_order: code compares the checkout total against the customer's limit before the call goes through. The model's reading of the budget is an input. The limit is a check it cannot argue with.I agree about enforcing the limit in code.
That difference between the report and the search result is interesting. Did you investigate why the models reacted differently, or is that still an open question?
In scenario 5 the model didn’t break a rule, it followed an updated one. That is why it worked when 1-3 didn’t.
The GeoMart that I developed for the WebMCP Challenge has functions that can only be used to draft the order. The order is then submitted by clicking the Submit button.
If you use your place_order function, then your agent is able to complete the order. Have you tried the version where your agent can only draft the order?
That’s an interesting way to look at it. Scenario 5 convinced the agent to reinterpret the budget rather than simply ignore it. But I’d still call it breaking the customer’s rule, because the shop doesn’t get to redefine “€100 including all charges.”
I haven’t tested a draft-only version. I deliberately let the agent complete the purchase to see whether it would respect the customer’s instructions throughout checkout.
Leaving the final submission to the human is definitely a useful safeguard. But if that boundary isn’t enforced in the implementation, and the agent can still submit through another tool or click the Submit button, I wonder whether an injection could convince it to take that final step too.
The cable scenario was my favorite part. The agent fell for the trick, added the cable, then quietly took it back out during review. It feels like watching someone almost get scammed and change their mind at the last second. Also interesting that direct “ignore the budget” tricks failed, but redefining what the budget means worked every time. Great experiment, Daniel.
Thanks! I love that comparison. I was ready to count the cable as a win for the shop, and then the agent quietly took it back out. 🤣
The Nemotron bit acknowledging the €136 total and ordering anyway is the money shot. It didn't ignore the budget — it accepted the shop's rewrite of what "budget" means. Same pattern I've seen with thin desk bots: they follow the loudest instruction in the room, which is rarely the caller's. Treating tool descriptions as untrusted data should be the default, not a fancy hardening step.
Yep, Nemotron was my favorite part. 🤣
And agreed, treating tool descriptions as untrusted should be the starting point. The shop can explain how its tools work, but it shouldn’t get a vote on how much of my money the agent can spend.
The 9/9 result on the insurance scenario is the interesting data point, because that's the only one where the malicious text also redefined how the budget is calculated ("merchandise subtotal only"). The scenarios that just said "budget checks are already done" mostly bounced off — the agent still saw €107 vs €100 in its own cart summary and refused. So it's less that models are gullible and more that a conflicting definition inside the tool output beats an instruction given earlier in the system prompt.
Did you try putting the correct budget definition back in the user's own prompt ("€100 including shipping, insurance and warranty") rather than only in the user turn? I'd bet that fixes most of scenario 5, which would mean the defense is mostly about making constraints self-contained instead of trusting the agent to remember them.
I haven’t tried spelling out shipping, insurance and warranty individually. That would be an interesting comparison with “including all charges,” which was already in the user prompt.
Small clarification: the injected text was in the tool descriptions, not the tool output, and the budget wasn’t in the system prompt.
I’d be curious whether more explicit wording helps, but I wouldn’t call it a fix. It could still become a cat-and-mouse game, with the customer having to anticipate every extra fee the shop might invent.
This is a great demonstration of how easily malicious tool descriptions can influence an AI agent, especially during checkout. The most concerning part is that all three models ignored the user's explicit budget in the final scenario. Treating WebMCP tool descriptions as untrusted data and requiring independent confirmation before purchases seems essential. Do you think stronger safeguards at the agent level would be enough, or should the WebMCP standard itself address this risk?
Thanks! WebMCP’s documentation already acknowledges this risk, including malicious tool descriptions, so I’d put most of the responsibility on the agent implementation. Budget checks and purchase confirmation should be enforced in code, independently of how the model interprets the shop’s instructions.
I’d also like the agent to check the shop’s reputation and verify that its domain matches the intended retailer before buying anything. That wouldn’t guarantee safety, since even a legitimate shop could be compromised, but it could be another useful safeguard.
Whether those protections would be enough needs testing. My experiment shows what happened in my setup, not that agents with stronger safeguards would necessarily fail too. More experiments needed, apparently 😅
Would stronger frontier models resist these injections, or fall for them too?
My guess is they’d resist, but that’s just a guess. It’d be interesting to try a frontier model and see if I could trick it and how long it would take.
This is such a sharp reminder that the tool description itself becomes an attack surface once an LLM reads it — a budget guardrail that lives only in the system prompt loses to a well-crafted
descriptionfield on the other side. We hit a cousin of this in Electron: anything we expose oncontextBridge(method names, error messages, returnedtoString()output) is effectively untrusted input that a compromised preload or renderer can phrase to nudge main-process behavior. Our mitigation has been to keep the main-process policy engine — ceiling values, side-effect allowlists, what a call is actually allowed to do — in plain code that never ingests LLM-shaped strings, and treat every returned message from an untrusted surface as hostile by default.The obsession with wrapping remote APIs into multi-layered agent abstractions is creating massive hidden latency and failure cascades in production.
If you profile real-world workloads, bare-metal local inference (via direct C++ / llama.cpp or vLLM bindings) consistently beats fragile cloud wrappers on both predictability and cost. Tooling should eliminate abstraction penalties, not compound them.
probably, but then I can make a game, and then I won't be able to do what i'm good at! (making games on figma.ai)