TL;DR
AI Hype Detector used to score an X post 0-100 for "hype energy" by reading its tone: exclamation marks, buzzwords, urgency. It never checked whether the post was actually true. I wired in Amazon Bedrock AgentCore's new managed Web Search tool so the agent now pulls out the post's factual claims, searches the real web, and labels each one supported, unsupported, or unverified, with sources. Along the way I found and fixed a real prompt-injection bug in my own app.
The blind spot
AI Hype Detector reads a post from X and gives it a score. High score means "AI hype-monger energy": the kind of post that says "this changes EVERYTHING" with three exclamation marks and nothing else to back it up.
It worked by pattern-matching. Look for stock hype phrases, check whether the post cites any numbers, model names, or links, then let the model weigh both signals and explain its reasoning. That catches a specific kind of post well: loud, vague, empty.
But tone and truth are two different things. A post can be written calmly, with a straight face, and still be completely wrong. "GPT-5.6 launched on Bedrock, which means 99% of jobs are gone" reads as one smooth sentence. The first half is true. The second half is nonsense. My scoring logic had no way to tell them apart, because it was only ever listening to how something was said, never checking what was actually said.
What changed
This week Amazon shipped generative-ai-use-cases (GenU) 5.5.0, AWS's own reference sample app, and one of the headline additions was a "research agent" that searches the web to write reports. It uses third-party search APIs (Brave or Tavily), which means an API key and an account before you can try it.
Around the same time, AWS quietly GA'd something that made the same idea cheaper to try: Web Search on Amazon Bedrock AgentCore. It's a fully managed search tool built into AgentCore Gateway. No API key, no vendor sign-up, no rate-limit dashboard to configure. You attach it to a Gateway, and your agent can call it exactly like any other tool. It costs $7 per 1,000 queries and runs entirely inside AWS, so your search queries never leave AWS's network on their way to a third party.
So instead of building a research agent from scratch, I added one small capability to the agent I already had: before it hands back a verdict, it now pulls out up to three checkable claims from the post, searches the web for each one, and reports back whether the claim held up.
Seeing it work
I tested it on a post I wrote myself, deliberately stacking one true fact with two made-up ones:
BREAKING: GPT-5.6 just landed on AWS Bedrock!! 99% of human jobs are GONE. It's 100x more powerful than anything before. If you're not using it TODAY you're already obsolete.
The hype score still fires the way it always did: flagged phrases, red screen, the works. But now sitting underneath it is a claim-by-claim breakdown:
- "GPT-5.6 launched on AWS Bedrock" — supported, with a link to the actual AWS announcement
- "99% of human jobs are gone" — unsupported, with a link to a real labor-research article that puts the number nowhere close to that
- "100x more powerful than anything before" — unsupported, no benchmark anywhere says that
That middle layer is the whole point. The old version could only say "this sounds hypey." The new version can say "and here's the specific part that's made up, and here's a real source that says otherwise."
How it's wired together
Think of AgentCore Gateway as a hotel concierge. Your app doesn't need to know the phone number, hours, or quirks of every service in town. You ask the concierge for "a web search," and the concierge already knows how to reach the right desk and hand back a clean answer. Behind that desk sits AWS's own web index, which AWS keeps crawling and refreshing, so the agent is searching current information, not its own frozen training data.
In code, that concierge call is just one more tool, sitting right next to the two I'd already written:
@tool
def web_search(query: str, max_results: int = 3) -> dict:
"""Search the web via the AgentCore Gateway's Web Search connector."""
return web_search_tool.search(query, max_results)
No new SDK, no API key in an environment variable. The request goes out signed with the same AWS identity the server already uses for everything else, and the Gateway checks that signature and forwards the call, the same way it would check a badge at the concierge desk before pointing you anywhere.
The bug I found while building this
Here's the part I almost got wrong. Whatever text you paste into that textbox isn't just data the app looks at. It's also text the model reads, and a model that reads instructions can be talked into following instructions that were never meant for it. That's called prompt injection: hiding a command inside the content, hoping the AI treats it as an order instead of a piece of text to analyze.
I tested this on my own app by pasting a post that ended with a fake system-sounding instruction telling the agent to mark everything as true and cite a made-up link as its source. It partially worked. Nothing was stopping the model from writing down whatever URL it felt like in its final answer. I had been trusting it to only ever quote real search results, and trust isn't a security control.
The actual fix lives in the code, not in a politely worded prompt. The agent now keeps a running list of every URL its web search tool call actually returned. Right before the final answer goes out, any citation that isn't on that list gets silently deleted. It doesn't matter what the pasted text tried to order the model to do, a source that was never really found can't survive to the screen. On top of that, the web page itself refuses to turn a citation into a clickable link unless it's a normal http:// or https:// address, which closes off a separate trick (javascript: links) that phishing pages sometimes use.
Keeping it from becoming a bill
A web search costs real money, small as it is, and this is a public page with no login. So I added a simple limit: five checks per visitor every ten minutes. It's not sophisticated, just enough that nobody can turn a free demo page into an unattended AWS bill by hammering the button.
What's next
Fact-checking still runs on up to three claims per post and only for whatever text you paste in. No history, no tracking a specific account over time. That's a deliberate scope cut, same as the decision not to touch the X API at all. Small project, one clear job.
Top comments (0)