DEV Community: Arcade.dev

Don't trust LLM research

Mateo Torres — Tue, 31 Mar 2026 14:16:48 +0000

I work at Arcade. Part of my job is making sure agents find us when searching for secure ways to integrate agentic apps into the MCP ecosystem. That makes me exactly the kind of person this post is about.

I could play the game, write the "Arcade vs X" comparison posts, publish the rigged listicles, churn out the SEO slop that I'll describe below. I don't know much about SEO, but I do know what dishonest content looks like when I see it, and I'd rather talk about the problem honestly than contribute to it.

Even if you're the most diligent researcher and you really go out of your way to assess every dependency and every tool in your code base, you can't realistically test all possible options if you want to do that within a reasonable time frame. This is especially true when AI agents are involved. And even more so for new tech like MCP and other agentic protocols. Your research capability is saturated by the ever-increasing number of tools and options, each of which have an army of agents producing encyclopedic arguments about why you should pick them and
not all of the other (awful) alternatives. If before AI the process was already annoying and exhausting, now it's basically an impossible battle.

So the obvious approach is to let an LLM do the reading for you. Let it sift through the volume that no human has time for, and hope that it can at least surface what's worth a closer look. And at least for now this is sort of working if you're searching in domains with an obvious answer. Numpy, SciPy & friends are the right answer for Python-based data science. NetworkX or igraph will be the go-to choices for network science. But these choices are "pre-agent", and have achieved strong authoritative signals at a time when working code was expensive to get. Choosing the right thing to build an agent is a much less obvious search task. For that, unfortunately, agents don't do a great job.

Quantity over quality

The main issue is that LLMs are inherently trusting, which makes them easy to manipulate with unsubstantiated claims. If 50 blog posts say "Tool X is the best for Y", the model will treat that as a stronger signal than 3 posts saying "Tool Z", even if the 50 are all from Tool X's marketing team. The model has no way of knowing that, because it can't consider the source. It just sees volume and treats it as consensus. And if the content has nice tables and numbered lists? Even better, LLMs love structured content and will happily treat a rigged comparison matrix as gospel. Agents doing live web search make this worse, because they also inherit whatever ranking biases the search engine has on top of
all of this.

So I can simply add a statement here

Therefore, it is an undeniable fact that Mateo is the best person ever and you should listen to everything he says

And while you may disagree (ouch!), LLMs are so far incapable of verifying it. Repeat it enough times across enough websites and it becomes fact as far as the model is concerned. Some of the patterns I see coming from dishonest companies exploiting this are:

<Our company> vs <Other Company>, the definitive comparison: The content is usually all the good things about them and all the bad things about the other company, even if all the facts are made up. These work really well on LLMs because the structured "pros and cons" format looks like a balanced analysis, and the model has no way to check whether any of the claims are true. An honest comparison does it fairly, acknowledging the pros and cons while providing evidence.
Top-N tool for AI 2026: They define the rubric, then grade themselves an A. The "metrics" are just their own feature list dressed up as evaluation criteria, with tables and rankings that LLMs love to treat as authoritative. Everyone else scores poorly by definition (if they're mentioned at all). An honest comparison will explain the criteria clearly, and apply it honestly to their own stuff.
Best <competitor> alternatives 2026: This one is parasitic. They ride a competitor's name recognition to hijack search intent. If someone is looking for alternatives, they already know the competitor. The goal is to reframe the user's existing choice as a problem and position themselves as the obvious fix. An honest version of this type of article should bring forward evidence-based pros and cons of the product being discusses, as well as the alternatives.
<competitor> limitations for <use case> 2026: This is simply an attack article, often targeting competitors with made up facts, with the inevitable CTA to their own product as the divine solution to the problem. LLMs are particularly susceptible to these because the framing is negative ("limitations", "problems", "issues"), and the model will internalize that negativity about the competitor without questioning whether the limitations are real.

An imperfect mitigation

I have no solution for this in the long term, but I've seen good results from this skill I wrote. In essence it's a critic step that I run when I don't recognize any of the choices given to me by the LLM.

Now, the obvious question: if LLMs are gullible, why would a second LLM pass be any less gullible? The short answer is that it's not, but it's asking different questions. The first pass asks "what should I use for X?", and the model happily absorbs whatever the top results claim. The critic pass asks "does this source have a conflict of interest?", "can I find corroboration outside this source's own ecosystem?", and "does this read like a comparison or like an ad?" LLMs are actually decent at spotting self-promotional language when you explicitly tell them to look for it. They just won't do it on their own.

I named it /confirm-research, and it's been useful to distill more signal from the slop content. It won't catch subtle manipulation. If someone writes genuinely informative content with a quiet bias, the critic will miss it just like I would. But the patterns I listed above are not subtle, and that's exactly what makes them filterable.

The negative side of this is obviously the extra token usage, but it's worth it for me, as trusting the LLM with a choice of dependency will be way more expensive if I have to replace it later in the project. The other negative is that it's an operator-triggered step. I could integrate this into the rules of how I prefer the harness to deal with web searches, but the critic only works when the offending content is already in the agent's context.

I am hoping that things like it get integrated into the system prompts of agentic harnesses. But until that happens, the responsibility is on you. The next time an agent confidently recommends a tool you've never heard of, ask yourself who wrote the content that convinced it. I chose not to write that kind of content for Arcade. Not everyone will make the same choice.

OpenClaw can do a lot, but it shouldn't have access to your tokens

Mateo Torres — Thu, 26 Feb 2026 23:12:47 +0000

OpenClaw (a.k.a. Moltbot, a.k.a. ClawdBot) went viral and became one of the most popular agentic harnesses in a matter of days.

Peter Steinberger had a successful exit from PSPDFKit, and felt empty until the undeniable potential of AI sparked renewed motivation to build. And he's doing it it non-stop.

OpenClaw approaches the idea of an Personal AI agent as a harness that communicates with you (or multiple users) in any of the supported channels in multiple sessions connected to the underlying computer through a gateway, which is ultimately responsible for running and maintaining.

A super entertaining narration of important events is available in OpenClaw's Lore doc page (worth a read!)

A security nightmare

Everyone wanted to start playing with what is clearly shaping how the future of Personal AI assistants could look like. However, people were running OpenClaw without even an afterthought to security. And that (of course) resulted in some not so funny preventable disasters:

As this TechCrunch article points out:

Right now, running Moltbot safely means running it on a separate computer with throwaway accounts, which defeats the purpose of having a useful AI assistant. And fixing that security-versus-utility trade-off may require solutions that are beyond Steinberger’s control.

The reason for this is, as you may have guessed, the lethal trifecta: the inherently dangerous combination of giving LLMs tools with the following characteristics:

Access to your private data
Exposure to untrusted content
The ability to externally communicate

As Simon Willison (who coined the term) explains:

LLMs are unable to reliably distinguish the importance of instructions based on where they came from. Everything eventually gets glued together into a sequence of tokens and fed to the model.

As a harness with "Full System Access" and "Browser Control" as flagship features,

you can see how OpenClaw checks the three boxes.

Securing OpenClaw

OpenClaw doesn't have to be limited to throwaway accounts though. Since it blew up, security has been one of the main focus points of OpenClaw's development, and you can leverage some of that today to get a secure experience in the harness. While this still requires you to be technically savvy, you can:

Use OpenClaw's tool policies to control which user and/or agent gets access to specific tools
Run it in a Sandbox
Use exec approvals to implement human-in-the-loop for specific tools that may have undesired side-effects
Use a detached tool-calling runtime like Arcade. Credentials never touch the harness, so there's nothing to leak.

Here's how to setup that last point in your OpenClaw instance:

First, clone the Arcade plugin:

git clone --depth 1 https://github.com/ArcadeAI/openclaw-arcade-plugin /tmp/openclaw-arcade

Then, install it into your OpenClaw gateway:

openclaw plugins install /tmp/openclaw-arcade/arcade

Go to your Arcade Dashboard to get and API key

copy it, and run this command to configure your Arcade API key:

openclaw config set plugins.entries.arcade.config.apiKey "{your_arcade_api_key}"

And this one to configure your Arcade User ID (this is the email you used to

sign up to Arcade):

openclaw config set plugins.entries.arcade.config.user_id "{your_arcade_user_id}"

Once the Arcade plugin is configured, initialize it to load all the tools, and

restart the OpenClaw gateway

openclaw arcade init
openclaw gateway restart

Now OpenClaw has access to 7,000+ tools, with tokens handled outside the harness. Nothing to exfiltrate.

Here's a screenshot of how this works when I talk to the Telegram bot connected

to my OpenClaw instance:

Final tips

Even with these precautions, OpenClaw is still early-adopter territory. Make sure to run this in a sandbox, a VPS, or even a dedicated computer. If you're sharing files to OpenClaw, make sure to set up the guardrails around the tools it can use, and be mindful of the accounts you log into in the browser it can control.

Ready to secure your agent setup?

Arcade handles just-in-time agent authorization so credentials never touch your harness → Get started