DEV Community: The Web3 Wizard

Why AI Coding Agents Hallucinate on EVM Bugs (And How We Packaged 5 Testable Skills to Fix It)

The Web3 Wizard — Mon, 19 Oct 2026 02:19:25 +0000

If you have tried using Claude Code, Cursor, or ChatGPT to debug a reverting Solidity transaction or write Foundry invariants, you have probably watched this sequence play out:

You feed the agent an error message or a raw transaction hash.
The model apologizes, guesses three different reasons why the call might have failed, and suggests rewriting a functioning helper contract.
If you ask it to generate an invariant test, it hallucinates a stateless property or creates a ghost variable that passes without ever touching real contract storage.

The problem isn't that current LLMs lack Solidity knowledge. The problem is that general-purpose coding agents lack a repeatable, evidence-first workflow for Web3 systems.

In Web3, a wrong suggestion isn't just a syntax error that fails the build. It can mean an unhandled reorg, an unauthorized external call, or a misconfigured slippage parameter.

To fix this, we structured and open-sourced Spellbook Skills: five testable, deterministic workflow packs designed for AI coding agents like Claude Code, Cursor, and Antigravity.

Here is why coding agents fail on EVM tasks, how structured skills change their behavior, and how you can install them today.

The Core Problem: Reasoning Without Evidence

When a human smart contract engineer investigates a revert, they follow a systematic loop:

Get tx receipt → Decode custom error selector → Pull trace via `cast run` → Inspect storage slot / state → Form hypothesis → Write reproducible test

When an unguided AI agent tries to do the same task, it skips directly from raw error → proposed code edit. It relies on pattern-matching against training data rather than verifying the actual state on-chain.

To make an agent reliable, you don't need a larger prompt. You need to constrain the agent with four operational boundaries:

Evidence before conclusions: The agent must pull the revert reason or cast run trace before suggesting code modifications.
Strict separation of roles: Scaffolding a test is different from verifying a vulnerability.
Zero secret handling: An agent must never ask for, log, or accept private keys, mnemonic phrases, or live signer credentials.
Deny-by-default execution: Any on-chain action or transaction proposal must pass an explicit policy gate before it can be executed.

The 5 Skills We Packaged

Rather than building a monolithic "all-in-one Web3 agent," we built five focused skills. Each skill contains a SKILL.md operating specification, reference check-lists, test fixtures, and runnable helpers.

1. `evm-tx-failure-diagnoser`

What it targets: Cryptic reverts, custom error selectors, gas exhaustion, and nonce collisions.

The workflow: It prevents the agent from guessing. The agent is forced to decode the 4-byte error selector, fetch the trace via cast run <TX_HASH> or local RPC, isolate the failing call frame, and document the root cause before touching contract files.

2. `foundry-invariant-scaffold`

What it targets: Brittle unit tests that miss edge-case multi-step exploits.

The workflow: Guides the agent through setting up genuine stateful fuzzing in Foundry. It scaffolds dedicated handler contracts, defines actor proxies, bounds inputs with realistic ghost variables, and structures assertions that check system-level invariants across sequences of calls.

3. `web3-agent-tx-safety`

What it targets: Autonomous agents that propose or broadcast transactions.

The workflow: Implements a strict, deny-by-default transaction policy gate. Every transaction payload must pass through parameter bounds (target allowlist, max value, calldata selector inspection) and a simulated dry-run before any signing payload can be constructed.

4. `slither-finding-verifier`

What it targets: Static analysis triage fatigue and false positives.

The workflow: Takes raw Slither JSON output and normalizes it into a triage worksheet. It requires the agent to verify the finding against actual source AST before generating a targeted, reproducible test case that confirms whether the vulnerability is exploitable or benign.

5. `rpc-event-reliability`

What it targets: Event listeners that miss logs during network congestion or reorgs.

The workflow: Hardens log ingestion backends. It structures durable event deduplication keys (chain + blockHash + txHash + logIndex), configures Viem fallback transports, and builds reorg reconciliation routines that handle chain reorganizations gracefully.

How to Install and Use Them

All five skills are open-source under the MIT License.

If you use the skills CLI, you can install any of them directly into your agent's workspace:

# Install the EVM Diagnoser for Claude Code
npx skills add theweb3wizard/spellbook-skills --skill evm-tx-failure-diagnoser -a claude-code

# Install the Foundry Invariant Scaffolder
npx skills add theweb3wizard/spellbook-skills --skill foundry-invariant-scaffold -a claude-code

# Or install all 5 skills into your project
npx skills add theweb3wizard/spellbook-skills --skill '*' -a claude-code

Manual Installation

You can also clone the repository and drop the skill directories directly into your workspace:

Claude Code: .claude/skills/<skill-name>/
Cursor / Codex: .agents/skills/<skill-name>/

git clone https://github.com/theweb3wizard/spellbook-skills.git

Example: What a Guarded Agent Run Looks Like

Here is what happens when you prompt an agent equipped with evm-tx-failure-diagnoser:

Prompt:

"Our deposit transaction reverted on Base: 0x7a8b...3f1c. Figure out why and tell me what needs fixing."

Without the skill:
The agent often guesses: "It looks like you didn't approve the ERC20 token first, or the contract is paused."

With the skill:
The agent executes the diagnostic workflow:

It queries the transaction receipt and extracts the revert data: 0x118cdaa7.
It decodes the selector against the project ABI: InsufficientCollateral(uint256 current, uint256 required).
It notes that the required collateral threshold was calculated using an oracle timestamp older than the protocol's staleness tolerance (maxOracleDelay = 3600).
It outputs an evidence-backed triage report and suggests a regression test fixture rather than blindly editing contract code.

Open Source & Community Feedback

We built these skills as part of our work at Spellbook, where we design and deploy production-ready AI agents for Web3 protocols.

The skills are completely free and testable offline. You can review the full code, test suites, and workflows here:

🐙 GitHub Repository: github.com/theweb3wizard/spellbook-skills
📖 Interactive Skills Directory: spellbook.theweb3wizard.xyz/skills

If you test them out on your smart contract projects, drop a reaction or let me know what other recurring Web3 workflows your agents struggle with in the comments below!

Why Vector Search Guesses and Graphs Prove: What Building ClearPath Taught Me

The Web3 Wizard — Sun, 18 Oct 2026 10:14:48 +0000

Let’s talk about something I see over and over in Web3 and AI right now.

Everyone wants to build AI agents.

And when builders decide to give an AI agent access to data, many reach for the same default stack:

Chunk documents, compute embeddings, store them in a vector database, and run cosine similarity search (RAG).

For many tasks, like answering questions about documentation or searching blog posts, vector search works just fine.

But the moment your problem depends on how things connect, vector search alone falls short.

Let me show you why.

The $15,000 OTC Dilemma

Meet Dave.

Dave runs an over-the-counter (OTC) crypto desk. He has Telegram open. A counterparty wants to buy $15,000 worth of stablecoins.

The trade window has a timer: Dave has 2 minutes to accept or decline the quote.

The counterparty drops a wallet address:

0x7a250d5630b4cf539739df2c5dacb4c659f2488d

Now Dave has a problem.

If those funds are connected to an exploit, a money mule, or a sanctioned mixer such as Tornado Cash just a few hops away, the transaction could expose his business to serious compliance and financial risks.

Enterprise tools like Chainalysis and TRM Labs can be expensive, putting advanced blockchain analytics out of reach for some independent operators.

So a builder says:

"Hey, let's build an AI agent with vector search to audit crypto wallets!"

And here's the fundamental flaw.

Vector Search Cannot Express a Graph Hop

A vector database stores numerical representations of semantic meaning in a high-dimensional space.

It can tell you that the phrase "illicit money laundering" is conceptually similar to "Tornado Cash mixer."

But semantic similarity alone cannot establish:

Did Address A send 5 ETH to Mule B?

Did Mule B retain a 0.5% fee and forward the rest to Consolidator C?

Did Consolidator C fund this exact counterparty 45 minutes later?

Did the trail terminate at a Binance hot wallet?

These are questions about relationships, direction, transaction history, and graph topology.

When an LLM investigates a wallet using only vector search, it sees retrieved information without necessarily seeing the connections between the entities.

That creates a dangerous gap between finding relevant information and establishing an evidence-backed transaction path.

In financial forensics:

Vector search finds semantic similarities. Graph queries trace relationships.

They solve different problems, and knowing when to use each matters.

The Removal Test: Making FalkorDB Load-Bearing

When I entered the Graph Hacks Hackathon, rule #3 caught my attention immediately:

"The graph has to do real work. If the product still works once FalkorDB is taken out, the graph is decoration."

That rule separated serious graph-based solutions from projects that merely used a graph database as a technical accessory.

I wanted to build ClearPath so that FalkorDB was the load-bearing engine of the entire product.

Remove FalkorDB, and ClearPath loses the graph traversal and relationship analysis that power its core verdicts.

Here's what happens under the hood when Dave clicks Evaluate.

Directional Multi-Hop Traversal with Exponential Decay

Instead of asking an LLM to guess whether two wallets might be connected, ClearPath runs an OpenCypher query against FalkorDB Cloud.

MATCH path = (source:Wallet)-[:TRANSFERRED*1..4]->(target:Wallet {address: $targetAddress})
WHERE source.entity_type IN ['mixer', 'sanctioned', 'mule']
AND ALL(rel IN relationships(path) WHERE rel.amount_usd >= 10.0)
AND NONE(node IN nodes(path)[1..-1] WHERE node.entity_type = 'exchange_hot_wallet')
RETURN
source.address AS source_address,
source.entity_type AS source_type,
length(path) AS hop_count,
[r IN relationships(path) | {
from: startNode(r).address,
to: endNode(r).address,
amount: r.amount_usd,
tx: r.tx_hash
}] AS steps

Notice what's happening here:

Multi-hop traversal: The query searches paths of up to four transfer edges between a flagged source and the target wallet.

Dust filtering: Transfers below $10 are excluded to help reduce noise and potential dust-attack false positives.

Exchange boundaries: Paths containing an exchange hot wallet as an intermediate node are excluded because centralized exchanges complicate direct on-chain attribution.

Distance-based scoring: ClearPath applies exponential decay using a factor of (0.6^{\text{hop}}), so the contribution of a flagged source decreases as the number of hops increases.

The key idea is that the system doesn't merely retrieve text about suspicious wallets. It traces actual relationships represented in the graph.

FalkorDB is built around graph-processing techniques that leverage sparse matrix operations, making it well suited to relationship-heavy workloads.

In my ClearPath implementation, the traversal returned results in 4 milliseconds during testing.

That result is specific to my test environment and query; performance will vary with graph size, query complexity, and infrastructure.

Still, it's a compelling demonstration of why graph databases deserve a place in the AI agent stack.

Track 2: Giving the Agent True Episodic Memory

Many projects describe their agents as having "memory" when they're simply appending chat transcripts to a database table.

A chat log is useful, but it isn't the same as structured memory about what an agent previously investigated and what has changed since.

In ClearPath, I used FalkorDB's multigraph capabilities to maintain separate graph contexts within the same cloud instance:

clearpath_trust: The authoritative financial network.

clearpath_mem_dave: Dave's private episodic memory graph.

clearpath_shared_cases: A shared compliance board for verified fraud cases.

When Dave investigates an address today that he scanned two hours ago, the agent can check for new transactions instead of starting from scratch.

For example, it can run a temporal delta query:

MATCH (inflow:Wallet)-[r:TRANSFERRED]->(target:Wallet {address: $targetAddress})
WHERE r.timestamp > $lastSessionTimestamp
RETURN
count(r) AS new_inflows,
coalesce(sum(r.amount_usd), 0.0) AS new_volume

If a new transaction occurred while Dave was away, the agent can surface an alert:

"Notice: 1 new transaction ($8,500) detected since your last screening."

That's the idea behind ClearPath's cross-session episodic memory: retain relevant investigative context and use it to identify changes over time.

Of course, the quality of those alerts depends on accurate timestamps, complete transaction data, and correctly implemented session tracking.

The "Theater Test"

During hackathon demos, judges have good reason to be skeptical. They've seen plenty of mockups with hardcoded states that look impressive but don't demonstrate a working backend.

So we built what I call the Theater Test.

Here's how it works:

Evaluate Exhibit C (0xDEMO_CLEAN_TARGET_03). The verdict comes back as NO-KNOWN-RISK.

Click "Inject Mule Transfer (Flip to Caution)."

Behind the scenes, an actual Cypher query executes against FalkorDB Cloud, creating a transfer edge from a known demo money-mule entity into Exhibit C.

Run the evaluation again. The screen changes to CAUTION, the visual transaction trail displays the new connection, and the temporal-delta banner appears.

Click "Rollback Taint." The newly created edge is deleted, and the screen returns to its previous state after re-evaluation.

The point isn't simply that the UI changes.

It's that the change is driven by a mutation to the underlying graph, followed by a fresh evaluation.

Hardcoded demos show a state. Live graph systems can demonstrate how that state changes when the underlying relationships change.

That distinction matters when you're building products that claim to reason over real data.

What I Took Away From This

Problem first, technology last

The goal was never to show off a graph database or an AI model for its own sake.

The goal was to help Dave make a better-informed decision during a two-minute deal window.

Every technical choice should serve that problem.

Deterministic computation beats generative guessing

One of the biggest mistakes in AI agent design is allowing an LLM to invent or independently calculate critical risk scores without a deterministic, auditable method.

In ClearPath, FalkorDB handles relationship traversal, while the scoring logic applies explicit rules. Google Gemini 2.5 Flash is used to explain the evidence in human-readable language.

The distinction is important:

Let deterministic systems calculate the evidence-based result. Let the LLM explain it.

Neither a graph connection nor an automated risk score, however, should automatically be treated as proof of wrongdoing. Real financial investigations require reliable data, context, and appropriate human review.

Graph databases are more than a visualization layer

FalkorDB isn't there just to draw pretty lines between wallets.

It's part of the computation that makes ClearPath work: traversing transaction paths, identifying relationships, and detecting changes in the network.

That makes the graph load-bearing rather than decorative.

Make demos prove the architecture

A convincing demo shouldn't just show what a product looks like when everything works.

It should demonstrate what happens when the underlying data changes, how the system responds, and whether the result can be reproduced.

That's why the Theater Test mattered to me.

Final Thoughts

If you're building AI agents and your problem involves relationships, hierarchies, dependencies, or networks, don't force vector search to do a graph database's job.

Vector search is useful for semantic retrieval. Graph databases are useful for traversing and analyzing relationships. In many real-world systems, combining them is more powerful than treating either as a universal solution.

Give your agent the right tools for the problem.

For ClearPath, that meant making the graph do real work.

Built by Khalid for Graph Hacks 2026.

🔗 Explore my work: https://github.com/theweb3wizard