DEV Community

Cover image for I gave open claw and codex the whole internet without any api keys using this tool and it was never performed better
Ajnas N B
Ajnas N B

Posted on

I gave open claw and codex the whole internet without any api keys using this tool and it was never performed better

#browser #agents #opensource #webdev

AI agents can reason about the web.

But giving an agent unrestricted browser or network access creates a serious authority problem.

The obvious solution is to restrict the tools available to the agent.

Then I kept running into the opposite problem:

Once the tool became sufficiently restricted, it lost many of the capabilities required to complete real work.

I wanted both sides:

  • Enough power to crawl, render, navigate, extract, capture, and investigate the web
  • Explicit operator control over origins, credentials, budgets, browser hooks, profiles, and evidence

So I built Cockroach Crawler.

It is an open-source Node.js and TypeScript toolkit for AI agents, RAG pipelines, documentation indexing, research, QA, and web-data workflows.

I connected it to OpenClaw and Codex, and the difference was honestly wild.

Instead of giving the agents one narrow search tool, I gave them a bounded web-research layer that could crawl websites, inspect JavaScript applications, extract structured data, process PDFs, take screenshots, generate PDFs, inspect public sources, and return evidence with provenance.

And for many public workflows, I did not need to configure a separate API key for every source.

What changed after I connected it to OpenClaw and Codex?

Before this, the agents could reason well, but their web access was limited.

They could answer questions, write code, and work with the context I gave them. But once a task required deeper live-web investigation, I still had to manually combine several tools.

After connecting Cockroach Crawler, they could:

  • Crawl public websites
  • Render JavaScript-heavy pages
  • Follow sitemaps
  • Search and map documentation sites
  • Extract readable Markdown
  • Extract structured fields with CSS, XPath, or restricted regular expressions
  • Read local and remote PDFs
  • Generate PDFs
  • Take screenshots
  • Handle bounded clicks and scrolling
  • Inspect open Shadow DOM
  • Read same-origin iframe content
  • Collect canonical URLs and redirect history
  • Preserve content hashes and retrieval metadata
  • Return warnings, failures, and crawl statistics
  • Work through MCP
  • Use Docker or a local dashboard
  • Inspect public provider availability before dispatch

It stopped feeling like I had given the agents a simple browser.

It felt like I had given them an actual web-research system.

The core idea

The design principle is simple:

Give AI agents the web. Keep the keys.

The agent gets useful capabilities.

The host application keeps authority.

That means model-facing input can narrow a job, but it cannot silently add:

  • New origins
  • New credentials
  • New proxy endpoints
  • New browser hooks
  • New persistent profiles
  • Larger resource limits
  • Broader network authority

This matters because an agent should not be able to expand its own permissions simply by generating a more aggressive tool call.

What can Cockroach Crawler do?

Cockroach Crawler currently provides 50 documented capabilities across crawling, browser automation, extraction, providers, agent integration, deployment, and security.

Crawling and discovery

  • Static HTTP crawling
  • Multiple crawl seeds
  • Breadth-first traversal
  • Depth-first traversal
  • Best-first traversal
  • Adaptive relevance traversal
  • Sitemap discovery
  • Include and exclude filters
  • robots.txt enforcement
  • Redirect validation
  • Persistent hash-verified cache
  • Searchable site maps
  • Crawl deadlines and cancellation
  • Explicit concurrency and politeness controls

Browser rendering and capture

  • JavaScript rendering with Chromium
  • Selector waits
  • Bounded clicks
  • Infinite and virtual scrolling
  • Open Shadow DOM flattening
  • Readable same-origin iframe flattening
  • Screenshots
  • PDF generation
  • Persistent browser profiles
  • Explicit operator-supplied page hooks

Extraction

  • Readable Markdown
  • CSS extraction
  • XPath extraction
  • Restricted regular-expression extraction
  • Optional host-model schema extraction
  • Local PDF parsing
  • Links and metadata
  • Content hashes
  • Provenance records
  • Retrieval warnings and failures

Public sources and providers

  • Public GitHub reads
  • YouTube search without a developer API key through an optional reviewed provider
  • RSS and Atom parsing
  • Official provider routes
  • Read-only session routes
  • Provider doctor and deterministic routing

Agent integration

  • Native MCP server
  • Strict agent-tool adapter
  • Capability inspection
  • Maqam integration for policy, approvals, traces, and evidence

Deployment

  • Authenticated Docker API
  • Dashboard and playground
  • Bounded asynchronous jobs
  • Restricted Cloudflare Worker deployment

Security and authority

  • Public-network admission
  • Private-network blocking
  • Redirect validation
  • DNS validation
  • Explicit origin policy
  • Credential isolation
  • Resource ceilings
  • Challenge-aware escalation

Start with one command

Install the package:

npm install cockroach-crawler@0.5.2
Enter fullscreen mode Exit fullscreen mode

Run a bounded crawl:

npx cockroach-crawl https://example.com/docs \
  --max-pages 20 \
  --max-requests 80 \
  --jsonl
Enter fullscreen mode Exit fullscreen mode

The result includes extracted content plus the information an agent or developer needs to verify it:

  • Canonical URL
  • Parent URL
  • Crawl depth
  • Redirect history
  • Response status
  • Content type
  • Content hash
  • Retrieval metadata
  • Warnings
  • Failures
  • Crawl statistics

Use it from JavaScript 

import { crawlDetailed } from "cockroach-crawler";

const result = await crawlDetailed({
  seeds: ["https://example.com/docs"],
  maxPages: 20,
  maxRequests: 80,
  maxDurationMs: 60_000
});

for (const page of result.pages) {
  console.log(page.url);
  console.log(page.text);
  console.log(page.contentHash);
}

console.log(result.failures);
console.log(result.stats);
Enter fullscreen mode Exit fullscreen mode

The application owns the limits.

The model can request less.

It cannot silently request more.

Deep crawling strategies

Different jobs need different traversal behavior.

Breadth-first crawling

Use breadth-first crawling when you want broad coverage near the seed:

const result = await crawlDetailed({
  seeds: ["https://example.com/docs"],
  strategy: "bfs",
  maxPages: 50
});
Enter fullscreen mode Exit fullscreen mode

Depth-first crawling

Use depth-first crawling for narrow or hierarchical documentation paths:

const result = await crawlDetailed({
  seeds: ["https://example.com/docs"],
  strategy: "dfs",
  maxPages: 50
});
Enter fullscreen mode Exit fullscreen mode

Best-first and adaptive relevance crawling

Best-first and adaptive traversal prioritize pages that appear more useful for the current task while remaining inside the configured crawl budget.

This is useful when an agent needs to investigate a large documentation site without blindly downloading everything.

Render JavaScript applications

Static HTTP is fast, but many modern applications require a real browser.

Cockroach Crawler provides optional Playwright integration for:

  • JavaScript rendering
  • Waiting for selectors or page states
  • Explicit clicks
  • Bounded scrolling
  • Shadow DOM inspection
  • Readable same-origin frames
  • Screenshots
  • PDF output
  • Dedicated profiles
  • Reviewed page hooks

Install the optional browser peer:

npm install cockroach-crawler playwright
npx playwright install chromium
Enter fullscreen mode Exit fullscreen mode

Browser mode still uses explicit origins and resource ceilings.

Enabling Chromium does not give the agent unrestricted operating-system or network authority.

Extract structured data

Readable Markdown is useful for agents and RAG pipelines.

But many applications need deterministic fields.

Cockroach Crawler supports bounded:

  • CSS extraction
  • XPath extraction
  • Restricted regular-expression extraction
  • Optional model-assisted schema extraction

The model-assisted route still remains host-controlled.

The operator supplies the model adapter, and the final output must pass JSON Schema validation.

Model input cannot choose arbitrary credentials, origins, hooks, or execution authority.

Search and map an entire site

The crawler can create compact, fetch-validated site maps and optionally search them.

This is useful for:

  • Documentation discovery
  • Migration inventories
  • Support knowledge bases
  • RAG ingestion
  • Broken-link investigation
  • Product audits
  • Content audits

Unlike a sitemap assembled only from discovered links, entries can retain retrieval state and source identity for later verification.

Connect it to OpenClaw, Codex, or another agent through MCP

Cockroach Crawler includes a native MCP stdio server.

Example configuration:

{
  "mcpServers": {
    "cockroach-crawler": {
      "command": "npx",
      "args": [
        "-y",
        "cockroach-crawler@0.5.2",
        "cockroach-mcp"
      ]
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

The MCP surface exposes bounded:

  • Crawling
  • Site mapping
  • Structured extraction
  • Capability inspection

The MCP client does not receive control over operator-owned credentials, browser hooks, proxy endpoints, persistent profiles, or origin ceilings.

That was the important part for me.

I wanted OpenClaw and Codex to become much more capable without allowing the model to decide its own security boundary.

Inspect providers before dispatch

The provider doctor reports what the current machine can actually use:

npx -y --package cockroach-crawler@0.5.2 \
  cockroach-sources doctor --json
Enter fullscreen mode Exit fullscreen mode

It distinguishes between:

  • Public routes
  • Official API routes
  • Optional no-key routes
  • Explicit read-only session routes
  • Missing configuration
  • Unsupported operations

A provider is not considered available just because its name appears in a configuration file.

The route must actually be usable in the current environment.

Docker and self-hosting

Cockroach Crawler includes an authenticated Node.js and Docker API with a dashboard and playground.

It can also run bounded process-local jobs with:

  • Status inspection
  • Cancellation
  • Result ceilings
  • Authentication
  • Fixed deployment authority

For edge deployments, the package includes a separate Cloudflare Worker profile for fixed deployment-owned HTTPS origins.

That Worker is intentionally treated as a different security boundary from the local Node.js crawler.

Public benchmark evidence

The current package was evaluated on all 511 held-out pages of the pinned WCEB v1.0 test split.

The published run produced:

  • 0.7653 macro word F1
  • 0.9041 recall
  • 87.13% required-snippet recall
  • 25/25 adapted robots dispatch vectors passed
  • 101/101 applicable credential-free WPT HTTP(S) URL cases passed

The evaluator, dataset revision, source fingerprint, page-level rows, and machine-readable results are committed to the repository.

These results describe performance on the named test corpus.

They are not a claim of universal extraction quality.

Benchmark details:

https://cockroachcrawler.com/benchmark/

What it is not designed to do

Cockroach Crawler is not designed to bypass:

  • CAPTCHAs
  • Access controls
  • Private-network restrictions
  • Explicit robots policies
  • Authentication boundaries
  • Platform challenges

It is designed to provide useful web capabilities while keeping authority visible, bounded, and operator-owned.

Why this made OpenClaw and Codex feel completely different

The real improvement was not simply that they could fetch more pages.

It was that they could perform an entire evidence-backed workflow:

  1. Discover the relevant pages
  2. Render the pages when necessary
  3. Extract the useful content
  4. Process PDFs
  5. Capture screenshots or generated PDFs
  6. Preserve canonical URLs and hashes
  7. Return failures instead of silently hiding them
  8. Stay within explicit origin and budget limits

That changed the agents from systems that merely talked about the web into systems that could actually investigate it.

And honestly, this was the first time it felt like I had given them the whole internet without giving away the keys.

Try to break it

If you are building an AI agent, research system, RAG pipeline, documentation indexer, or browser-assisted workflow, test it on something real.

Open an issue with:

  • Operating system
  • Node.js version
  • Public test URL
  • Smallest reproducible command
  • Expected behavior
  • Actual behavior

Install:

npm install cockroach-crawler@0.5.2
Enter fullscreen mode Exit fullscreen mode

Links:

Give AI agents the web. Keep the keys.

Top comments (0)