I gave open claw and codex the whole internet without any api keys using this tool and it was never performed better

Ajnas N B — Fri, 24 Jul 2026 18:48:02 +0000

AI agents can reason about the web.

But giving an agent unrestricted browser or network access creates a serious authority problem.

The obvious solution is to restrict the tools available to the agent.

Then I kept running into the opposite problem:

Once the tool became sufficiently restricted, it lost many of the capabilities required to complete real work.

I wanted both sides:

Enough power to crawl, render, navigate, extract, capture, and investigate the web
Explicit operator control over origins, credentials, budgets, browser hooks, profiles, and evidence

So I built Cockroach Crawler.

It is an open-source Node.js and TypeScript toolkit for AI agents, RAG pipelines, documentation indexing, research, QA, and web-data workflows.

I connected it to OpenClaw and Codex, and the difference was honestly wild.

Instead of giving the agents one narrow search tool, I gave them a bounded web-research layer that could crawl websites, inspect JavaScript applications, extract structured data, process PDFs, take screenshots, generate PDFs, inspect public sources, and return evidence with provenance.

And for many public workflows, I did not need to configure a separate API key for every source.

GitHub: https://github.com/AjnasNB/cockroach-crawler
Documentation: https://cockroachcrawler.com/docs/
npm: https://www.npmjs.com/package/cockroach-crawler

What changed after I connected it to OpenClaw and Codex?

Before this, the agents could reason well, but their web access was limited.

They could answer questions, write code, and work with the context I gave them. But once a task required deeper live-web investigation, I still had to manually combine several tools.

After connecting Cockroach Crawler, they could:

Crawl public websites
Render JavaScript-heavy pages
Follow sitemaps
Search and map documentation sites
Extract readable Markdown
Extract structured fields with CSS, XPath, or restricted regular expressions
Read local and remote PDFs
Generate PDFs
Take screenshots
Handle bounded clicks and scrolling
Inspect open Shadow DOM
Read same-origin iframe content
Collect canonical URLs and redirect history
Preserve content hashes and retrieval metadata
Return warnings, failures, and crawl statistics
Work through MCP
Use Docker or a local dashboard
Inspect public provider availability before dispatch

It stopped feeling like I had given the agents a simple browser.

It felt like I had given them an actual web-research system.

The core idea

The design principle is simple:

Give AI agents the web. Keep the keys.

The agent gets useful capabilities.

The host application keeps authority.

That means model-facing input can narrow a job, but it cannot silently add:

New origins
New credentials
New proxy endpoints
New browser hooks
New persistent profiles
Larger resource limits
Broader network authority

This matters because an agent should not be able to expand its own permissions simply by generating a more aggressive tool call.

What can Cockroach Crawler do?

Cockroach Crawler currently provides 50 documented capabilities across crawling, browser automation, extraction, providers, agent integration, deployment, and security.

Crawling and discovery

Static HTTP crawling
Multiple crawl seeds
Breadth-first traversal
Depth-first traversal
Best-first traversal
Adaptive relevance traversal
Sitemap discovery
Include and exclude filters
robots.txt enforcement
Redirect validation
Persistent hash-verified cache
Searchable site maps
Crawl deadlines and cancellation
Explicit concurrency and politeness controls

Browser rendering and capture

JavaScript rendering with Chromium
Selector waits
Bounded clicks
Infinite and virtual scrolling
Open Shadow DOM flattening
Readable same-origin iframe flattening
Screenshots
PDF generation
Persistent browser profiles
Explicit operator-supplied page hooks

Extraction

Readable Markdown
CSS extraction
XPath extraction
Restricted regular-expression extraction
Optional host-model schema extraction
Local PDF parsing
Links and metadata
Content hashes
Provenance records
Retrieval warnings and failures

Public sources and providers

Public GitHub reads
YouTube search without a developer API key through an optional reviewed provider
RSS and Atom parsing
Official provider routes
Read-only session routes
Provider doctor and deterministic routing

Agent integration

Native MCP server
Strict agent-tool adapter
Capability inspection
Maqam integration for policy, approvals, traces, and evidence

Deployment

Authenticated Docker API
Dashboard and playground
Bounded asynchronous jobs
Restricted Cloudflare Worker deployment

Security and authority

Public-network admission
Private-network blocking
Redirect validation
DNS validation
Explicit origin policy
Credential isolation
Resource ceilings
Challenge-aware escalation

Start with one command

Install the package:

npm install cockroach-crawler@0.5.2

Run a bounded crawl:

npx cockroach-crawl https://example.com/docs \
  --max-pages 20 \
  --max-requests 80 \
  --jsonl

The result includes extracted content plus the information an agent or developer needs to verify it:

Canonical URL
Parent URL
Crawl depth
Redirect history
Response status
Content type
Content hash
Retrieval metadata
Warnings
Failures
Crawl statistics

Use it from JavaScript

import { crawlDetailed } from "cockroach-crawler";

const result = await crawlDetailed({
  seeds: ["https://example.com/docs"],
  maxPages: 20,
  maxRequests: 80,
  maxDurationMs: 60_000
});

for (const page of result.pages) {
  console.log(page.url);
  console.log(page.text);
  console.log(page.contentHash);
}

console.log(result.failures);
console.log(result.stats);

The application owns the limits.

The model can request less.

It cannot silently request more.

Deep crawling strategies

Different jobs need different traversal behavior.

Breadth-first crawling

Use breadth-first crawling when you want broad coverage near the seed:

const result = await crawlDetailed({
  seeds: ["https://example.com/docs"],
  strategy: "bfs",
  maxPages: 50
});

Depth-first crawling

Use depth-first crawling for narrow or hierarchical documentation paths:

const result = await crawlDetailed({
  seeds: ["https://example.com/docs"],
  strategy: "dfs",
  maxPages: 50
});

Best-first and adaptive relevance crawling

Best-first and adaptive traversal prioritize pages that appear more useful for the current task while remaining inside the configured crawl budget.

This is useful when an agent needs to investigate a large documentation site without blindly downloading everything.

Render JavaScript applications

Static HTTP is fast, but many modern applications require a real browser.

Cockroach Crawler provides optional Playwright integration for:

JavaScript rendering
Waiting for selectors or page states
Explicit clicks
Bounded scrolling
Shadow DOM inspection
Readable same-origin frames
Screenshots
PDF output
Dedicated profiles
Reviewed page hooks

Install the optional browser peer:

npm install cockroach-crawler playwright
npx playwright install chromium

Browser mode still uses explicit origins and resource ceilings.

Enabling Chromium does not give the agent unrestricted operating-system or network authority.

Extract structured data

Readable Markdown is useful for agents and RAG pipelines.

But many applications need deterministic fields.

Cockroach Crawler supports bounded:

CSS extraction
XPath extraction
Restricted regular-expression extraction
Optional model-assisted schema extraction

The model-assisted route still remains host-controlled.

The operator supplies the model adapter, and the final output must pass JSON Schema validation.

Model input cannot choose arbitrary credentials, origins, hooks, or execution authority.

Search and map an entire site

The crawler can create compact, fetch-validated site maps and optionally search them.

This is useful for:

Documentation discovery
Migration inventories
Support knowledge bases
RAG ingestion
Broken-link investigation
Product audits
Content audits

Unlike a sitemap assembled only from discovered links, entries can retain retrieval state and source identity for later verification.

Connect it to OpenClaw, Codex, or another agent through MCP

Cockroach Crawler includes a native MCP stdio server.

Example configuration:

{
  "mcpServers": {
    "cockroach-crawler": {
      "command": "npx",
      "args": [
        "-y",
        "cockroach-crawler@0.5.2",
        "cockroach-mcp"
      ]
    }
  }
}

The MCP surface exposes bounded:

Crawling
Site mapping
Structured extraction
Capability inspection

The MCP client does not receive control over operator-owned credentials, browser hooks, proxy endpoints, persistent profiles, or origin ceilings.

That was the important part for me.

I wanted OpenClaw and Codex to become much more capable without allowing the model to decide its own security boundary.

Inspect providers before dispatch

The provider doctor reports what the current machine can actually use:

npx -y --package cockroach-crawler@0.5.2 \
  cockroach-sources doctor --json

It distinguishes between:

Public routes
Official API routes
Optional no-key routes
Explicit read-only session routes
Missing configuration
Unsupported operations

A provider is not considered available just because its name appears in a configuration file.

The route must actually be usable in the current environment.

Docker and self-hosting

Cockroach Crawler includes an authenticated Node.js and Docker API with a dashboard and playground.

It can also run bounded process-local jobs with:

Status inspection
Cancellation
Result ceilings
Authentication
Fixed deployment authority

For edge deployments, the package includes a separate Cloudflare Worker profile for fixed deployment-owned HTTPS origins.

That Worker is intentionally treated as a different security boundary from the local Node.js crawler.

Public benchmark evidence

The current package was evaluated on all 511 held-out pages of the pinned WCEB v1.0 test split.

The published run produced:

0.7653 macro word F1
0.9041 recall
87.13% required-snippet recall
25/25 adapted robots dispatch vectors passed
101/101 applicable credential-free WPT HTTP(S) URL cases passed

The evaluator, dataset revision, source fingerprint, page-level rows, and machine-readable results are committed to the repository.

These results describe performance on the named test corpus.

They are not a claim of universal extraction quality.

Benchmark details:

https://cockroachcrawler.com/benchmark/

What it is not designed to do

Cockroach Crawler is not designed to bypass:

CAPTCHAs
Access controls
Private-network restrictions
Explicit robots policies
Authentication boundaries
Platform challenges

It is designed to provide useful web capabilities while keeping authority visible, bounded, and operator-owned.

Why this made OpenClaw and Codex feel completely different

The real improvement was not simply that they could fetch more pages.

It was that they could perform an entire evidence-backed workflow:

Discover the relevant pages
Render the pages when necessary
Extract the useful content
Process PDFs
Capture screenshots or generated PDFs
Preserve canonical URLs and hashes
Return failures instead of silently hiding them
Stay within explicit origin and budget limits

That changed the agents from systems that merely talked about the web into systems that could actually investigate it.

And honestly, this was the first time it felt like I had given them the whole internet without giving away the keys.

Try to break it

If you are building an AI agent, research system, RAG pipeline, documentation indexer, or browser-assisted workflow, test it on something real.

Open an issue with:

Operating system
Node.js version
Public test URL
Smallest reproducible command
Expected behavior
Actual behavior

Install:

npm install cockroach-crawler@0.5.2

Links:

GitHub: https://github.com/AjnasNB/cockroach-crawler
Documentation: https://cockroachcrawler.com/docs/
Capability index: https://cockroachcrawler.com/docs/capabilities/
Security model: https://cockroachcrawler.com/security/
Benchmark: https://cockroachcrawler.com/benchmark/
npm: https://www.npmjs.com/package/cockroach-crawler