DEV Community: Ajnas N B

What you save when project context stops repeating

Ajnas N B — Tue, 11 Aug 2026 03:20:45 +0000

Qarinah compiles a compact, cited project-memory pack instead of asking every new coding-agent session to replay the entire available history.

The published estimate

Across six committed software-task fixtures, the full-history baseline contained 442,113 portable estimated input-context tokens. The Qarinah path used 5,682. Every required target was still directly covered in the top five results.

That is:

436,431 fewer estimated input-context tokens;
98.71% less repeated context; and
a 77.81:1 baseline-to-pack ratio.

The ratio is not a claim that every provider bill drops by 98.71%, or that an agent session lasts 77.81 times longer. It measures the compared input-context volume in the published six-fixture estimate.

What the same token rate would cost

The table applies four flat, uncached input-token rates to the same two token estimates. It is arithmetic, not a provider invoice.

Flat uncached input rate	Full-history baseline	Qarinah pack	Estimated saving
$1 / million tokens	$0.442113	$0.005682	$0.436431
$3 / million tokens	$1.326339	$0.017046	$1.309293
$5 / million tokens	$2.210565	$0.028410	$2.182155
$15 / million tokens	$6.631695	$0.085230	$6.546465

The calculation is:

estimated tokens / 1,000,000 x flat input rate

It deliberately excludes provider-native tokenization, caching, output tokens, reasoning tokens, tool calls, retrieval, hosting, and fixed fees. Real cost depends on the provider, model, cache behavior, context composition, and how often the same history would otherwise be resent.

Why the pack remains useful

Compression only matters if the next task can still find its evidence. The benchmark checks both volume and retrieval coverage: every required target had to be directly present in the top five. Qarinah preserves the source event ID and content hash for selected context, so a later agent receives a bounded handoff that can be inspected instead of an opaque story.

Qarinah also passed 380 of 380 deterministic file-specific exact and typo-tolerant queries across 40-, 50-, and 100-file projects. Those tests verify retrieval behavior; they do not establish universal task quality or a ranking against every memory system.

Reproduce it

The methodology, fixture records, costs, and exclusions are public:

Qarinah is Apache-2.0 and local-first. It works with Codex, Claude Code, Cursor, CLI tools, and compatible MCP clients. Project capture is opt-in, metadata-only by default, and does not capture hidden reasoning.

Would you trust agent memory more if every selected item pointed back to its source?

A 50-capability map for governed web crawling and AI agents

Ajnas N B — Sun, 09 Aug 2026 15:23:12 +0000

Giving an agent “web access” sounds like one feature. In practice, it is a stack of separate decisions:

How does the system discover URLs?
Which destinations can it contact?
Does it need a browser, or is static HTTP enough?
What turns the response into agent-ready data?
Where are request, byte, depth, and time limits enforced?
What evidence comes back with the extracted content?

Treating all of that as one unrestricted browser capability makes systems difficult to reason about. A better approach is to choose the smallest acquisition surface that completes the job, then make its authority explicit.

This article maps 50 current Cockroach Crawler capabilities into seven jobs. It is also a practical checklist you can use with another crawler: if a capability matters to your workflow, identify its input contract, output contract, failure behavior, and authority boundary before an agent depends on it.

Disclosure: I’m Ajnas N B, the developer of Cockroach Crawler. The project is open source under the MIT license.

Start with a finite crawl contract

The next channel currently contains the reviewed 0.7.0-rc.1 prerelease. A bounded documentation crawl can start like this:

npm install cockroach-crawler@next

import { crawlDetailed } from "cockroach-crawler";

const result = await crawlDetailed({
  seeds: ["https://docs.example.com"],
  allowedOrigins: ["https://docs.example.com"],
  include: ["/guides/", "/reference/"],
  exclude: ["/archive/"],
  traversal: "bfs",
  obeyRobots: true,
  maxPages: 25,
  maxRequests: 120,
  maxDepth: 4,
  maxTotalBytes: 10_000_000,
  maxDurationMs: 60_000,
  concurrency: 4
});

for (const page of result.pages) {
  console.log(page.url, page.contentHash, page.markdown.length);
}

The important part is not the number of options. It is ownership: the creator of the agent sets the origins and ceilings. Model-facing input can narrow that contract, but it should not be able to expand it.

1. Crawl and discover — 15 capabilities

These capabilities decide what enters the queue, what is contacted, and when the job stops.

Static HTTP crawling — fetch public HTTP(S) pages without starting a browser.
Multiple seeds — begin one bounded job from several explicit entry points.
Breadth-first traversal — cover each depth level before going deeper.
Depth-first traversal — follow the newest admitted path first.
Best-first traversal — rank admitted links against a bounded relevance query.
Adaptive relevance traversal — reprioritize the queue as relevant page text appears.
Sitemap discovery — read robots-declared and conventional sitemap locations, including nested indexes.
Robots enforcement — evaluate robots policy before page contact and preserve the decision.
Include and exclude filters — admit only the paths that belong to the job.
Validated redirects — inspect and admit every redirect destination before following it.
Concurrency and politeness — combine exact concurrent work with per-origin delays and global ceilings.
Deadlines and cancellation — stop by wall-clock budget or AbortSignal.
Persistent cache — reuse hash-verified results inside an explicit namespace, TTL, entry, and byte budget.
Compact fetch-validated site maps — return URL metadata without retaining complete page bodies.
Searchable fetch-validated site maps — rank only entries already admitted and fetched under the crawl policy.

For discovery work, the key distinction is between ranking and authority. A relevance score may reorder already admitted links; it must not broaden the origin policy or resource budget.

2. Render and capture — 9 capabilities

Static HTTP should remain the default when it works. Browser execution is useful when content genuinely depends on client-side rendering or bounded interaction.

JavaScript rendering through optional Chromium.
Selector waits and bounded clicks for explicit page states and interactions.
Infinite and virtual scroll with finite steps and stability checks.
Open Shadow DOM flattening into a bounded extraction snapshot.
Readable same-origin iframe flattening while preserving cross-origin isolation.
Full-page screenshots with format, size, and SHA-256 evidence metadata.
PDF generation with explicit print settings.
Trusted operator page hooks that are reviewed configuration, not model input.
Explicit persistent browser profiles using a dedicated directory rather than discovering a personal browser profile.

Browser mode is not a process sandbox. Host isolation, egress policy, CPU and memory limits, and sensitive-data separation still belong to the deployment.

3. Extract agent-ready data — 8 capabilities

Retrieval is not finished when bytes arrive. Agents need a bounded record that preserves enough source identity to verify or revisit the result.

Readable Markdown through the dependency-light core or opt-in Node quality backend.
CSS schema extraction for visible text, cleaned HTML, and named attributes.
XPath extraction for deterministic fields in inactive markup.
Restricted regex extraction with safe flags and hard input, item, value, and total ceilings.
Optional host-model JSON Schema extraction where returned JSON must validate against the supplied schema.
Local PDF parsing with signature, page, byte, and text ceilings.
Links and page metadata including canonical URL, title, description, language, status, ETag, and Last-Modified.
Evidence hashes and retrieval provenance including SHA-256, fetch time, parent, depth, and redirect history.

Deterministic extraction and model-assisted extraction are different contracts. If a host model is used, its output should be treated as untrusted until it passes the supplied schema and size limits.

4. Reach public sources — 6 capabilities

Provider integrations should say what access state they require before dispatch. “Supported” is not enough if the operator cannot tell whether a route is public, credentialed, session-backed, or unavailable.

Public GitHub repository and issue reads.
YouTube search and metadata without a developer API key through an optional reviewed route.
Official YouTube, X, and Reddit provider adapters.
Optional read-only session providers for X, Reddit, Facebook, Instagram, LinkedIn, and Xiaohongshu.
Offline RSS and Atom parsing.
Provider doctor, capability reporting, and deterministic routing.

Run the doctor before choosing a source route:

npx cockroach-sources doctor --json
npx cockroach-reach doctor --json

Optional session providers are operator-installed read routes. They do not expose posting, liking, following, messaging, deleting, cookie extraction, or personal profile discovery.

5. Connect agents — 3 capabilities

Strict creator-bounded agent tool whose model input may narrow but cannot broaden host-owned origins and budgets.
Native MCP stdio server with crawl, map, extraction, and machine-readable capability surfaces.
Optional Maqam policy, approval, trace, and evidence integration for registered crawler operations.

A minimal MCP launch keeps authority in environment configuration:

COCKROACH_ALLOWED_ORIGINS=https://docs.example.com \
COCKROACH_MAX_PAGES=10 \
npx cockroach-mcp

6. Deploy and operate — 4 capabilities

Authenticated Node.js and Docker API for health, playground, crawl, map, and extraction routes.
Responsive dashboard and browser playground for local inspection.
Bounded process-local asynchronous jobs with concurrency, pending, retained-result, and result-byte ceilings.
Fixed-origin Cloudflare Worker profile for a small deployment-configured HTTPS fetch tier.

The process-local queue is intentionally not presented as a durable distributed queue. If a workflow needs cross-machine durability, retries across restarts, or independent worker scaling, connect external infrastructure rather than pretending an in-memory queue provides it.

7. Keep authority bounded — 5 capabilities

Public-network admission and SSRF defenses that reject unsafe schemes, credentials, private ranges, and metadata destinations.
DNS pinning and explicit origin policy for the Node transport.
Exact resource ceilings across pages, requests, queue, depth, bytes, retries, redirects, callbacks, and duration.
Fixed self-hosted proxy-gateway adapter that does not accept model-selected endpoints or credentials.
Challenge-aware provider escalation that stops without access-control bypass.

The fixed-origin Worker profile is a smaller deployment tier and does not provide the Node transport’s DNS-resolution and pinning guarantees. That boundary matters when deciding where a job may run.

How to choose the smallest useful surface

Use this order:

Start with static HTTP and deterministic extraction.
Add sitemap or relevance traversal only when the queue needs better discovery.
Add browser rendering only when the target content cannot be obtained correctly without it.
Add bounded interaction only for a specific, reviewed state transition.
Add provider adapters only after capability inspection reports the expected access state.
Give an agent the strict tool or MCP surface only after the host fixes origins and budgets.
Preserve hashes, redirects, warnings, and retrieval metadata beside the content.

This makes failures easier to interpret. A robots refusal, origin denial, byte ceiling, extraction warning, provider-unavailable state, and browser challenge are not the same failure and should not collapse into “the crawler returned nothing.”

The current extraction measurement

The published 0.7.0-rc.1 quality path reports the following on the 511-page observed WCEB partition:

precision: 0.894101
recall: 0.926022
macro F1: 0.890524

That partition is labeled observed development evidence because the project had previously iterated against it. It is useful for reproducing the published candidate, but it is not an untouched confirmatory test set and it does not support a universal “0.90” or best-crawler claim.

The complete benchmark method, artifacts, hashes, and wider 1,497-page development result are published separately so the number can be inspected in context.

Inspect, reproduce, or contribute

Complete 50-capability library: https://cockroachcrawler.com/docs/capabilities/
Documentation: https://cockroachcrawler.com/docs/
Benchmark and artifacts: https://cockroachcrawler.com/benchmark/
Source and issues: https://github.com/AjnasNB/cockroach-crawler
npm stable and prerelease channels: https://www.npmjs.com/package/cockroach-crawler
Technical paper: https://doi.org/10.5281/zenodo.21851008
Contributor guide: https://github.com/AjnasNB/cockroach-crawler/blob/main/CONTRIBUTING.md

Useful contributions include documentation examples, public conformance fixtures, provider diagnostics, reproducible browser cases, and narrowly scoped bug reports with a URL pattern and expected boundary. If you test one capability, include the package version, runtime, exact input contract, observed output, and whether the route was static, browser-rendered, official-provider, or session-backed.

The goal of a capability map is not to make a crawler look bigger. It is to let a developer identify the smallest surface that does the job—and know exactly where that surface stops.

Evidence-linked memory for coding agents: a reproducible alternative to replaying project history

Ajnas N B — Sun, 09 Aug 2026 07:20:34 +0000

Coding agents often inherit project context in one of two ways:

replay a long transcript or project history, which is expensive and noisy; or
pass along a short summary, which is compact but difficult to audit.

There is a useful third design: keep an authoritative local record, derive searchable views from it, and compile a small task-specific context pack whose claims point back to exact evidence.

This article explains that pattern through Qarinah, an open-source implementation for software projects. The interesting part is not a particular CLI. It is the separation of authority, retrieval, and model-facing context.

The core idea: memory is a compiler, not a transcript

Treat retained project activity as source material rather than as the prompt itself.

permitted events and decisions
            |
            v
authoritative append-only ledger
            |
            +--> rebuildable SQLite / graph / Markdown views
            |
            v
bounded, task-specific retrieval
            |
            v
cited context pack for the next agent

The ledger answers: What was actually retained, by whom, and in what order?

The projections answer: How can we search and navigate it efficiently?

The context compiler answers: What is the smallest complete evidence set this task should receive?

Keeping those questions separate prevents a convenient index—or an opaque rolling summary—from silently becoming the source of truth.

Five architecture choices that make the pattern auditable

1. Capture is explicit and scoped

Qarinah initializes per workspace. A machine-local permit controls whether allowed metadata or reviewed content may be retained. It does not scrape hidden reasoning or private transcript stores.

That boundary matters: a memory system should not gain authority merely because an agent can call it.

2. One event chain is authoritative

Canonical JSONL events bind identifiers, provenance, confidence, typed relations, the previous hash, a content hash, and a record hash. The chain establishes continuity relative to a verified checkpoint.

It does not magically prove every recorded claim is true. Provenance and truth are different properties, and the data model keeps confidence classes such as claimed, inferred, and verified distinct.

3. Search state is disposable

SQLite FTS5, a graph, lexical indexes, Markdown views, project-structure views, and OKF exports are derived from the ledger. If a projection becomes stale, it can be rebuilt after the authoritative chain verifies.

This is a practical reliability rule: optimize the read path aggressively, but never make the optimization irreplaceable.

4. Retrieval composes relevance with authority

The retrieval path combines lexical ranking, typo tolerance, graph relations, time, freshness, repository identity, conflicts, supersession, diversity, evidence coverage, and complete-output budgets.

Optional semantic rerankers can reorder already admitted evidence, but they cannot introduce authority. This distinction lets teams experiment with retrieval without allowing a model or embedding service to widen the disclosure boundary.

5. The output carries citations and may fail closed

Each selected item records an event ID and hash. Callers can require direct evidence coverage. If the complete cited pack cannot satisfy that requirement within the configured boundary, the query can abstain rather than return a confident-looking partial memory.

For agent workflows, an explicit “insufficient evidence” result is often more useful than an uncited answer that merely sounds continuous.

A minimal local setup

Install Qarinah inside a project and opt in to the hosts and capture mode you want:

npm install --save-dev qarinah
npx qarinah setup . --codex --claude --cursor --capture content --allow-query

The --capture content flag is an explicit choice. Use metadata-only capture when event bodies should not be retained.

Record a reviewed decision, rebuild the derived views, and request a directly cited pack:

npx qarinah record \
  --kind decision \
  --title "Keep releases provenance-bound" \
  --body "Publish only the reviewed artifact."

npx qarinah scan
npx qarinah build
npx qarinah query "release provenance" \
  --minimum-coverage direct \
  --format markdown
npx qarinah doctor

The important behavior is not that a search result exists. It is that the result fits a declared budget, exposes its evidence identities, and can be regenerated from the verified local record.

Measuring context reduction without overstating it

“Uses fewer tokens” is easy to claim and surprisingly easy to measure badly. A useful evaluation must say exactly what was replaced and what remained constant.

Qarinah's committed software-task fixture creates 240 retained project-history records and evaluates six reproducible scenarios:

a React accessibility edit;
a database schema migration;
a repository-wide TypeScript refactor;
web research leading to implementation;
production regression debugging; and
governed release preparation.

For every scenario, both paths receive the same current-task source snippets. The baseline additionally receives the complete retained history; the Qarinah path receives the cited pack compiled for that task.

The evaluator uses the portable estimate ceil(characters / 4). It is deterministic, but it is not a provider billing receipt.

Compared context slice	Estimated tokens
Full-history replay across six tasks	442,113
Qarinah packs plus the identical task sources	5,682
Reduction	98.7148%
Compression ratio	77.81:1

Every required target in this fixture had direct evidence coverage in the top five, and the packs contained zero model-written summary records.

Those qualifications belong beside the number. The result measures repeated input-context volume on this fixed fixture. It does not establish universal model quality, latency, provider-native usage, total application cost, or the same reduction on every repository.

You can inspect the benchmark method and limitations, the machine-readable six-task result, and the evaluator in the repository. To reproduce it from source:

git clone https://github.com/AjnasNB/qarinah.git
cd qarinah
npm ci
npm run evaluate:software-tasks

What this design changes in an agent workflow

A cross-agent handoff no longer has to mean “copy the conversation.” The next tool can receive a compact pack containing the relevant decision, implementation outcome, conflict, source identity, and evidence hash.

That has several useful consequences:

Agent changes are less disruptive. Codex, Claude Code, Cursor, a CLI client, or an MCP client can query the same project-local record through supported integration boundaries.
Derived memory is reviewable. A human can inspect the Markdown or dashboard view without treating it as a second authority.
Corrections stay visible. Supersession and contradiction are relations, not destructive edits to history.
Temporal queries become possible. A query can resolve what was valid at a chosen time rather than flattening every old and new claim together.
Repository boundaries remain explicit. Cross-repository relationships aid navigation without merging permissions or evidence authority.

This is not a replacement for source code, tests, logs, or the current task's working set. It is a way to avoid repeatedly shipping unrelated retained history alongside those sources.

Design lessons you can reuse without Qarinah

If you are building your own agent-memory layer, four questions expose most architectural problems:

What is authoritative? If the vector database disappears, can you reconstruct the memory from a verified record?
Who grants capture and disclosure authority? Can agent input widen its own scope?
Can every model-facing item name its evidence? A citation should be part of the output contract, not a post-processing decoration.
What exactly does the benchmark replace? Keep current-task sources constant, publish the estimator, commit the fixtures, and state what the result does not prove.

A smaller prompt is valuable. A smaller prompt whose contents can be traced, rebuilt, and refused when incomplete is much more useful.

Explore, reproduce, or contribute

Qarinah was created and is maintained by Ajnas N B. Contributions are welcome—especially reproducible fixtures, retrieval edge cases, integration feedback, documentation improvements, and reviews of evidence or privacy boundaries.

Three open-source building blocks for AI agents: memory, safer tools, and web access

Ajnas N B — Mon, 27 Jul 2026 11:02:43 +0000

AI agents are becoming capable enough to work across whole projects, but three practical gaps keep appearing: they forget why decisions were made, tool approvals can drift away from the action that was reviewed, and web access is often either too limited or too broad.

I have been building three open-source projects around those gaps.

Qarinah - evidence-linked project memory

Qarinah keeps project decisions, code structure, tool outcomes, approvals, and sources in a local record. For each task it compiles a compact, cited context pack instead of replaying the entire project history.

Website: https://qarinah.io

GitHub: https://github.com/AjnasNB/qarinah

Maqam - policy and exact approval for tool calls

Maqam checks policy before registered operations, binds approval to the exact run, tool, and input, and records traces and evidence after execution. It is designed to sit around agent workflows rather than replace the framework or model you already use.

Website: https://maqamagent.com

GitHub: https://github.com/AjnasNB/maqam

Cockroach Crawler - web access for AI agents

Cockroach Crawler is a JavaScript toolkit for static and browser crawling, structured extraction, PDFs, screenshots, site maps, source adapters, MCP, Docker, and agent integrations. It gives agents useful web reach while keeping network authority and resource limits with the host application.

Website: https://cockroachcrawler.com

GitHub: https://github.com/AjnasNB/cockroach-crawler

Why they fit together

Qarinah remembers what happened and why. Maqam decides whether a registered action may run. Cockroach Crawler collects web evidence the agent can use. Each project is independently usable, but together they form a practical memory, action, and research layer for agent systems.

All three are open source. I am looking for developers who will run them in real workflows, report failures, challenge the boundaries, improve adapters and documentation, and contribute code.

If you build with AI agents, which problem is most painful in your current stack: memory, tool safety, or web access?

I stopped replaying my entire codebase to Codex and Claude Code

Ajnas N B — Sat, 25 Jul 2026 14:13:03 +0000

I kept hitting the same problem in long Codex and Claude Code sessions: the agent was capable, but the project memory was expensive.

A new task would trigger another repository scan. Decisions from older sessions had to be explained again. Tool results were buried in transcripts. A summary was smaller, but I could not always tell which source supported a statement or whether a newer decision had replaced it.

So I built Qarinah, an Apache-2.0 local project-memory compiler for coding agents.

The problem is not only context size

A large context window can hold more text, but it does not answer four important questions:

Which facts matter to the current task?
Where did each fact come from?
Has a newer decision superseded an older one?
Can another coding agent verify the same answer?

Replaying the full history preserves information, but it also sends large amounts of irrelevant material. A single generated summary is smaller, but it becomes another authority that can omit or blur evidence. Ordinary embedding retrieval finds similar text, but similarity is not the same as authority, recency, or approval.

What Qarinah does differently

Qarinah keeps the source record and the context pack separate.

Permitted decisions, tool outcomes, approvals, project structure, and evidence become typed events. Relations connect records such as supersedes, supports, blocks, and produced-by. Deterministic Markdown and JSON views remain inspectable in the repository. When a task begins, Qarinah retrieves a bounded pack and includes citations back to event IDs, hashes, and sources.

The result is not one giant memory prompt. It is a small evidence-linked pack built for the current task.

Approach	Useful for	What remains difficult
Full-history replay	Maximum raw recall	Repeated tokens, noise, stale decisions
AGENTS.md or CLAUDE.md	Stable instructions	Tool outcomes and changing decisions
Embedding retrieval	Similar passages	Authority, supersession, exact provenance
Generated summary	Small handoff	Verifying omissions and source claims
Qarinah	Compact project memory	Requires explicit workspace opt-in and capture policy

Reproduce the basic flow

Install it inside one project:

npm install --save-dev qarinah
npx qarinah init . --capture content
npx qarinah scan
npx qarinah doctor

Record a decision:

npx qarinah record --kind decision \
  --title "Use additive database migrations" \
  --body "Add, backfill, switch, then remove. Never rename a production column in place."

Then Codex, Claude Code, a CLI workflow, or an MCP client can request the smallest cited pack relevant to a task. The project owns the files. Qarinah does not require uploading the repository to a hosted Qarinah backend.

What the published evaluator measured

The committed evaluator compared full-history input with bounded Qarinah packs across six software tasks covering React editing, database migration, TypeScript refactoring, web research, production debugging, and governed release work.

The compared input changed from 442,113 estimated input-context tokens to 5,682. That is 98.71% less repeated context, or 77.81:1 compression, while every required target ranked in the top five. The repository includes the inputs, arithmetic, methodology, and machine-readable result.

This is a result for that task set, not a promise that every repository or total provider bill drops by exactly 98.71%. Output tokens, caching, tool calls, and provider pricing are separate.

Codex and Claude Code

Qarinah ships installable plugins for both hosts.

Codex:

codex plugin marketplace add AjnasNB/qarinah --ref v0.1.1
codex plugin add qarinah@qarinah

Claude Code:

claude plugin marketplace add AjnasNB/qarinah@v0.1.1 --scope user
claude plugin install qarinah@qarinah --scope user

Both can read the same opted-in local project memory. This does not copy private provider chat history between services. It gives both agents access to the same project-owned evidence.

What I would like help testing

Qarinah is open source and early. I am looking for real failure cases, especially:

repositories with conflicting or superseded architecture decisions
multi-agent handoffs
large monorepos
trust-policy and provenance review
retrieval quality under very small token budgets
Codex and Claude Code interoperability

Website: https://qarinah.io/

Source: https://github.com/AjnasNB/qarinah

Technical paper: https://qarinah.io/paper/

If you test it, I would rather receive one reproducible failure than ten generic compliments. That is the fastest way to make the project useful.

Solution for the Web Access Dilemma for AI Agents Too Much Power, Too Little Control

Ajnas N B — Fri, 24 Jul 2026 18:48:02 +0000

AI agents can reason about the web.

But giving an agent unrestricted browser or network access creates a serious authority problem.

The obvious solution is to restrict the tools available to the agent.

Then I kept running into the opposite problem:

Once the tool became sufficiently restricted, it lost many of the capabilities required to complete real work.

I wanted both sides:

Enough power to crawl, render, navigate, extract, capture, and investigate the web
Explicit operator control over origins, credentials, budgets, browser hooks, profiles, and evidence

So I built Cockroach Crawler.

It is an open-source Node.js and TypeScript toolkit for AI agents, RAG pipelines, documentation indexing, research, QA, and web-data workflows.

I connected it to OpenClaw and Codex, and the difference was honestly wild.

Instead of giving the agents one narrow search tool, I gave them a bounded web-research layer that could crawl websites, inspect JavaScript applications, extract structured data, process PDFs, take screenshots, generate PDFs, inspect public sources, and return evidence with provenance.

And for many public workflows, I did not need to configure a separate API key for every source.

GitHub: https://github.com/AjnasNB/cockroach-crawler
Documentation: https://cockroachcrawler.com/docs/
npm: https://www.npmjs.com/package/cockroach-crawler

What changed after I connected it to OpenClaw and Codex?

Before this, the agents could reason well, but their web access was limited.

They could answer questions, write code, and work with the context I gave them. But once a task required deeper live-web investigation, I still had to manually combine several tools.

After connecting Cockroach Crawler, they could:

Crawl public websites
Render JavaScript-heavy pages
Follow sitemaps
Search and map documentation sites
Extract readable Markdown
Extract structured fields with CSS, XPath, or restricted regular expressions
Read local and remote PDFs
Generate PDFs
Take screenshots
Handle bounded clicks and scrolling
Inspect open Shadow DOM
Read same-origin iframe content
Collect canonical URLs and redirect history
Preserve content hashes and retrieval metadata
Return warnings, failures, and crawl statistics
Work through MCP
Use Docker or a local dashboard
Inspect public provider availability before dispatch

It stopped feeling like I had given the agents a simple browser.

It felt like I had given them an actual web-research system.

The core idea

The design principle is simple:

Give AI agents the web. Keep the keys.

The agent gets useful capabilities.

The host application keeps authority.

That means model-facing input can narrow a job, but it cannot silently add:

New origins
New credentials
New proxy endpoints
New browser hooks
New persistent profiles
Larger resource limits
Broader network authority

This matters because an agent should not be able to expand its own permissions simply by generating a more aggressive tool call.

What can Cockroach Crawler do?

Cockroach Crawler currently provides 50 documented capabilities across crawling, browser automation, extraction, providers, agent integration, deployment, and security.

Crawling and discovery

Static HTTP crawling
Multiple crawl seeds
Breadth-first traversal
Depth-first traversal
Best-first traversal
Adaptive relevance traversal
Sitemap discovery
Include and exclude filters
robots.txt enforcement
Redirect validation
Persistent hash-verified cache
Searchable site maps
Crawl deadlines and cancellation
Explicit concurrency and politeness controls

Browser rendering and capture

JavaScript rendering with Chromium
Selector waits
Bounded clicks
Infinite and virtual scrolling
Open Shadow DOM flattening
Readable same-origin iframe flattening
Screenshots
PDF generation
Persistent browser profiles
Explicit operator-supplied page hooks

Extraction

Readable Markdown
CSS extraction
XPath extraction
Restricted regular-expression extraction
Optional host-model schema extraction
Local PDF parsing
Links and metadata
Content hashes
Provenance records
Retrieval warnings and failures

Public sources and providers

Public GitHub reads
YouTube search without a developer API key through an optional reviewed provider
RSS and Atom parsing
Official provider routes
Read-only session routes
Provider doctor and deterministic routing

Agent integration

Native MCP server
Strict agent-tool adapter
Capability inspection
Maqam integration for policy, approvals, traces, and evidence

Deployment

Authenticated Docker API
Dashboard and playground
Bounded asynchronous jobs
Restricted Cloudflare Worker deployment

Security and authority

Public-network admission
Private-network blocking
Redirect validation
DNS validation
Explicit origin policy
Credential isolation
Resource ceilings
Challenge-aware escalation

Start with one command

Install the package:

npm install cockroach-crawler@0.5.2

Run a bounded crawl:

npx cockroach-crawl https://example.com/docs \
  --max-pages 20 \
  --max-requests 80 \
  --jsonl

The result includes extracted content plus the information an agent or developer needs to verify it:

Canonical URL
Parent URL
Crawl depth
Redirect history
Response status
Content type
Content hash
Retrieval metadata
Warnings
Failures
Crawl statistics

Use it from JavaScript

import { crawlDetailed } from "cockroach-crawler";

const result = await crawlDetailed({
  seeds: ["https://example.com/docs"],
  maxPages: 20,
  maxRequests: 80,
  maxDurationMs: 60_000
});

for (const page of result.pages) {
  console.log(page.url);
  console.log(page.text);
  console.log(page.contentHash);
}

console.log(result.failures);
console.log(result.stats);

The application owns the limits.

The model can request less.

It cannot silently request more.

Deep crawling strategies

Different jobs need different traversal behavior.

Breadth-first crawling

Use breadth-first crawling when you want broad coverage near the seed:

const result = await crawlDetailed({
  seeds: ["https://example.com/docs"],
  strategy: "bfs",
  maxPages: 50
});

Depth-first crawling

Use depth-first crawling for narrow or hierarchical documentation paths:

const result = await crawlDetailed({
  seeds: ["https://example.com/docs"],
  strategy: "dfs",
  maxPages: 50
});

Best-first and adaptive relevance crawling

Best-first and adaptive traversal prioritize pages that appear more useful for the current task while remaining inside the configured crawl budget.

This is useful when an agent needs to investigate a large documentation site without blindly downloading everything.

Render JavaScript applications

Static HTTP is fast, but many modern applications require a real browser.

Cockroach Crawler provides optional Playwright integration for:

JavaScript rendering
Waiting for selectors or page states
Explicit clicks
Bounded scrolling
Shadow DOM inspection
Readable same-origin frames
Screenshots
PDF output
Dedicated profiles
Reviewed page hooks

Install the optional browser peer:

npm install cockroach-crawler playwright
npx playwright install chromium

Browser mode still uses explicit origins and resource ceilings.

Enabling Chromium does not give the agent unrestricted operating-system or network authority.

Extract structured data

Readable Markdown is useful for agents and RAG pipelines.

But many applications need deterministic fields.

Cockroach Crawler supports bounded:

CSS extraction
XPath extraction
Restricted regular-expression extraction
Optional model-assisted schema extraction

The model-assisted route still remains host-controlled.

The operator supplies the model adapter, and the final output must pass JSON Schema validation.

Model input cannot choose arbitrary credentials, origins, hooks, or execution authority.

Search and map an entire site

The crawler can create compact, fetch-validated site maps and optionally search them.

This is useful for:

Documentation discovery
Migration inventories
Support knowledge bases
RAG ingestion
Broken-link investigation
Product audits
Content audits

Unlike a sitemap assembled only from discovered links, entries can retain retrieval state and source identity for later verification.

Connect it to OpenClaw, Codex, or another agent through MCP

Cockroach Crawler includes a native MCP stdio server.

Example configuration:

{
  "mcpServers": {
    "cockroach-crawler": {
      "command": "npx",
      "args": [
        "-y",
        "cockroach-crawler@0.5.2",
        "cockroach-mcp"
      ]
    }
  }
}

The MCP surface exposes bounded:

Crawling
Site mapping
Structured extraction
Capability inspection

The MCP client does not receive control over operator-owned credentials, browser hooks, proxy endpoints, persistent profiles, or origin ceilings.

That was the important part for me.

I wanted OpenClaw and Codex to become much more capable without allowing the model to decide its own security boundary.

Inspect providers before dispatch

The provider doctor reports what the current machine can actually use:

npx -y --package cockroach-crawler@0.5.2 \
  cockroach-sources doctor --json

It distinguishes between:

Public routes
Official API routes
Optional no-key routes
Explicit read-only session routes
Missing configuration
Unsupported operations

A provider is not considered available just because its name appears in a configuration file.

The route must actually be usable in the current environment.

Docker and self-hosting

Cockroach Crawler includes an authenticated Node.js and Docker API with a dashboard and playground.

It can also run bounded process-local jobs with:

Status inspection
Cancellation
Result ceilings
Authentication
Fixed deployment authority

For edge deployments, the package includes a separate Cloudflare Worker profile for fixed deployment-owned HTTPS origins.

That Worker is intentionally treated as a different security boundary from the local Node.js crawler.

Public benchmark evidence

The current package was evaluated on all 511 held-out pages of the pinned WCEB v1.0 test split.

The published run produced:

0.7653 macro word F1
0.9041 recall
87.13% required-snippet recall
25/25 adapted robots dispatch vectors passed
101/101 applicable credential-free WPT HTTP(S) URL cases passed

The evaluator, dataset revision, source fingerprint, page-level rows, and machine-readable results are committed to the repository.

These results describe performance on the named test corpus.

They are not a claim of universal extraction quality.

Benchmark details:

https://cockroachcrawler.com/benchmark/

What it is not designed to do

Cockroach Crawler is not designed to bypass:

CAPTCHAs
Access controls
Private-network restrictions
Explicit robots policies
Authentication boundaries
Platform challenges

It is designed to provide useful web capabilities while keeping authority visible, bounded, and operator-owned.

Why this made OpenClaw and Codex feel completely different

The real improvement was not simply that they could fetch more pages.

It was that they could perform an entire evidence-backed workflow:

Discover the relevant pages
Render the pages when necessary
Extract the useful content
Process PDFs
Capture screenshots or generated PDFs
Preserve canonical URLs and hashes
Return failures instead of silently hiding them
Stay within explicit origin and budget limits

That changed the agents from systems that merely talked about the web into systems that could actually investigate it.

And honestly, this was the first time it felt like I had given them the whole internet without giving away the keys.

Try to break it

If you are building an AI agent, research system, RAG pipeline, documentation indexer, or browser-assisted workflow, test it on something real.

Open an issue with:

Operating system
Node.js version
Public test URL
Smallest reproducible command
Expected behavior
Actual behavior

Install:

npm install cockroach-crawler@0.5.2

Links:

GitHub: https://github.com/AjnasNB/cockroach-crawler
Documentation: https://cockroachcrawler.com/docs/
Capability index: https://cockroachcrawler.com/docs/capabilities/
Security model: https://cockroachcrawler.com/security/
Benchmark: https://cockroachcrawler.com/benchmark/
npm: https://www.npmjs.com/package/cockroach-crawler

Give AI agents the web. Keep the keys.