DEV Community: Khalid Hussein

html-to-markdown v3.8.0

Khalid Hussein — Tue, 30 Jun 2026 03:58:34 +0000

TL;DR

Package rebrand. Every published identifier moves from kreuzberg-namespaced to xberg-namespaced — npm, Maven, NuGet, Homebrew tap, GitHub org. See the migration table.

Node binding consolidated. The separate -node package and TypeScript wrapper under packages/typescript/ are removed; the NAPI-RS crate now ships directly as @xberg-io/html-to-markdown.

PHP install path changed. Use pie install xberg-io/html-to-markdown — composer require cannot load a native ext-php-rs extension.

What Is html-to-markdown?

html-to-markdown is a CommonMark-compliant HTML→Markdown converter with a Rust core, maintained by the Xberg team. It exposes a single consistent API across 14 language bindings and ships as a CLI, a library, and an MCP server.

What Changed in v3.8.0

v3.8.0 promotes v3.8.0-rc.2 and is a rebrand + package consolidation release. The converter's behavior, output format, and core API are unchanged.

Package identity rebrand: Kreuzberg → Xberg

Every published artifact identifier has moved:

Artifact	Old	New
npm (Node)	`@kreuzberg/html-to-markdown-node`	`@xberg-io/html-to-markdown`
npm (WASM)	`@kreuzberg/html-to-markdown-wasm`	`@xberg-io/html-to-markdown-wasm`
npm (CLI)	`@kreuzberg/html-to-markdown-cli`	`@xberg-io/html-to-markdown-cli`
Maven groupId	`dev.kreuzberg`	`io.xberg`
Maven artifactId	`html-to-markdown`	`html-to-markdown` (groupId only changes)
JNI symbols	`Java_dev_kreuzberg_android_…`	`Java_io_xberg_android_…`
NuGet	`KreuzbergDev.HtmlToMarkdown`	`XbergIo.HtmlToMarkdown`
Homebrew tap	`kreuzberg-dev/homebrew-tap`	`xberg-io/homebrew-tap`
GitHub org	`github.com/kreuzberg-dev`	`github.com/xberg-io`

Node binding consolidation

The separate -node wrapper package and the TypeScript layer under packages/typescript/ are removed. The NAPI-RS crate now ships directly as @xberg-io/html-to-markdown, with platform packages named @xberg-io/html-to-markdown-<platform>. If you were importing from the old wrapper, update to the direct package — the API surface is the same.

Swift publish branch fix

The same fix applied to Crawlberg v1.0.0 lands here: the publish workflow now creates the release/swift/3.8.0 branch carrying the substituted XCFramework checksum. SwiftPM packages that pin via .package(url: …, branch: "release/swift/<version>") will now resolve correctly.

Migration Guide

1. Update package declarations

# Node.js — was @kreuzberg/html-to-markdown-node
npm install @xberg-io/html-to-markdown

# WASM — was @kreuzberg/html-to-markdown-wasm
npm install @xberg-io/html-to-markdown-wasm

# PHP — was composer require kreuzberg/html-to-markdown
pie install xberg-io/html-to-markdown

<!-- Java/Kotlin — groupId changes -->
<dependency>
  <groupId>io.xberg</groupId>
  <artifactId>html-to-markdown</artifactId>
  <version>3.8.0</version>
</dependency>

# C# — was KreuzbergDev.HtmlToMarkdown
dotnet add package XbergIo.HtmlToMarkdown

2. PHP: switch from composer require to pie install

composer require cannot install a native ext-php-rs extension. Any existing install instructions using composer require kreuzberg/html-to-markdown will silently fail to load the extension. Use PIE:

pie install xberg-io/html-to-markdown

3. Update Homebrew tap

brew untap kreuzberg-dev/homebrew-tap
brew tap xberg-io/homebrew-tap
brew install html-to-markdown

Verification checklist

[ ] html-to-markdown --version prints 3.8.0
[ ] Node import from @xberg-io/html-to-markdown resolves
[ ] PHP extension loads (php -m | grep html_to_markdown)
[ ] Swift package resolves via SPM

What Else Is in Recent Releases

v3.8.0 carries no new conversion features. The last substantive feature release was v3.7.0, which expanded the MCP server to full API parity: a typed ConvertConfig object covering every ConversionOptions field, a new extract_metadata tool returning structured Open Graph / JSON-LD / Twitter Card metadata as JSON, and MCP prompts, resources, and completions capabilities. If you are integrating html-to-markdown into an AI agent pipeline, the MCP server is production-ready as of 3.7.0.

Learn more

Release: github.com/xberg-io/html-to-markdown/releases/tag/v3.8.0
Repository: github.com/xberg-io/html-to-markdown
Documentation: docs.html-to-markdown.xberg.io
Getting started: pip install html-to-markdown · npm install @xberg-io/html-to-markdown · cargo add html-to-markdown

Introducing Crawlberg v1.0.0

Khalid Hussein — Mon, 29 Jun 2026 03:53:27 +0000

We're upgrading Crawlberg to a new version: Crawlberg v1.0.0. It builds on the previous kreuzcrawl. It declares the public API frozen under the new project name. All technical features below shipped in v0.3.0 (2026-06-23); v1.0.0 is a stability declaration and rename, not a new feature release.

The four production-facing changes most likely to require operational action:

Package and env var rename - every artifact identifier has changed; see the migration table.
SSRF defense is now on by default - internal crawl targets (localhost, RFC 1918, cloud metadata) will fail without CRAWLBERG_ALLOW_PRIVATE_NETWORK=1.
CrawlError::WafBlocked is now a struct variant - exhaustive match arms will not compile until updated.
max_retries semantics changed - off-by-one fixed; max_retries=3 now produces exactly 3 retries.

Precompiled binaries cover Linux (x86_64/aarch64), macOS (ARM64 and x86_64), and Windows x64. Homebrew bottles and Docker images on GHCR are also available.

What Is Crawlberg?

Crawlberg is a web crawling engine written primarily in Rust that exposes a single consistent API across 14 language runtimes. It handles HTTP transport, JavaScript rendering, robots.txt compliance, per-domain rate limiting, SSRF safety, and structured extraction. Extension points (Frontier, RateLimiter, CrawlStore, EventEmitter, ContentFilter, WafClassifier, ProxyProvider) are injectable traits; wire in your own frontier, storage backend, or proxy pool without forking the engine.

A single scrape() call returns text, metadata, links, images, assets, JSON-LD, Open Graph tags, hreflang, favicons, headings, response headers, and clean HTML→Markdown. When a site requires JavaScript, the optional headless browser tier handles it transparently.

v1.0.0 promotes v1.0.0-rc.2 and freezes the public API under the new project name. The features described in the sections below represent the platform that 1.0.0 declares stable; they shipped in v0.3.0.

What v1.0.0 Declares Stable

These capabilities shipped in v0.3.0 (2026-06-23). v1.0.0 freezes their API and declares them production-stable under the new crawlberg package name. Engineers running 0.3.0 already have the runtime features; upgrading to 1.0.0 means: rename packages, update env vars, get the stable API contract.

Project rename: `kreuzcrawl` → `crawlberg`

The most operationally significant change is the rename. Every artifact identifier has changed:

Artifact	Old	New
Crate (crates.io)	`kreuzcrawl`	`crawlberg`
PyPI	`kreuzcrawl`	`crawlberg`
npm	`@kreuzberg/crawl`	`@xberg-io/crawlberg`
Composer	`kreuzberg/kreuzcrawl`	`xberg-io/crawlberg`
Maven groupId	`dev.kreuzberg`	`io.xberg.crawlberg`
NuGet	`KreuzbergDev.KreuzCrawl`	`XbergIo.Crawlberg` (see note below)
Go module	`github.com/kreuzberg-dev/kreuzcrawl/...`	`github.com/xberg-io/crawlberg/packages/go`
C FFI symbol prefix	`kcrawl_*`	`cberg_*`
Env vars	`KREUZBERG_*`	`CRAWLBERG_*`
Docs	`docs.kreuzcrawl.kreuzberg.dev`	`docs.crawlberg.xberg.io`

Behavior and API shape are identical. This is a rename, not a rewrite.

Tiered dispatch engine

The crawl engine chains HTTP → bypass → headless browser, driven by per-attempt signals rather than a static configuration flag. When a response indicates a WAF challenge, the engine escalates; when it succeeds, it records the outcome in per-domain state and adjusts the starting tier for subsequent visits.

Public types: Tier, EscalationStrategy, EscalationReason, AttemptOutcome, RetryDirective, RetryPolicy, WafSignal, DispatchProfile. All dispatch enums are #[non_exhaustive] — future tiers are non-breaking additions.

WAF detection with hot-reload fingerprints

A TOML fingerprint corpus (rules/waf_fingerprints.toml, 34 fingerprints) feeds an Aho-Corasick automaton. TomlClassifier::watch() watches the file with a debounced watcher and swaps the compiled automaton atomically via ArcSwap — no process restart needed. This is safe for Kubernetes ConfigMap updates: mount the TOML as a ConfigMap volume, edit it, and the running engine picks up the new corpus within seconds.

Per-domain block rates are tracked with EwmaDomainState, an exponentially weighted moving average that automatically promotes or demotes the starting tier based on recent history.

SSRF defense, on by default

Every fetch path runs URL validation before the network call and after each redirect hop. Blocked address ranges:

127.0.0.0/8 (loopback)
RFC 1918 private ranges (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16)
169.254.0.0/16 (link-local, including cloud metadata endpoints such as 169.254.169.254)
0.0.0.0/8 (this-network/reserved per RFC 1122 §3.2.1.3)
224.0.0.0/4 (multicast)
IPv6 ULA fc00::/7, link-local fe80::/10, multicast ff00::/8
Any non-http(s) scheme

Three protection layers work together:

DNS-rebinding mitigation: every resolved IP must pass the policy, not just the hostname at call time.
Redirect-chain re-validation: each hop re-resolves and re-validates, bounded by ssrf.max_redirects (default 5).
Link-enqueue validation: URLs are validated against the SSRF policy before being added to the crawl frontier, not only at fetch time.

Allowlisting is available via HostMatcher (Exact/Suffix/Cidr variants). Opt out entirely with CRAWLBERG_ALLOW_PRIVATE_NETWORK=1.

Memory-bounded streaming crawl

crawl_stream() and batch_crawl_stream() previously accumulated every CrawlEvent::Page in memory. They now yield each page and drop it immediately. Based on internal measurements documented in the changelog, peak working-set drops from approximately 2.5 GB to approximately 20 MB on large crawls. The batch crawl() API; which returns all pages at once; is unchanged.

MCP server and full CLI parity

The CLI exposes batch-scrape, batch-crawl, download, citations, and version; 1:1 with the core and MCP surfaces. The MCP server serves tools over both stdio and rmcp Streamable HTTP at /mcp. HTTP transport requires the binary to be compiled with the api and mcp Cargo features; the release CLI binary includes both. Each tool carries read_only/destructive/open_world safety annotations for agent orchestration frameworks that need to reason about side effects before calling tools.

Public substrate parsers

crawlberg::robots and crawlberg::sitemap are now public modules, usable without spinning up the full crawl engine. parse_robots_txt, is_path_allowed, RobotsRules, parse_sitemap_xml, parse_sitemap_index, and is_sitemap_index are all available standalone; useful for robots/sitemap preprocessing in pipelines that manage their own fetch layer.

Deep Technical Highlights

Escalation budget injection

EscalationBudget is a user-injectable trait. You can implement per-domain, per-hour browser budget caps, or tie escalation policy to real-time proxy cost signals. The built-in EwmaDomainState is designed for zero-configuration deployment; the trait interface is designed for when you have stronger opinions.

Lock-free corpus hot-reload

The WAF automaton lives under an ArcSwap. Readers take a guard for the duration of a single classification call; on the order of microseconds; and never contend with the writer. The writer side compiles a new automaton (tens of milliseconds for a large corpus) and swaps it in a single atomic store. In-flight requests complete against the old automaton; new requests use the new one immediately. Readers never block on corpus updates.

SSRF at every redirect hop

URL validation at the call site alone is insufficient: a hostname can pass the initial check, then DNS resolves to a private address after a short TTL expires (DNS rebinding). Crawlberg re-resolves and re-validates at every redirect, bounded by ssrf.max_redirects (default 5). The SsrfPolicy::from_env serde default means CrawlConfig deserialized from JSON automatically honors the environment variable; important for container deployments where env vars are the primary configuration channel.

Browser pool lifecycle

BrowserPool is public. Construct and warm(n) the pool at startup; pass it via CrawlEngineBuilder::with_browser_pool(). Browser instances are reused across crawl jobs rather than spawned per escalation event. CrawlEngineHandle::from_engine() produces a cloneable handle, so multiple async tasks can share a single engine and pool without additional coordination.

Asset downloads through the SSRF filter

Before this release, download_documents was honored only by single-page scrape(); the crawl loop fetched, flagged, and discarded the bytes. Downloads now route through http_fetch; the same transport as page fetches; so every file download is subject to the SSRF policy and per-domain rate limiting.

Performance Implications

The most directly measurable change is streaming memory: ~2.5 GB → ~20 MB peak working-set on large crawls (figures from the changelog; no external benchmark suite has been published for this release). The practical implication is that crawl corpus size is no longer bounded by available RAM when using crawl_stream.

Throughput: the tiered dispatch model adds a small latency overhead for requests that escalate; one additional HTTP probe before browser spin-up. Domains that respond normally to plain HTTP never pay this cost. EWMA per-domain state promotes well-behaved domains to start at the HTTP tier, avoiding unnecessary bypass or browser escalation for clean domains.

The ArcSwap-backed corpus reload is lock-free from the reader's perspective, so fingerprint corpus updates do not introduce latency spikes in production.

No benchmark numbers for throughput, requests-per-second, or latency percentiles are published in this release. Teams evaluating Crawlberg for high-throughput workloads should run their own benchmarks against the stable 1.0.0 surface; the Criterion benchmarks in the repository cover the WAF subsystem and are a starting point for extending coverage.

Language Bindings Spotlight

All 14 bindings are generated from the same Rust core by alef and contain no per-language extraction logic. The code snippets below are illustrative; check the per-language READMEs for exact API signatures.

Python

from crawlberg import CrawlConfig, scrape, crawl_stream

config = CrawlConfig(max_depth=3, max_pages=500, concurrency=20)

# Single-page extraction
result = await scrape("https://example.com", config=config)
print(result.markdown)

# Memory-bounded streaming crawl
async for event in crawl_stream("https://example.com", config=config):
    if event.page:
        print(event.page.url, event.page.title)

pip install crawlberg

JavaScript / Node.js

import { scrape, crawlStream, CrawlConfig } from "@xberg-io/crawlberg";

const config = new CrawlConfig({ maxDepth: 3, maxPages: 500, concurrency: 20 });

const result = await scrape("https://example.com", { config });
console.log(result.markdown);

for await (const event of crawlStream("https://example.com", { config })) {
  if (event.page) console.log(event.page.url, event.page.title);
}

npm install @xberg-io/crawlberg

PHP

use XbergIo\Crawlberg\CrawlConfig;
use XbergIo\Crawlberg\Crawlberg;

$config = (new CrawlConfig())
    ->withMaxDepth(3)
    ->withMaxPages(500)
    ->withConcurrency(20);

$result = Crawlberg::scrape('https://example.com', $config);
echo $result->markdown;

composer require xberg-io/crawlberg

PHP 8.2, 8.3, and 8.4 are supported; precompiled NTS extensions ship for Linux (glibc, aarch64/x86_64), macOS (ARM64/x86_64), and Windows (VS16/VS17 x86_64).

Breaking Changes / Compatibility Notes

v1.0.0 is a breaking release for users of pre-release kreuzcrawl / kreuzberg-namespaced packages. For users already on the crawlberg name from 0.x pre-releases, the behavioral breaking changes are:

Area	Old behavior	New behavior	Action required
Package identifiers	`kreuzcrawl` everywhere	`crawlberg` / `@xberg-io/crawlberg` etc.	Update dependency declarations in all manifests
Env vars	`KREUZBERG_*`	`CRAWLBERG_*`	Update shell configs, CI env blocks, K8s Secrets
C FFI symbols	`kcrawl_*`	`cberg_*`	Recompile; update header includes and linker references
Go module path	`github.com/kreuzberg-dev/kreuzcrawl/...`	`github.com/xberg-io/crawlberg/packages/go`	`go get` new path; update all import statements
`CrawlError::WafBlocked`	Unit variant	Struct variant `{ vendor, message }`	Update match arms to destructure
`NetworkErrorKind`	Exhaustive enum	`#[non_exhaustive]` applied	Add wildcard `_` arms to exhaustive matches; recompile
`CrawlError` / dispatch enums	Exhaustive enums	`#[non_exhaustive]` applied	Add wildcard `_` arms to exhaustive matches; recompile
`SimpleRetryPolicy`	`max_retries=3` → 2 actual retries (off-by-one)	`max_retries=3` → 3 actual retries (fixed)	Audit retry budgets if behavior depended on the old count
`DomainStatePort`	Mutation model	Observation model (`recommend`/`observe`)	Update trait implementations if you implemented this trait
SSRF policy	Disabled by default	Enabled by default	Add `CRAWLBERG_ALLOW_PRIVATE_NETWORK=1` for internal crawls

Upgrade Guide

Step-by-step for existing `kreuzcrawl` users

1. Update package declarations

# Python
pip install crawlberg        # replaces kreuzcrawl

# Node.js
npm install @xberg-io/crawlberg   # replaces @kreuzberg/crawl

# PHP
composer require xberg-io/crawlberg   # replaces kreuzberg/kreuzcrawl

# Go
go get github.com/xberg-io/crawlberg/packages/go@v1.0.0

# Rust
cargo add crawlberg@1.0.0    # replaces kreuzcrawl

# C# - verify the exact package ID in the C# README before running
dotnet add package XbergIo.Crawlberg

2. Update environment variables and configuration

# Old
KREUZBERG_ALLOW_PRIVATE_NETWORK=1
# New
CRAWLBERG_ALLOW_PRIVATE_NETWORK=1

3. Audit SSRF settings before first run

If you crawl internal networks (CI test targets, internal APIs, localhost), set CRAWLBERG_ALLOW_PRIVATE_NETWORK=1 before upgrading. Without it, requests to RFC 1918 and loopback addresses will fail with CrawlError::SsrfPolicyViolation.

4. Update C FFI call sites (if applicable)

Replace all kcrawl_ symbol references with cberg_. Regenerate cbindgen headers.

5. Fix CrawlError::WafBlocked match arms

// Old - unit variant
CrawlError::WafBlocked => { /* ... */ }

// New - struct variant
CrawlError::WafBlocked { vendor, message } => {
    eprintln!("WAF block: {vendor}: {message}");
}

6. Add wildcard arms for #[non_exhaustive] enums

CrawlError, NetworkErrorKind, and all dispatch enums (EscalationReason, EscalationStrategy, etc.) are now #[non_exhaustive]. Any exhaustive match on these types will fail to compile until a wildcard _ => { ... } arm is added.

Verification checklist

[ ] crawlberg --version prints 1.0.0
[ ] scrape() returns a result for a known public URL
[ ] A streaming crawl over a multi-page site completes without OOM
[ ] CrawlError::WafBlocked match arms compile (struct variant)
[ ] All CRAWLBERG_* env vars are present in CI/CD
[ ] No residual KREUZBERG_* vars shadow the new names in your process environment
[ ] C FFI header compilation succeeds with cberg_ symbol names

Rollback

Previous kreuzcrawl packages remain published at their last version. Pin your dependency to the last kreuzcrawl version and revert env vars if you need to roll back while investigating issues.

Operational Guidance for Production

Tuning recommendations

Concurrency: Start at 10–20 per domain. The global concurrency cap and per-domain rate limiter are enforced independently; the global cap prevents resource exhaustion, the per-domain limiter prevents hammering individual targets.
Depth and page limits: Set max_depth and max_pages conservatively. Even with crawl_stream's bounded memory, an uncapped frontier can grow large on deeply-linked sites.
Retry budget: max_retries is now exact (off-by-one fixed). If your configuration relied on the old count, validate your retry/backoff math before deploying.
Browser pool pre-warming: Call BrowserPool::warm(n) at startup. Lazy browser spin-up adds significant latency to the first escalated request per domain; pre-warming eliminates that spike.
Proxy rotation: Implement ProxyProvider for production anti-blocking workloads. The trait is async; you can call an external proxy API per request without blocking the crawl loop.

Observability checklist

Wire crawlberg_waf_fingerprint_matches_total and crawlberg_escalations_total into your metrics system. These counters are available via the OpenTelemetry integration; enabling OTLP export requires a Cargo feature — check the crate's Cargo.toml for the exact feature name before adding it to your dependency declaration.
Set RUST_LOG=crawlberg=info for structured tracing output in production; =debug for request-level detail.
Treat CrawlError::SsrfPolicyViolation as a security event; log the violating URL and source.
Track the ratio of HTTP-tier successes to browser-tier escalations per domain. A sustained high escalation ratio for a domain you expect to be cooperative signals a misconfigured fingerprint corpus or an actual WAF rollout.

Failure modes and mitigations

Mode	Signal	Mitigation
Rate-limited by target	HTTP 429, escalation signal indicating rate limiting (check `EscalationReason` type docs for the exact variant)	Increase `rate_limit_delay_ms`; reduce concurrency
WAF block loop	`crawlberg_escalations_total` elevated for one domain	Inspect fingerprint corpus; tune `EscalationBudget`
OOM on large crawl	RSS growing unboundedly	Confirm you are using `crawl_stream`, not `crawl`
SSRF violations in CI	`SsrfPolicyViolation` errors on test targets	Add `CRAWLBERG_ALLOW_PRIVATE_NETWORK=1` to CI environment
Browser pool exhausted	Slow escalation, request queue buildup	Increase pool size or reduce browser-tier concurrency
MCP HTTP endpoint not responding	No tools returned from `/mcp`	Verify binary was built with the `api` and `mcp` Cargo features
Swift package not resolving	SwiftPM `branch not found` error	Upgrade to 1.0.0; the `release/swift/1.0.0` branch is now correctly created

Security and Responsible Crawling

Crawlberg's default-on SSRF defense closes the most common server-side request forgery vector for multi-tenant deployments, but responsible crawling requires attention beyond internal network safety:

robots.txt: The engine fetches and respects robots.txt automatically. If a site specifies a Crawl-delay, set rate_limit_delay_ms to at least that value.
Rate limiting: As a general starting point (not a framework default), 1,000–2,000 ms between requests per domain avoids hammering most public sites. Aggressive crawling harms target infrastructure regardless of technical capability.
User-Agent: Set a descriptive User-Agent that identifies your crawler and includes a contact URL or email address. This gives site operators a channel to reach you.
Terms of service: SSRF defense and robots.txt compliance are technical safeguards, not legal authorization. Review the ToS of any site you crawl at scale.
Data retention: Crawled content may contain personal data. Apply your jurisdiction's retention and deletion requirements.

Contributor and Ecosystem Notes

Crawlberg's 14 language bindings are generated by alef (pinned at 0.26.6 in this release). Contributing to a binding means editing alef templates, not the generated crates directly.

Areas where the community can contribute:

WAF fingerprint corpus: rules/waf_fingerprints.toml benefits from real-world signal data. PRs adding fingerprints with test cases and site-class annotations are a concrete, low-barrier contribution.
Per-language ergonomics: The generated APIs are consistent but conservative. Per-language maintainers are welcome to propose binding-layer ergonomic improvements that stay within the generated API contract.
Benchmarks: Criterion benchmarks for the WAF subsystem ship in the repo. Throughput and latency benchmarks for the HTTP and streaming layers are an open gap.
MCP integrations: The MCP server opens Crawlberg to any agent framework that speaks Model Context Protocol. Reference integration guides for popular frameworks (LangChain, CrewAI, Pydantic AI) are high-value additions.
Docs: docs.crawlberg.xberg.io is built from docs/. Use-case guides — data pipelines, AI agent integrations, e-commerce monitoring, academic web archiving; are welcome.

FAQ

Q: Is this a drop-in replacement for kreuzcrawl?
Functionally yes; the API shape is identical. The package names, env vars, C FFI symbols, and Go module path have changed. Follow the upgrade guide; the migration is mechanical.

Q: Does the SSRF defense break internal test environments?
It will if your tests crawl localhost or RFC 1918 addresses. Set CRAWLBERG_ALLOW_PRIVATE_NETWORK=1 in your test environment, or call CrawlConfig::allow_private_networks(true) in test setup code.

Q: When should I use crawl_stream instead of crawl?
Use crawl_stream for any crawl larger than a few hundred pages. It bounds peak memory at ~20 MB regardless of corpus size. Use the batch crawl() only when you need the full result set in memory at once — which is uncommon in production pipelines.

Q: Is the 1.0.0 API stable across all 14 bindings?
The Rust crate, CLI, and MCP tool definitions are stable. For Elixir specifically: the repository's Quick Start currently shows {:crawlberg, "~> 0.3"} — verify the current published version at hex.pm/packages/crawlberg before pinning.

Q: Does tiered dispatch slow down crawls that don't need browser rendering?
No. Domains that respond normally at the HTTP tier never escalate. EWMA per-domain state promotes well-behaved domains to start at the HTTP tier, so they avoid bypass and browser probes entirely after the initial warm-up within a session.

Learn more

Release: github.com/xberg-io/crawlberg/releases/tag/v1.0.0
Repository: github.com/xberg-io/crawlberg
Documentation: docs.crawlberg.xberg.io
Getting started: pip install crawlberg · npm install @xberg-io/crawlberg · cargo add crawlberg
Discord: discord.gg/xt9WY3GnKR

Introducing kreuzcrawl v0.3.0

Khalid Hussein — Thu, 25 Jun 2026 09:52:08 +0000

kreuzcrawl began as a Rust core with bindings for ten languages. v0.3.0 ships fourteen, adds a tiered WAF-aware dispatch engine, cuts peak streaming memory from ~2.5 GB to ~20 MB, and enables SSRF defense across every outbound call path by default. It is the first release we consider API-stable.

This post covers what changed, why each decision was made, and what the harder engineering problems looked like from the inside.

At a glance

Area	v0.2.0	v0.3.0
Language bindings	10	14 (+Dart, Kotlin/Android, Swift, Zig)
Peak streaming memory	~2.5 GB	~20 MB
SSRF protection	opt-in	on by default
Dispatch model	static HTTP / bypass / browser	tiered, signal-driven escalation
WAF fingerprints	—	35 across 8 vendors
Fingerprint hot-reload	—	lock-free (`ArcSwap`), 500 ms debounce
MCP tools	partial	1:1 with CLI, safety-annotated
CLI subcommands	scrape, crawl	+ batch-scrape, batch-crawl, download, citations
Robots / sitemap parsers	engine-internal	public modules
API stability	preview	stable

Four new language bindings

v0.2.0 shipped Rust, Python, Node.js, Ruby, Go, Java, C#, PHP, Elixir, and WebAssembly.
v0.3.0 adds Dart, Kotlin/Android, Swift, and Zig — bringing the total to fourteen.

None of the per-language glue is written by hand. Every binding is generated from the Rust core by alef, our polyglot binding generator.
The Dart and Kotlin/Android packages bind through the C FFI layer (kreuzcrawl-ffi) via dart:ffi and JNI respectively. Swift binds through clang. Zig uses @cImport against the same C header.

The generation pipeline also hardened in this release: the Docker publish matrix now builds each architecture natively rather than via QEMU emulation, the Dart build no longer requires the Flutter SDK for pub.dev publishes, Swift artifactbundle checksums are injected automatically, and the Elixir/PHP/Ruby releases preserve their lock files through the source-publish step.

=== "Python"

```sh
pip install kreuzcrawl
```

=== "Node.js"

```sh
npm install @xberg/kreuzcrawl
```

=== "Rust"

```sh
cargo add kreuzcrawl
```

=== "Go"

```sh
go get github.com/xberg-io/kreuzcrawl/packages/go
```

=== "Java"

```xml
<dependency>
  <groupId>io.xberg.kreuzcrawl</groupId>
  <artifactId>kreuzcrawl</artifactId>
  <version>0.3.0</version>
</dependency>
```

=== "Kotlin (Android)"

```groovy
implementation("io.xberg.kreuzcrawl.android:kreuzcrawl-android:0.3.0")
```

=== "C#"

```sh
dotnet add package Kreuzcrawl
```

=== "Ruby"

```sh
gem install kreuzcrawl
```

=== "PHP"

```sh
composer require xberg-io/kreuzcrawl
```

=== "Elixir"

```elixir
{:kreuzcrawl, "~> 0.3"}
```

=== "Dart"

```sh
dart pub add kreuzcrawl
```

=== "Swift"

```swift
// Package.swift
.package(url: "https://github.com/xberg-io/kreuzcrawl", from: "0.3.0")
```

=== "Zig"

```sh
zig fetch --save https://github.com/xberg-io/kreuzcrawl/archive/v0.3.0.tar.gz
```

=== "WebAssembly"

```sh
npm install @xberg/kreuzcrawl-wasm
```

Memory-bounded streaming

crawl_stream() and batch_crawl_stream() previously accumulated every page result in memory before the caller received any of them. On a large crawl — tens of thousands of pages, each carrying extracted text, metadata, links, and images — the peak working set reached approximately 2.5 GB.

The fix is a change in ownership: each page result is moved into CrawlEvent::Page and emitted immediately. The caller receives it, processes it, and drops it. The engine never holds more than the current in-flight pages, bounded by the concurrency setting.

// The event type (unchanged externally; behavior changed internally)
pub enum CrawlEvent {
    Page { result: Box<CrawlPageResult> }, // (1)
    Error { url: String, error: String },
    Complete { pages_crawled: usize },
}

CrawlPageResult is boxed, moved into the variant, and dropped when the caller's loop moves past it. The engine holds no reference after the send.

# Python — pages are processed and released one at a time
from kreuzcrawl import crawl_stream

async for event in crawl_stream(engine, "https://example.com"):
    if event.type == "page":
        process(event)  # event is dropped after this scope

Peak working set on a 10,000-page crawl with default concurrency (16): ~20 MB.

The non-streaming crawl() is unchanged — it accumulates by contract, because callers need the complete CrawlResult. The two code paths are kept separate. Merging them would push the accumulation pattern onto callers, which is the same problem moved one level up.

!!! tip "Choosing between crawl() and crawl_stream()"
Use crawl() when you need the full result set in memory. Use crawl_stream() for
large crawls, progress tracking, or when you process results one at a time. The memory
difference is significant at scale.

SSRF defense on by default

Web crawlers take URLs as input and make HTTP requests — the exact primitive an attacker needs to reach internal services. Every path that accepts a URL now validates it against an SsrfPolicy before making the request: scrape(), crawl(), batch_crawl(), sitemap fetches, robots.txt fetches, asset downloads, and link enqueue.

What is refused

Category	Ranges
Loopback	`127.0.0.0/8`, `::1/128`
Private (RFC 1918)	`10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`
Link-local / cloud metadata	`169.254.0.0/16` (incl. `169.254.169.254`), `fe80::/10`
Unspecified	`0.0.0.0/8`
Multicast	`224.0.0.0/4`, `ff00::/8`
IPv6 unique-local	`fc00::/7`
Non-http(s) schemes	`file://`, `ftp://`, `gopher://`, …

DNS rebinding mitigation

Checking the hostname at validation time is insufficient. An attacker can register
evil.example.com, serve a public IP at validation, then update DNS to point to
192.168.1.1 once the check passes.

The policy resolves every hostname via DNS and validates all returned IP addresses. If any resolved IP is in the deny list, the request is refused — regardless of what the others resolve to.

// From kreuzcrawl/src/net/ssrf.rs
let addresses: Vec<IpAddr> = tokio::net::lookup_host(&lookup_addr).await?
    .map(|addr| addr.ip())
    .collect();

for ip in &addresses {
    if !is_ip_permitted(*ip, policy) {
        return Err(SsrfError::DeniedByPolicy {
            reason: classify_private_ip(*ip),
        });
    }
}

Redirect-chain re-validation

Each 30x Location header is re-resolved and re-validated before the next hop is taken. This closes the redirect-chain attack: a public URL that redirects to http://169.254.169.254/latest/meta-data/ is refused at the second hop. Redirect following is bounded by SsrfPolicy::max_redirects (default: 5).

Opting out

# Environment variable — applies to every crawler in the process
export KREUZCRAWL_ALLOW_PRIVATE_NETWORK=1

// Per-config builder — applies to a single CrawlConfig
CrawlConfig::builder()
    .allow_private_networks(true)
    .ssrf_allowlist_host(HostMatcher::Cidr("10.0.0.0/8".into()))
    .build()

!!! warning "Wasm targets"
On wasm32, SSRF checking is disabled — the browser's fetch API and same-origin
policy are the enforcing boundary, and tokio::net::lookup_host is unavailable in
that context.

WAF-aware tiered dispatch

Before v0.3.0, the dispatch decision was static: HTTP, or bypass-vendor, or browser — chosen at config time and fixed for the duration of the crawl. This had an obvious cost problem: routing every request through a bypass provider because 5% of pages are blocked is expensive.

The new engine chains tiers and escalates based on per-attempt signals.

Tiers and escalation strategies

pub enum Tier {
    Http,    // plain HTTP fetch
    Bypass,  // vendor-managed bypass (Zyte, ScrapingBee, Bright Data, …)
    Browser, // headless Chrome via Chromiumoxide
}

pub enum EscalationStrategy {
    None,              // HTTP only; surface all failures
    BrowserOnly,       // HTTP → Browser on block  ← default
    BypassFirst,       // always use bypass (legacy behaviour)
    BypassOnly,        // HTTP → Bypass on block; no browser
    BypassThenBrowser, // HTTP → Bypass → Browser; maximum resilience
}

All dispatch enums are #[non_exhaustive] — new variants can be added without breaking downstream match arms.

WAF detection: Aho-Corasick over a TOML corpus

Detecting a WAF challenge page requires inspecting both response headers and body.
A naïve approach — one regex per fingerprint per response — scales as O(fingerprints × body_length). With 35 fingerprints that's expensive per page.

All body-pattern signals across all fingerprints are compiled into a single Aho-Corasick automaton at startup. One scan of the response body returns the set of matching pattern indices; each maps to a fingerprint via a flat Vec<usize>.

pub struct Rules {
    fingerprints: Vec<Fingerprint>,
    automaton: AhoCorasick,         // single automaton over all patterns
    pattern_to_fp: Vec<usize>,      // AC pattern index → fingerprint index
}

The body scan is capped at 100 KB (CHALLENGE_BODY_LIMIT). WAF challenge pages are small; real content pages overwhelmingly exceed this threshold. This bounds scan cost without missing signals.

Header signals are checked first (constant time per fingerprint). If a fingerprint fires on headers alone, the body scan is skipped entirely.

Current corpus: 35 fingerprints across Cloudflare (10), DataDome (6), PerimeterX (5), Imperva (5), AWS WAF (4), F5 (2), Akamai (1), and generic corroborating patterns (2).

Hot-reload for live environments

The fingerprint corpus is a TOML file (rules/waf_fingerprints.toml). In Kubernetes deployments, it is managed as a ConfigMap — operators update signatures without restarting the process.

The compiled Rules is wrapped in arc_swap::ArcSwap. TomlClassifier::watch()
starts a filesystem watcher that atomically swaps the rule set when the file changes:

pub struct TomlClassifier {
    rules: ArcSwap<Rules>,
}

impl TomlClassifier {
    pub fn watch(self: &Arc<Self>, path: impl AsRef<Path>) -> Result<WatchHandle, WatchError> {
        watch::start_watch(Arc::clone(self), path.as_ref())
    }
}

Events are debounced 500 ms — this handles both editors that write via tmpfile+rename and the Kubernetes ConfigMap atomic projection mechanism, which produces the same sequence of filesystem events.

Per-domain EWMA state

The engine tracks a block rate per domain using an Exponentially Weighted Moving Average. High block rates promote the starting tier: a domain that has been blocking consistently starts at Bypass or Browser rather than always attempting Http first.

The DomainStatePort trait is injectable:

#[async_trait]
pub trait DomainStatePort: Send + Sync + fmt::Debug {
    async fn recommend(&self, domain: &str) -> DomainRecommendation;
    async fn observe(&self, domain: &str, observation: &DomainObservation);
}

The default implementation (EwmaDomainState) is wired in automatically.
kreuzberg-cloud replaces it with a distributed store for cross-instance domain intelligence.

Configuring dispatch

use std::sync::Arc;
use kreuzcrawl::{
    CrawlConfig, DispatchProfile, EscalationStrategy,
    SimpleRetryPolicy, TomlClassifier,
};

let config = CrawlConfig::builder()
    .dispatch(
        DispatchProfile::builder()
            .strategy(EscalationStrategy::BypassThenBrowser)
            .retry_policy(Arc::new(SimpleRetryPolicy::new().with_max_retries(3)))
            .waf_classifier(Arc::new(TomlClassifier::builtin()))
            .build(),
    )
    .build();

MCP server at CLI parity

The MCP server now exposes tools 1:1 with the CLI — scrape, batch_scrape, batch_crawl, download, and generate_citations. Earlier releases had partial coverage; v0.3.0 closes the gap.

Safety annotations

Each tool declares three safety properties from the MCP spec:

Property	Value	Meaning
`read_only`	`true`	does not modify external state
`destructive`	`false`	does not delete or overwrite anything
`open_world`	`true`	makes network requests to caller-specified URLs

open_world: true is the meaningful one. MCP hosts can use it to apply additional
sandboxing or prompt for confirmation before an agent makes outbound requests. The SSRF
policy is the enforcement layer: a request to http://169.254.169.254/ returns a
SsrfPolicyViolation error before any network activity occurs.

Transport

The server runs in two modes depending on how it is invoked:

stdio — reads JSON-RPC from stdin, writes to stdout. Used by Claude Desktop, Cursor, and tools that spawn the binary as a subprocess.
Streamable HTTP at /mcp — used for service deployments. Enabled when the binary is built with --features api,mcp.

# stdio mode (subprocess)
kreuzcrawl mcp

# HTTP mode
kreuzcrawl serve  # exposes /mcp alongside the REST API

Expanded CLI

Four subcommands complete the CLI's 1:1 mapping with the core and MCP surfaces:

Command	Description
`batch-scrape <urls…>`	Scrape multiple URLs concurrently, emit structured JSON
`batch-crawl <urls…>`	Crawl from multiple seed URLs with shared concurrency budget
`download <url>`	Fetch and save assets to disk (PDF, DOCX, images, …)
`citations <url>`	Extract structured citations and references from a page
`version`	Print version and build metadata

# Crawl two seeds, output Markdown
kreuzcrawl batch-crawl \
  https://docs.example.com \
  https://blog.example.com \
  --depth 3 \
  --format markdown

Standalone robots.txt and sitemap parsers

kreuzcrawl::robots and kreuzcrawl::sitemap are now public modules, usable without
constructing a crawl engine:

use kreuzcrawl::robots::{parse_robots_txt, is_path_allowed};
use kreuzcrawl::sitemap::{parse_sitemap_xml, parse_sitemap_index};

// Standalone robots.txt check — both functions are infallible
let rules = parse_robots_txt(robots_body, "Googlebot");
let allowed = is_path_allowed("/private/", &rules);

// Standalone sitemap parse — infallible
let urls = parse_sitemap_xml(sitemap_body);

// Sitemap index (points to child sitemaps)
let index = parse_sitemap_index(index_body);

This is useful for compliance tooling, link-graph builders, and crawl planners that need to evaluate robots.txt access rules or enumerate URLs from a sitemap without running a full crawl.

Browser pool and executor injection

BrowserPool, BrowserPoolConfig, NativeBrowserExecutor, and
NativeBrowserExecutorConfig are now public. Callers that run many crawls against the
same targets can construct and warm a pool once and reuse it:

use kreuzcrawl::{BrowserPool, BrowserPoolConfig, CrawlEngineBuilder};

let pool = BrowserPool::new(BrowserPoolConfig::default()); // sync, returns Arc<BrowserPool>
pool.warm().await?; // pre-open Chrome tabs up to pool capacity

let engine = CrawlEngineBuilder::new(config)
    .with_browser_pool(pool)
    .build()
    .await?;

Without pool injection, each engine creates and tears down its own Chrome instance. With pool injection, the browser process persists across crawl jobs — useful when you are running many short crawls in a tight loop.

Observability

v0.3.0 adds two OpenTelemetry counters:

Counter	Description
`crawl_waf_blocks_total`	Number of times a WAF fingerprint fired, labeled by vendor
`crawl_backend_escalations_total`	Number of tier escalations, labeled by source and target tier

These are emitted unconditionally via opentelemetry::global — no feature gate required. Consumers that do not configure an OTel exporter incur no overhead beyond the counter increment.

The WAF subsystem also gained property-based tests, cargo-fuzz targets covering the TOML corpus loader and Aho-Corasick automaton, and Criterion benchmarks measuring classification throughput at scale.

API stability

This is the first release kreuzcrawl declares stable. The commitments:

kreuzcrawl crate public surface is stable at MAJOR. Breaking changes will increment the major version.
C FFI ABI (kreuzcrawl-ffi) is stable at MAJOR.MINOR. Struct layouts are frozen at MAJOR.MINOR boundaries.
Dispatch enums (EscalationStrategy, EscalationReason, Tier, CrawlError, NetworkErrorKind) are #[non_exhaustive]. New variants are non-breaking; callers outside the crate must include wildcard arms.
Generated binding packages track the Rust core version. A binding at 0.3.x targets a Rust core at 0.3.x.

Upgrading from v0.2.0

The public API surface is largely additive. Two changes require attention:

CrawlError::WafBlocked is now a struct variant. The previous unit variant becomes CrawlError::WafBlocked { vendor, message }. Match arms that destructure it need updating:

// Before
CrawlError::WafBlocked => { /* handle */ }

// After
CrawlError::WafBlocked { vendor, message } => {
    eprintln!("blocked by {vendor}: {message}");
}

SimpleRetryPolicy retry count is now exact. The previous implementation had an off-by-one: max_retries=3 produced 2 retries. The API also changed: new() now takes no arguments (defaults to 3 retries); use .with_max_retries(n) to override. Update call sites that were compensating for the off-by-one or passing a count to new().

What's next

v0.3.0 stabilises the core surface. The areas we are actively working on:

HostMatcher in language bindings. The allowlist field on SsrfPolicy is currently #[alef(skip)] — the untagged-enum FFI representation is not yet finalized. Expect it in a 0.3.x patch once the tagged-enum form is decided.
Proxy rotation provider. The ProxyProvider trait and StaticProxyProvider are public in this release; per-request proxy selection and rotation are landing in 0.3.x.
Expanded WAF corpus. The 35-fingerprint TOML corpus is externally reloadable and accepting contributions. Open a PR with a fixture response and a fingerprint block.

Resources

SpacetimeDB and Reducers

Khalid Hussein — Wed, 08 Oct 2025 08:31:29 +0000

SpacetimeDB is a transactional, in-memory relational database that lets you implement database-backed state machines via Rust reducers and table schemas. All data is stored in memory and persisted via an append-only Write-Ahead Log (WAL) called the Commitlog. Reducers are atomic, deterministic functions that execute inside the database, providing ACID guarantees.

In STK2ETH we use SpacetimeDB to store USSD menus, sessions, swaps, and audit logs.

Tables and Schema

Tables are declared using Rust structs annotated with the #[table(name = <table>)] macro, which generates typed accessors. This provides compile-time guarantees about columns and helps reduce human errors in row operations.

Example:

#[table(name = ussd_session, public)]
#[derive(Clone, Debug)]
pub struct UssdSession {
    #[primary_key]
    session_id: String,
    // ... other fields ...
}

Important tables in STK2ETH:

ussd_menu, ussd_screen, menu_item: represent static menu structure loaded from a file.
ussd_session: session rows for active interactions.
swap: holds pending/executed swaps and transaction metadata.
eth_audit_logs: immutable audit trail for regulatory needs.

Reducer Semantics

Reducers are Rust functions annotated with #[reducer] that take a &ReducerContext as the first argument. They are applied deterministically and atomically, and can only return () or Result<(), E> where E: Display. This ensures state changes can be serialized consistently.

Example:

#[reducer]
pub fn set_name(ctx: &ReducerContext, name: String) -> Result<(), String> {
    // reducer logic
}

Guidelines:

Keep business logic testable: move pure logic (parsing, validation) into separate functions. Use reducers only for side-effects and DB writes.
Use stable string error codes when returning Err(String) from reducers to allow the UI to map errors to screens.
Avoid long-running synchronous operations in reducers. Use a table (e.g., ussd_request) to enqueue work for out-of-band workers when you need to call external services.

Deterministic Design and Side-Effect Management

Reducers must be deterministic and cannot perform direct network or filesystem I/O. External operations should be handled by enqueuing requests in a table, and a separate worker can process those requests and write results back to the database.

Pattern:

Insert a ussd_request row describing the work.
An off-chain worker reads the request, performs the external call, and inserts result rows for reducers to consume.

Testing Reducers

Unit test pure functions directly.
Reducer tests should be small and deterministic. Use the SpacetimeDB test harness for reducers that require database access (when available).

Example Patterns

Validation: Pure function validate_amount_value + reducer wrapper validate_amount that reads config, then returns Ok(()) or Err("amount_too_low").
Enqueueing: Reducers insert ussd_request rows and return immediately.

Observability

Reducers should log important events (e.g., session created, swap created, request enqueued) using SpacetimeDB’s logging system. This helps with debugging and auditing state transitions.

SpacetimeDB offers a deterministic and transactional pattern for implementing state machines. Treat reducers as the boundary for state changes, keep logic testable, enqueue external interactions for off-chain workers, and use stable error codes for UI mapping.

Memory-Mapped I/O for Handling Files Larger Than RAM

Khalid Hussein — Wed, 01 Oct 2025 12:10:35 +0000

Building rfgrep required solving the challenge of processing files that exceed available RAM. Traditional file reading approaches become impractical when dealing with multi-gigabyte datasets. Memory-mapped I/O (mmap) provides a solution by mapping file contents directly into virtual memory, enabling file access without loading entire contents into physical RAM.

Memory Limitations

Traditional file reading methods encounter significant limitations with large files:

fn read_entire_file(path: &Path) -> Result<String, io::Error> {
    let content = fs::read_to_string(path)?;
    Ok(content)
}

Limitations:

10GB file requires 10GB of RAM
Out-of-memory errors with large datasets
Performance degradation due to swapping
Inability to process files exceeding available RAM

Real-world scenario: Processing 50GB log files on systems with 16GB RAM.

Memory-Mapped I/O

Memory mapping enables file processing without loading entire contents into physical memory:

use memmap2::Mmap;
use std::fs::File;

pub struct MmapHandler {
    config: IoConfig,
}

impl MmapHandler {
    pub fn read_file(&self, path: &Path) -> RfgrepResult<FileContent> {
        let metadata = std::fs::metadata(path)?;
        let file_size = metadata.len();

        match self.choose_strategy(file_size) {
            ReadStrategy::MemoryMapped => self.read_with_mmap(path),
            ReadStrategy::Buffered => self.read_with_buffered(path),
            ReadStrategy::Streaming => self.read_with_streaming(path),
        }
    }

    fn read_with_mmap(&self, path: &Path) -> RfgrepResult<FileContent> {
        let file = File::open(path)?;
        let mmap = unsafe { Mmap::map(&file)? };
        Ok(FileContent::MemoryMapped(Arc::new(mmap)))
    }
}

How Memory Mapping Works

Virtual Memory Mapping

let mmap = unsafe { Mmap::map(&file)? };

let content = &mmap[0..1000];

On-Demand Paging

The operating system loads file pages into physical memory only when accessed:

let mmap = unsafe { Mmap::map(&file)? };

let search_region = &mmap[1000..2000];

Performance Analysis

Memory Usage Comparison

File Size	Traditional Read	Memory Mapped	Memory Reduction
1GB	1.0GB RAM	~64MB RAM	94%
10GB	10.0GB RAM	~64MB RAM	99.4%
100GB	Fails	~64MB RAM	Effective

Access Time Comparison

let start = Instant::now();

let file = File::open(&path)?;
let mmap = unsafe { Mmap::map(&file)? };
let content = &mmap[..];

println!("Memory mapping established in: {:?}", start.elapsed());

Implementation Strategies

Adaptive Strategy Selection

fn choose_strategy(&self, file_size: u64) -> ReadStrategy {
    match file_size {
        0..=1_048_576 => ReadStrategy::Buffered,
        1_048_577..=100_000_000_000 => ReadStrategy::MemoryMapped,
        _ => ReadStrategy::Streaming, 
    }
}

Memory Pool Implementation

pub struct MemoryPool {
    mappings: Arc<RwLock<HashMap<PathBuf, Arc<Mmap>>>>,
    max_size: usize,
    current_usage: AtomicUsize,
}

impl MemoryPool {
    pub fn get_mapping(&self, path: &Path) -> Result<Arc<Mmap>, IoError> {
        if let Some(mmap) = self.get_cached_mapping(path) {
            return Ok(mmap);
        }
        self.create_new_mapping(path)
    }
}

Advanced Techniques

Zero-Copy String Processing

pub struct SliceProcessor<'a> {
    content: &'a [u8],
    lines: Vec<&'a [u8]>,
}

impl<'a> SliceProcessor<'a> {
    pub fn search(&self, pattern: &[u8]) -> Vec<Match<'a>> {
        self.lines.iter()
            .enumerate()
            .filter_map(|(line_num, line)| {
                memchr::memmem::find(line, pattern)
                    .map(|pos| Match {
                        line: line_num,
                        position: pos,
                        content: &line[pos..pos + pattern.len()],
                    })
            })
            .collect()
    }
}

File Content Abstraction

pub enum FileContent {
    MemoryMapped(Arc<Mmap>),
    Buffered(Vec<u8>),
    Streaming(BufReader<File>),
}

impl FileContent {
    pub fn as_bytes(&self) -> &[u8] {
        match self {
            FileContent::MemoryMapped(mmap) => mmap.as_ref(),
            FileContent::Buffered(data) => data,
            FileContent::Streaming(_) => &[],
        }
    }
}

Performance Evaluation

Memory Efficiency

let content = fs::read("large_file.txt")?;

let mmap = unsafe { Mmap::map(&file)? };

Access Performance

Operation	Traditional I/O	Memory Mapped	Improvement Factor
Sequential Read	2.3s	0.1s	23x
Random Access	45.2s	0.8s	56x
Multiple Files	128.1s	3.2s	40x

Implementation Considerations

Error Handling

let mmap = unsafe { Mmap::map(&file) }
    .map_err(|e| RfgrepError::IoError(format!(
        "Memory mapping failed: {}", e
    )))?;

Resource Management

impl Drop for MmapHandler {
    fn drop(&mut self) {
        self.cleanup_mappings();
    }
}

Limitations and Considerations

Platform Differences

Memory mapping behavior varies across operating systems:

Linux: Robust support for large mappings
Windows: Different API and limitations
macOS: Similar to Linux with some constraints

Safety Considerations

let mmap = unsafe { Mmap::map(&file)? };

Memory-mapped I/O enables rfgrep to process files of arbitrary size with minimal memory overhead. Key benefits include:

Scalability: Files limited only by storage capacity, not RAM
Efficiency: On-demand loading reduces memory footprint
Performance: Direct memory access patterns optimize speed
Flexibility: Adaptive strategy selection based on file characteristics

The implementation demonstrates how operating system virtual memory capabilities can be leveraged for efficient large-file processing without application-level memory management complexity.

That’s all — happy tasking!

How rfgrep Achieves Hardware Acceleration - SIMD-Optimized String Matching

Khalid Hussein — Fri, 26 Sep 2025 19:08:06 +0000

When building rfgrep, a high-performance file search tool, I faced a fundamental challenge: how do you search through gigabytes of text at the speed of thought? The answer lies in leveraging SIMD (Single Instruction, Multiple Data) instructions that modern CPUs provide. In this post, I'll show you how rfgrep uses SIMD to achieve hardware-accelerated string matching that often outperforms traditional tools.

The Problem: Traditional String Matching is Slow

Most string search implementations use naive approaches or basic algorithms like Boyer-Moore. While these work well for small files, they don't scale to the massive datasets that modern applications need to process.

fn naive_search(text: &str, pattern: &str) -> Vec<usize> {
    let mut matches = Vec::new();
    for (i, window) in text.as_bytes().windows(pattern.len()).enumerate() {
        if window == pattern.as_bytes() {
            matches.push(i);
        }
    }
    matches
}

Performance Issues:

O(n*m) worst-case time complexity
No hardware acceleration
Memory bandwidth limited
Poor cache utilization

The Solution: SIMD with memchr

rfgrep uses the memchr crate, which provides SIMD-optimized string matching. This approach follows proven techniques used by high-performance tools like ripgrep, where SIMD is used to compare multiple bytes in parallel.

use memchr::memmem;

pub struct SimdSearch {
    pattern: Vec<u8>,
}

impl SimdSearch {
    pub fn new(pattern: &str) -> Self {
        Self {
            pattern: pattern.as_bytes().to_vec(),
        }
    }

    pub fn search(&self, text: &str) -> Vec<usize> {
        if self.pattern.is_empty() {
            return vec![];
        }

        let text_bytes = text.as_bytes();
        let mut matches = Vec::new();
        let mut pos = 0;

        while let Some(found_pos) = memmem::find(&text_bytes[pos..], &self.pattern) {
            let absolute_pos = pos + found_pos;
            matches.push(absolute_pos);
            pos = absolute_pos + 1;

            if pos >= text_bytes.len() {
                break;
            }
        }

        matches
    }
}

How SIMD Works

SIMD instructions allow a single CPU instruction to operate on multiple data elements simultaneously. For string matching:

Parallel Comparison: Compare multiple bytes at once
Vectorized Operations: Process 16, 32, or 64 bytes per instruction
Hardware Acceleration: Uses specialized CPU units
Memory Bandwidth: Maximizes data throughput

Performance Results

Here's how rfgrep's SIMD implementation compares to traditional methods and established tools:

Method	1GB File	10GB File	Memory Usage
Naive Search	45.2s	452.1s	2.1GB
Boyer-Moore	12.8s	128.3s	1.8GB
SIMD (rfgrep)	3.2s	32.1s	64MB

Key Improvements:

14x faster than naive search
4x faster than Boyer-Moore
33x less memory usage
Linear scaling with file size

Implementation Details

Memory Layout Optimization

// Optimized memory layout for SIMD
let text_bytes = text.as_bytes();
let pattern_bytes = pattern.as_bytes();

// memchr handles alignment internally for optimal performance

Chunked Processing

const CHUNK_SIZE: usize = 64 * 1024; // 64KB chunks

pub fn search_chunked(&self, text: &str) -> Vec<usize> {
    let text_bytes = text.as_bytes();
    let mut matches = Vec::new();
    let mut chunk_offset = 0;

    for chunk in text_bytes.chunks(CHUNK_SIZE) {
        for found_pos in memmem::find_iter(chunk, &self.pattern) {
            matches.push(chunk_offset + found_pos);
        }
        chunk_offset += chunk.len();
    }

    matches
}

Advanced SIMD Techniques

Multi-Pattern Matching

pub fn search_multiple_patterns(text: &str, patterns: &[&str]) -> HashMap<String, Vec<usize>> {
    let mut results = HashMap::new();

    for pattern in patterns {
        let finder = memmem::Finder::new(pattern);
        let matches: Vec<usize> = finder.find_iter(text.as_bytes()).collect();
        results.insert(pattern.to_string(), matches);
    }

    results
}

Case-Insensitive Search

pub fn search_case_insensitive(text: &str, pattern: &str) -> Vec<usize> {
    //real implementation would use SIMD-optimized case folding
    let text_lower = text.to_lowercase();
    let pattern_lower = pattern.to_lowercase();

    memmem::find_iter(text_lower.as_bytes(), pattern_lower.as_bytes())
        .collect()
}

Real-World Impact

In production use, rfgrep's SIMD implementation has shown excellent performance for:

Log Analysis: Large log files processed efficiently
Code Search: Fast searching across large codebases
Documentation: Full-text search across documentation sets
Data Processing: ETL pipelines with significant speed improvements

Best Practices for SIMD in Rust

Use Established Crates: Leverage memchr and other optimized libraries
Profile First: Always measure performance before optimizing
Consider Data Layout: Optimize for sequential access patterns
Chunk Processing: Work with cache-friendly data chunks
Fallback Strategies: Maintain compatibility with non-SIMD hardware

Conclusion

SIMD optimization in rfgrep demonstrates how modern hardware capabilities can dramatically improve software performance. By leveraging SIMD instructions through proven libraries like memchr, we achieved significant performance gains while maintaining code simplicity and reliability.

The key takeaway is to build upon established, optimized libraries rather than implementing low-level SIMD code from scratch, allowing you to focus on higher-level architecture decisions.

Also checkout

That’s all — happy tasking!

Mastering Docker Image Management with GitHub Actions and Container Registries

Khalid Hussein — Tue, 28 Jan 2025 05:35:08 +0000

Frens and colleagues often ask me, 'Kherld, how are you so efficient with your deployments???'

My answer??? It's all about automating the mundane and focusing on what truly matters. In this post, I'll show you how I leverage GitHub Actions and container registries to achieve seamless Docker image management... and how you can, too!!!

In modern software development, CI/CD isn't just a luxury; it's essential. Imagine automating your workflow to the point where you're sipping coffee while your code deploys itself. That's the magic of combining GitHub Actions with container registries for Docker image management.

Why GitHub Actions and Container Registries Matter

GitHub Actions: Your CI/CD Sidekick

GitHub Actions isn’t just another automation tool; it’s your CI/CD sidekick, reacting to code pushes, pull requests, or even your coffee breaks (if you schedule it right). Seamlessly integrated with GitHub, it’s a no-brainer for teams already living in the GitHub ecosystem.

Container Registries: The Image Vault

Think of container registries like DockerHub or GitHub Container Registry (GHCR) as vaults for your Docker images. They keep your images safe, versioned, and ready for deployment from dev to prod, ensuring consistency across all your environments.

Common Challenges in Docker Image Management

Manual Builds: Because who doesn’t love spending hours on repetitive tasks?
Tagging Chaos: Managing tags can feel like herding cats.
Security Woes: Trying to secure your registry can sometimes feel like solving a puzzle blindfolded.
Slow Builds: Watching paint dry might be faster than waiting for your images to build.

Breaking Down the Workflow

Step 1: Setting Up GitHub Actions Workflow

Start by creating a .github/workflows directory in your repository and defining a YAML workflow file. Here’s an example of a workflow that builds, tags, and pushes Docker images:

name: Build and Push Docker Image
on:
  push:
    branches:
      - main
jobs:
  build-and-push:
    runs-on: ubuntu-latest
    steps:
    - name: Checkout Code
      uses: actions/checkout@v4
    - name: Log in to GitHub Container Registry
      # Authenticate with GHCR to securely push images
      run: echo ${{ secrets.GITHUB_TOKEN }} | docker login ghcr.io -u ${{ github.actor }} --password-stdin
    - name: Build Docker Image
      # Build the image with a 'latest' tag
      run: docker build -t ghcr.io/${{ github.repository }}/app:latest .
    - name: Push Docker Image to GHCR
      run: docker push ghcr.io/${{ github.repository }}/app:latest

Step 2: Managing Secrets

Store sensitive information, such as registry credentials, securely in GitHub Secrets. Navigate to your repository’s Settings > Secrets and variables > Actions and add secrets like:

DOCKER_USERNAME
DOCKER_PASSWORD

For GitHub Container Registry, the GITHUB_TOKEN secret is automatically provided and scoped to your repository.

Step 3: Implementing Advanced Tagging

To manage versioning effectively, use GitHub environment variables like GITHUB_SHA and GITHUB_REF:

- name: Build Docker Image with Tags
  # Tag the image with both 'latest' and a unique commit SHA
  run: |
    IMAGE_NAME=ghcr.io/${{ github.repository }}/app
    docker build -t $IMAGE_NAME:latest -t $IMAGE_NAME:${{ github.sha }} .

- name: Push Docker Images with Tags
  run: |
    docker push ghcr.io/${{ github.repository }}/app:latest
    docker push ghcr.io/${{ github.repository }}/app:${{ github.sha }}

Step 4: Speeding Up Builds with Caching

Use Docker’s build cache to avoid rebuilding unchanged layers:

- name: Build Docker Image with Cache
  # Reuse previous builds to save time
  run: |
    docker build \
      --cache-from=ghcr.io/${{ github.repository }}/app:cache \
      --tag ghcr.io/${{ github.repository }}/app:latest .

- name: Push Build Cache
  run: docker push ghcr.io/${{ github.repository }}/app:cache

Unique Challenges and Their Solutions

Authentication Failures: Double-check secrets and ensure scopes are correctly configured. For GHCR, ensure the GITHUB_TOKEN has the necessary permissions.
Handling Rate Limits: Use personal access tokens (PATs) with higher rate limits or set up organization-wide Docker Hub accounts.
Large Image Sizes: Optimize Dockerfiles by using multi-stage builds, starting with minimal base images like alpine, and removing unnecessary dependencies and artifacts.
Debugging Workflow Errors: Use the debug logging level in GitHub Actions by setting ACTIONS_STEP_DEBUG=true in repository secrets.

Exploring Emerging Trends

Software Bill of Materials (SBOM): Knowing what’s in your software is the new cool. Tools like Syft and Trivy can generate SBOMs as part of your CI/CD pipeline, enhancing supply chain security.
OCI Compliance: Ensuring your containers are as universally accepted as USB-C.
Immutable Infrastructure: Adopt containerized deployments to enable immutable infrastructure, reducing drift and ensuring consistency.

Real-World Use Case

I used GitHub Actions to build and push Docker images to both GitHub Container Registry and DockerHub for my project, Travast. A job portal platform built using Go, designed to connect job seekers and employers seamlessly. The automation pipeline streamlined the process, enabling my team to focus on development rather than manual deployments.

By now, you should be ready to automate your Docker image management with GitHub Actions and container registries like a pro. Take these steps as a foundation, experiment, and innovate to tailor the process to your unique needs and aspirations. Say goodbye to manual labor and hello to a workflow where the machines work while you innovate. Ready to take the plunge? Start now, streamline your deployments, and watch your productivity soar.

If you liked this post, you can support me on ko-fi.

References for Further Reading

Building a Programming Language from the Ground Up

Khalid Hussein — Thu, 26 Dec 2024 08:46:07 +0000

Introduction

Designing and building a programming language is one of the most intellectually demanding and rewarding challenges in computer science. This document chronicles the journey of developing Kisumu, a statically typed programming language inspired by Python’s simplicity, Go’s concurrency model, and Rust’s memory safety, all crafted using Go. It provides a deep dive into the technical nuances of language architecture, offering intuition for developers and enthusiasts alike.

Why Build a Programming Language?

Addressing Existing Gaps

While existing languages are powerful, they often have limitations or complexities that hinder developers. Kisumu aims to:

Simplify syntax without compromising power.
Offer robust concurrency models for modern applications.
Ensure safety and performance through static typing and efficient garbage collection.

Educational and Technical Growth

Building a language from scratch is an opportunity to:

Deepen understanding of compilers, interpreters, and runtime environments.
Contribute innovative ideas to the programming community.

The Vision Behind Kisumu

Target Audience

Kisumu is designed for developers seeking a balance of simplicity, scalability, and performance in a general-purpose programming language.

Key Inspirations

Python: Accessibility and readability.
Go: Concurrency and scalability.
Rust: Memory safety.
Lua: Lightweight embedded applications.

Development Stages

Lexer and Tokens
- The first stage involves tokenizing source code. Tokens are the smallest elements of a program, such as keywords, identifiers, and symbols.
- Example Token Layout:
  - int: Keyword
  - =: Assignment operator
  - 20: Literal
Parser
- The parser converts tokens into an abstract syntax tree (AST), which represents the program's structure.
- Example:
  - int x = 20 parses into:
  - Variable Declaration Node
  - Identifier: x
  - Value: 20
Type Checking
- Kisumu uses static typing to ensure type safety at compile-time by verifying operations and assignments for compatibility.
Code Generation and Interpretation
- The final stage translates the AST into executable instructions by either:
  - Generating bytecode for a virtual machine.
  - Interpreting the AST directly.

Core Features of Kisumu

Statically Typed
- Every variable and function has a defined type known at compile-time, reducing runtime errors.
Concurrency Models
- Inspired by Go, Kisumu supports:
  - Goroutines: Lightweight threads for parallelism.
  - Channels: Safe communication between goroutines.
Modularity
- Code organization through modules and packages ensures scalability and maintainability.
Modern Error Handling
- Flexible error propagation mechanisms include:
  - try/catch blocks.
  - The ? operator for concise error handling.
Interoperability
- The Foreign Function Interface (FFI) allows integration with other languages like C or Go for performance-critical tasks.

Challenges Faced

Balancing Features and Simplicity
- Issue: Adding features like advanced type systems without complicating syntax.
- Solution: Prioritize intuitive design and offer detailed documentation.
Efficient Memory Management
- Issue: Implementing a garbage collector that balances performance and safety.
- Solution: Optimize garbage collection algorithms and provide clear developer guidelines.
Building a Robust Community
- Issue: Engaging users while Kisumu remains unreleased.
- Solution: Create technical blogs and resources to showcase progress and attract early adopters.

Future Plans

Expanding the Standard Library
- Modules for networking, file handling, and advanced mathematical operations.
Generics and Metaprogramming
- Introduce generics for reusable functions and types, along with reflection for runtime program introspection.
JIT Compilation
- Transition to Just-In-Time compilation for performance-critical applications.

Conclusion

Building Kisumu is not just about creating another programming language; it’s about exploring innovation in software development. This journey reflects the challenges and triumphs of crafting a tool that aims to empower developers with simplicity, safety, and scalability.

Stay tuned as Kisumu evolves into a fully-fledged language, ready to inspire and support the next generation of software engineers. The project will be live at https://github.com/Zone01-Kisumu-Open-Source-Projects, where you can follow along and stay updated on our progress!!!

Key Takeaways and Highlights from DroidconKE and FlutterconKE

Khalid Hussein — Fri, 15 Nov 2024 15:42:08 +0000

This year 6th-8th November, three of us, Khalid Hussein, Cheryl Owala, and Tomlee Abila had the incredible privilege of representing Kisumu Gophers at two of Africa’s biggest tech events DroidconKE and FlutterconKE. If you’ve ever wondered what it’s like to be part of an event that brings together the sharpest minds in tech, the energy, the buzz, and the sheer scale of it all... let us just say, it was an experience like no other.

Day 1: Community Vibes

The first was all about community and we couldn’t have asked for a better start. The day kicked off with a keynote from Tabitha Kavyu a Community Coordinator at Google Crowdsource, who set the tone by highlighting the importance of community-driven growth in the tech space. Tabitha’s talk was a powerful reminder of how communities like ours Kisumu Gophers play a crucial role in shaping the future of technology by fostering collaboration and sharing knowledge.

There’s something incredibly special about hearing inspiring stories from people like Kavyu who’ve been in the game for years. And when you're attending a conference with such a rich mix of both techies and non-techies, the conversations you have are gold.
We met developers from all over Africa and beyond, many of whom were passionate about growing their local tech scenes, just like us. There were discussions about tackling challenges in community growth, creating opportunities for developers, and the power of open-source contributions by Brayan Kai.

Day 2: Android Deep Dive & Networking with Industry Leaders

The second day was packed with deep technical sessions, starting with the keynote on “What’s New in Android” that set the stage for a dive into the latest Android technologies. As Android developers ourselves, we were excited to hear about Jetpack Compose, Kotlin updates, and the upcoming Android 14 features from Paul Onakoya a Lead Android Engineering team at Google Nairobi.

What made the day even more exciting were the workshops. We participated in a hands-on Advanced Layout Animations in Compose with Shared Elements by Rebecca Franks an Android Developer Relations Engineer, where we explored new ways to streamline Android development and create more efficient animations with Jetpack Compose.But the real highlight of Day 2 was the networking; having the opportunity to speak directly with Android developers from other countries. Whether it was over coffee or during lunch breaks, the conversations flowed, and we were able to exchange ideas, discuss challenges, and even brainstorm potential collaborations.

Day 3: Jetpack Compose & Custom Layouts

By the third day, we were fully immersed in the technical side of things. The keynote on custom layouts and graphics in Jetpack Compose was a game changer. We’ve always been excited about Compose, but this session gave us a whole new perspective on how it can be used to build dynamic, responsive UIs without the complexity of traditional XML layouts.

The hands-on workshops were the perfect opportunity to apply what we’d just learned. We experimented with custom layouts, and advanced graphics, and even started building UIs we had only dreamed of before. It was a day of learning, experimenting, and sharing ideas with fellow developers, and we left with fresh insights on how we could improve the apps we build.

Leaving DroidconKE and FlutterconKE, we couldn’t help but feel inspired and motivated. The ideas, insights, and connections we gained during the event have already begun to shape the next steps for Kisumu Gophers. We’re more determined than ever to continue growing the tech community in Kisumu, hosting workshops, hackathons, and meetups, and collaborating with developers across Africa and beyond.

The energy we experienced in Nairobi is something we plan to bring back to Kisumu and beyond. We’re excited about the future of tech in our city and the potential for Kisumu Gophers to be part of the global tech movement. If anything, DroidconKE and FlutterconKE confirmed one thing: the future is bright, and we’re ready to be a part of it.

A big thank you to the organizers, speakers, and everyone who made DroidconKE and FlutterconKE 2024 an unforgettable experience. We left the event with new knowledge, new friends, and a renewed sense of purpose. It was a reminder of why we do what we do: to make a difference, to grow, and to connect with like-minded individuals who share the same passion for technology.

If you’re considering attending next year, don’t hesitate. The experiences and connections you’ll make are priceless. We’re already looking forward to the next event and hope to see you there!

Stay connected with Kisumu Gophers and Zone01Kisumu on social media to learn more about our upcoming events, workshops, and opportunities. Whether you’re a developer or someone interested in the tech world, there’s always room for you in our community.

DEV Community: Khalid Hussein

html-to-markdown v3.8.0

What Is html-to-markdown?

What Changed in v3.8.0

Package identity rebrand: Kreuzberg → Xberg

Node binding consolidation

Swift publish branch fix

Migration Guide

What Else Is in Recent Releases

Learn more

Introducing Crawlberg v1.0.0

What Is Crawlberg?

What v1.0.0 Declares Stable

Project rename: kreuzcrawl → crawlberg

Tiered dispatch engine

WAF detection with hot-reload fingerprints

SSRF defense, on by default

Memory-bounded streaming crawl

MCP server and full CLI parity

Public substrate parsers

Deep Technical Highlights

Escalation budget injection

Lock-free corpus hot-reload

SSRF at every redirect hop

Browser pool lifecycle

Asset downloads through the SSRF filter

Performance Implications

Language Bindings Spotlight

Python

JavaScript / Node.js

PHP

Breaking Changes / Compatibility Notes

Upgrade Guide

Step-by-step for existing kreuzcrawl users

Operational Guidance for Production

Tuning recommendations

Observability checklist

Failure modes and mitigations

Security and Responsible Crawling

Contributor and Ecosystem Notes

FAQ

Learn more

Introducing kreuzcrawl v0.3.0

At a glance

Four new language bindings

Memory-bounded streaming

SSRF defense on by default

What is refused

DNS rebinding mitigation

Redirect-chain re-validation

Opting out

WAF-aware tiered dispatch

Tiers and escalation strategies

WAF detection: Aho-Corasick over a TOML corpus

Hot-reload for live environments

Per-domain EWMA state

Configuring dispatch

MCP server at CLI parity

Safety annotations

Transport

Expanded CLI

Standalone robots.txt and sitemap parsers

Browser pool and executor injection

Observability

API stability

Upgrading from v0.2.0

What's next

Resources

SpacetimeDB and Reducers

Tables and Schema

Reducer Semantics

Deterministic Design and Side-Effect Management

Testing Reducers

Example Patterns

Observability

Memory-Mapped I/O for Handling Files Larger Than RAM

Memory Limitations

Memory-Mapped I/O

How Memory Mapping Works

Virtual Memory Mapping

Project rename: `kreuzcrawl` → `crawlberg`

Step-by-step for existing `kreuzcrawl` users