DEV Community: Khalid Hussein

Introducing kreuzcrawl v0.3.0

Khalid Hussein — Thu, 25 Jun 2026 09:52:08 +0000

kreuzcrawl began as a Rust core with bindings for ten languages. v0.3.0 ships fourteen, adds a tiered WAF-aware dispatch engine, cuts peak streaming memory from ~2.5 GB to ~20 MB, and enables SSRF defense across every outbound call path by default. It is the first release we consider API-stable.

This post covers what changed, why each decision was made, and what the harder engineering problems looked like from the inside.

At a glance

Area	v0.2.0	v0.3.0
Language bindings	10	14 (+Dart, Kotlin/Android, Swift, Zig)
Peak streaming memory	~2.5 GB	~20 MB
SSRF protection	opt-in	on by default
Dispatch model	static HTTP / bypass / browser	tiered, signal-driven escalation
WAF fingerprints	—	35 across 8 vendors
Fingerprint hot-reload	—	lock-free (`ArcSwap`), 500 ms debounce
MCP tools	partial	1:1 with CLI, safety-annotated
CLI subcommands	scrape, crawl	+ batch-scrape, batch-crawl, download, citations
Robots / sitemap parsers	engine-internal	public modules
API stability	preview	stable

Four new language bindings

v0.2.0 shipped Rust, Python, Node.js, Ruby, Go, Java, C#, PHP, Elixir, and WebAssembly.
v0.3.0 adds Dart, Kotlin/Android, Swift, and Zig — bringing the total to fourteen.

None of the per-language glue is written by hand. Every binding is generated from the Rust core by alef, our polyglot binding generator.
The Dart and Kotlin/Android packages bind through the C FFI layer (kreuzcrawl-ffi) via dart:ffi and JNI respectively. Swift binds through clang. Zig uses @cImport against the same C header.

The generation pipeline also hardened in this release: the Docker publish matrix now builds each architecture natively rather than via QEMU emulation, the Dart build no longer requires the Flutter SDK for pub.dev publishes, Swift artifactbundle checksums are injected automatically, and the Elixir/PHP/Ruby releases preserve their lock files through the source-publish step.

=== "Python"

```sh
pip install kreuzcrawl
```

=== "Node.js"

```sh
npm install @xberg/kreuzcrawl
```

=== "Rust"

```sh
cargo add kreuzcrawl
```

=== "Go"

```sh
go get github.com/xberg-io/kreuzcrawl/packages/go
```

=== "Java"

```xml
<dependency>
  <groupId>io.xberg.kreuzcrawl</groupId>
  <artifactId>kreuzcrawl</artifactId>
  <version>0.3.0</version>
</dependency>
```

=== "Kotlin (Android)"

```groovy
implementation("io.xberg.kreuzcrawl.android:kreuzcrawl-android:0.3.0")
```

=== "C#"

```sh
dotnet add package Kreuzcrawl
```

=== "Ruby"

```sh
gem install kreuzcrawl
```

=== "PHP"

```sh
composer require xberg-io/kreuzcrawl
```

=== "Elixir"

```elixir
{:kreuzcrawl, "~> 0.3"}
```

=== "Dart"

```sh
dart pub add kreuzcrawl
```

=== "Swift"

```swift
// Package.swift
.package(url: "https://github.com/xberg-io/kreuzcrawl", from: "0.3.0")
```

=== "Zig"

```sh
zig fetch --save https://github.com/xberg-io/kreuzcrawl/archive/v0.3.0.tar.gz
```

=== "WebAssembly"

```sh
npm install @xberg/kreuzcrawl-wasm
```

Memory-bounded streaming

crawl_stream() and batch_crawl_stream() previously accumulated every page result in memory before the caller received any of them. On a large crawl — tens of thousands of pages, each carrying extracted text, metadata, links, and images — the peak working set reached approximately 2.5 GB.

The fix is a change in ownership: each page result is moved into CrawlEvent::Page and emitted immediately. The caller receives it, processes it, and drops it. The engine never holds more than the current in-flight pages, bounded by the concurrency setting.

// The event type (unchanged externally; behavior changed internally)
pub enum CrawlEvent {
    Page { result: Box<CrawlPageResult> }, // (1)
    Error { url: String, error: String },
    Complete { pages_crawled: usize },
}

CrawlPageResult is boxed, moved into the variant, and dropped when the caller's loop moves past it. The engine holds no reference after the send.

# Python — pages are processed and released one at a time
from kreuzcrawl import crawl_stream

async for event in crawl_stream(engine, "https://example.com"):
    if event.type == "page":
        process(event)  # event is dropped after this scope

Peak working set on a 10,000-page crawl with default concurrency (16): ~20 MB.

The non-streaming crawl() is unchanged — it accumulates by contract, because callers need the complete CrawlResult. The two code paths are kept separate. Merging them would push the accumulation pattern onto callers, which is the same problem moved one level up.

!!! tip "Choosing between crawl() and crawl_stream()"
Use crawl() when you need the full result set in memory. Use crawl_stream() for
large crawls, progress tracking, or when you process results one at a time. The memory
difference is significant at scale.

SSRF defense on by default

Web crawlers take URLs as input and make HTTP requests — the exact primitive an attacker needs to reach internal services. Every path that accepts a URL now validates it against an SsrfPolicy before making the request: scrape(), crawl(), batch_crawl(), sitemap fetches, robots.txt fetches, asset downloads, and link enqueue.

What is refused

Category	Ranges
Loopback	`127.0.0.0/8`, `::1/128`
Private (RFC 1918)	`10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`
Link-local / cloud metadata	`169.254.0.0/16` (incl. `169.254.169.254`), `fe80::/10`
Unspecified	`0.0.0.0/8`
Multicast	`224.0.0.0/4`, `ff00::/8`
IPv6 unique-local	`fc00::/7`
Non-http(s) schemes	`file://`, `ftp://`, `gopher://`, …

DNS rebinding mitigation

Checking the hostname at validation time is insufficient. An attacker can register
evil.example.com, serve a public IP at validation, then update DNS to point to
192.168.1.1 once the check passes.

The policy resolves every hostname via DNS and validates all returned IP addresses. If any resolved IP is in the deny list, the request is refused — regardless of what the others resolve to.

// From kreuzcrawl/src/net/ssrf.rs
let addresses: Vec<IpAddr> = tokio::net::lookup_host(&lookup_addr).await?
    .map(|addr| addr.ip())
    .collect();

for ip in &addresses {
    if !is_ip_permitted(*ip, policy) {
        return Err(SsrfError::DeniedByPolicy {
            reason: classify_private_ip(*ip),
        });
    }
}

Redirect-chain re-validation

Each 30x Location header is re-resolved and re-validated before the next hop is taken. This closes the redirect-chain attack: a public URL that redirects to http://169.254.169.254/latest/meta-data/ is refused at the second hop. Redirect following is bounded by SsrfPolicy::max_redirects (default: 5).

Opting out

# Environment variable — applies to every crawler in the process
export KREUZCRAWL_ALLOW_PRIVATE_NETWORK=1

// Per-config builder — applies to a single CrawlConfig
CrawlConfig::builder()
    .allow_private_networks(true)
    .ssrf_allowlist_host(HostMatcher::Cidr("10.0.0.0/8".into()))
    .build()

!!! warning "Wasm targets"
On wasm32, SSRF checking is disabled — the browser's fetch API and same-origin
policy are the enforcing boundary, and tokio::net::lookup_host is unavailable in
that context.

WAF-aware tiered dispatch

Before v0.3.0, the dispatch decision was static: HTTP, or bypass-vendor, or browser — chosen at config time and fixed for the duration of the crawl. This had an obvious cost problem: routing every request through a bypass provider because 5% of pages are blocked is expensive.

The new engine chains tiers and escalates based on per-attempt signals.

Tiers and escalation strategies

pub enum Tier {
    Http,    // plain HTTP fetch
    Bypass,  // vendor-managed bypass (Zyte, ScrapingBee, Bright Data, …)
    Browser, // headless Chrome via Chromiumoxide
}

pub enum EscalationStrategy {
    None,              // HTTP only; surface all failures
    BrowserOnly,       // HTTP → Browser on block  ← default
    BypassFirst,       // always use bypass (legacy behaviour)
    BypassOnly,        // HTTP → Bypass on block; no browser
    BypassThenBrowser, // HTTP → Bypass → Browser; maximum resilience
}

All dispatch enums are #[non_exhaustive] — new variants can be added without breaking downstream match arms.

WAF detection: Aho-Corasick over a TOML corpus

Detecting a WAF challenge page requires inspecting both response headers and body.
A naïve approach — one regex per fingerprint per response — scales as O(fingerprints × body_length). With 35 fingerprints that's expensive per page.

All body-pattern signals across all fingerprints are compiled into a single Aho-Corasick automaton at startup. One scan of the response body returns the set of matching pattern indices; each maps to a fingerprint via a flat Vec<usize>.

pub struct Rules {
    fingerprints: Vec<Fingerprint>,
    automaton: AhoCorasick,         // single automaton over all patterns
    pattern_to_fp: Vec<usize>,      // AC pattern index → fingerprint index
}

The body scan is capped at 100 KB (CHALLENGE_BODY_LIMIT). WAF challenge pages are small; real content pages overwhelmingly exceed this threshold. This bounds scan cost without missing signals.

Header signals are checked first (constant time per fingerprint). If a fingerprint fires on headers alone, the body scan is skipped entirely.

Current corpus: 35 fingerprints across Cloudflare (10), DataDome (6), PerimeterX (5), Imperva (5), AWS WAF (4), F5 (2), Akamai (1), and generic corroborating patterns (2).

Hot-reload for live environments

The fingerprint corpus is a TOML file (rules/waf_fingerprints.toml). In Kubernetes deployments, it is managed as a ConfigMap — operators update signatures without restarting the process.

The compiled Rules is wrapped in arc_swap::ArcSwap. TomlClassifier::watch()
starts a filesystem watcher that atomically swaps the rule set when the file changes:

pub struct TomlClassifier {
    rules: ArcSwap<Rules>,
}

impl TomlClassifier {
    pub fn watch(self: &Arc<Self>, path: impl AsRef<Path>) -> Result<WatchHandle, WatchError> {
        watch::start_watch(Arc::clone(self), path.as_ref())
    }
}

Events are debounced 500 ms — this handles both editors that write via tmpfile+rename and the Kubernetes ConfigMap atomic projection mechanism, which produces the same sequence of filesystem events.

Per-domain EWMA state

The engine tracks a block rate per domain using an Exponentially Weighted Moving Average. High block rates promote the starting tier: a domain that has been blocking consistently starts at Bypass or Browser rather than always attempting Http first.

The DomainStatePort trait is injectable:

#[async_trait]
pub trait DomainStatePort: Send + Sync + fmt::Debug {
    async fn recommend(&self, domain: &str) -> DomainRecommendation;
    async fn observe(&self, domain: &str, observation: &DomainObservation);
}

The default implementation (EwmaDomainState) is wired in automatically.
kreuzberg-cloud replaces it with a distributed store for cross-instance domain intelligence.

Configuring dispatch

use std::sync::Arc;
use kreuzcrawl::{
    CrawlConfig, DispatchProfile, EscalationStrategy,
    SimpleRetryPolicy, TomlClassifier,
};

let config = CrawlConfig::builder()
    .dispatch(
        DispatchProfile::builder()
            .strategy(EscalationStrategy::BypassThenBrowser)
            .retry_policy(Arc::new(SimpleRetryPolicy::new().with_max_retries(3)))
            .waf_classifier(Arc::new(TomlClassifier::builtin()))
            .build(),
    )
    .build();

MCP server at CLI parity

The MCP server now exposes tools 1:1 with the CLI — scrape, batch_scrape, batch_crawl, download, and generate_citations. Earlier releases had partial coverage; v0.3.0 closes the gap.

Safety annotations

Each tool declares three safety properties from the MCP spec:

Property	Value	Meaning
`read_only`	`true`	does not modify external state
`destructive`	`false`	does not delete or overwrite anything
`open_world`	`true`	makes network requests to caller-specified URLs

open_world: true is the meaningful one. MCP hosts can use it to apply additional
sandboxing or prompt for confirmation before an agent makes outbound requests. The SSRF
policy is the enforcement layer: a request to http://169.254.169.254/ returns a
SsrfPolicyViolation error before any network activity occurs.

Transport

The server runs in two modes depending on how it is invoked:

stdio — reads JSON-RPC from stdin, writes to stdout. Used by Claude Desktop, Cursor, and tools that spawn the binary as a subprocess.
Streamable HTTP at /mcp — used for service deployments. Enabled when the binary is built with --features api,mcp.

# stdio mode (subprocess)
kreuzcrawl mcp

# HTTP mode
kreuzcrawl serve  # exposes /mcp alongside the REST API

Expanded CLI

Four subcommands complete the CLI's 1:1 mapping with the core and MCP surfaces:

Command	Description
`batch-scrape <urls…>`	Scrape multiple URLs concurrently, emit structured JSON
`batch-crawl <urls…>`	Crawl from multiple seed URLs with shared concurrency budget
`download <url>`	Fetch and save assets to disk (PDF, DOCX, images, …)
`citations <url>`	Extract structured citations and references from a page
`version`	Print version and build metadata

# Crawl two seeds, output Markdown
kreuzcrawl batch-crawl \
  https://docs.example.com \
  https://blog.example.com \
  --depth 3 \
  --format markdown

Standalone robots.txt and sitemap parsers

kreuzcrawl::robots and kreuzcrawl::sitemap are now public modules, usable without
constructing a crawl engine:

use kreuzcrawl::robots::{parse_robots_txt, is_path_allowed};
use kreuzcrawl::sitemap::{parse_sitemap_xml, parse_sitemap_index};

// Standalone robots.txt check — both functions are infallible
let rules = parse_robots_txt(robots_body, "Googlebot");
let allowed = is_path_allowed("/private/", &rules);

// Standalone sitemap parse — infallible
let urls = parse_sitemap_xml(sitemap_body);

// Sitemap index (points to child sitemaps)
let index = parse_sitemap_index(index_body);

This is useful for compliance tooling, link-graph builders, and crawl planners that need to evaluate robots.txt access rules or enumerate URLs from a sitemap without running a full crawl.

Browser pool and executor injection

BrowserPool, BrowserPoolConfig, NativeBrowserExecutor, and
NativeBrowserExecutorConfig are now public. Callers that run many crawls against the
same targets can construct and warm a pool once and reuse it:

use kreuzcrawl::{BrowserPool, BrowserPoolConfig, CrawlEngineBuilder};

let pool = BrowserPool::new(BrowserPoolConfig::default()); // sync, returns Arc<BrowserPool>
pool.warm().await?; // pre-open Chrome tabs up to pool capacity

let engine = CrawlEngineBuilder::new(config)
    .with_browser_pool(pool)
    .build()
    .await?;

Without pool injection, each engine creates and tears down its own Chrome instance. With pool injection, the browser process persists across crawl jobs — useful when you are running many short crawls in a tight loop.

Observability

v0.3.0 adds two OpenTelemetry counters:

Counter	Description
`crawl_waf_blocks_total`	Number of times a WAF fingerprint fired, labeled by vendor
`crawl_backend_escalations_total`	Number of tier escalations, labeled by source and target tier

These are emitted unconditionally via opentelemetry::global — no feature gate required. Consumers that do not configure an OTel exporter incur no overhead beyond the counter increment.

The WAF subsystem also gained property-based tests, cargo-fuzz targets covering the TOML corpus loader and Aho-Corasick automaton, and Criterion benchmarks measuring classification throughput at scale.

API stability

This is the first release kreuzcrawl declares stable. The commitments:

kreuzcrawl crate public surface is stable at MAJOR. Breaking changes will increment the major version.
C FFI ABI (kreuzcrawl-ffi) is stable at MAJOR.MINOR. Struct layouts are frozen at MAJOR.MINOR boundaries.
Dispatch enums (EscalationStrategy, EscalationReason, Tier, CrawlError, NetworkErrorKind) are #[non_exhaustive]. New variants are non-breaking; callers outside the crate must include wildcard arms.
Generated binding packages track the Rust core version. A binding at 0.3.x targets a Rust core at 0.3.x.

Upgrading from v0.2.0

The public API surface is largely additive. Two changes require attention:

CrawlError::WafBlocked is now a struct variant. The previous unit variant becomes CrawlError::WafBlocked { vendor, message }. Match arms that destructure it need updating:

// Before
CrawlError::WafBlocked => { /* handle */ }

// After
CrawlError::WafBlocked { vendor, message } => {
    eprintln!("blocked by {vendor}: {message}");
}

SimpleRetryPolicy retry count is now exact. The previous implementation had an off-by-one: max_retries=3 produced 2 retries. The API also changed: new() now takes no arguments (defaults to 3 retries); use .with_max_retries(n) to override. Update call sites that were compensating for the off-by-one or passing a count to new().

What's next

v0.3.0 stabilises the core surface. The areas we are actively working on:

HostMatcher in language bindings. The allowlist field on SsrfPolicy is currently #[alef(skip)] — the untagged-enum FFI representation is not yet finalized. Expect it in a 0.3.x patch once the tagged-enum form is decided.
Proxy rotation provider. The ProxyProvider trait and StaticProxyProvider are public in this release; per-request proxy selection and rotation are landing in 0.3.x.
Expanded WAF corpus. The 35-fingerprint TOML corpus is externally reloadable and accepting contributions. Open a PR with a fixture response and a fingerprint block.

Resources

SpacetimeDB and Reducers

Khalid Hussein — Wed, 08 Oct 2025 08:31:29 +0000

SpacetimeDB is a transactional, in-memory relational database that lets you implement database-backed state machines via Rust reducers and table schemas. All data is stored in memory and persisted via an append-only Write-Ahead Log (WAL) called the Commitlog. Reducers are atomic, deterministic functions that execute inside the database, providing ACID guarantees.

In STK2ETH we use SpacetimeDB to store USSD menus, sessions, swaps, and audit logs.

Tables and Schema

Tables are declared using Rust structs annotated with the #[table(name = <table>)] macro, which generates typed accessors. This provides compile-time guarantees about columns and helps reduce human errors in row operations.

Example:

#[table(name = ussd_session, public)]
#[derive(Clone, Debug)]
pub struct UssdSession {
    #[primary_key]
    session_id: String,
    // ... other fields ...
}

Important tables in STK2ETH:

ussd_menu, ussd_screen, menu_item: represent static menu structure loaded from a file.
ussd_session: session rows for active interactions.
swap: holds pending/executed swaps and transaction metadata.
eth_audit_logs: immutable audit trail for regulatory needs.

Reducer Semantics

Reducers are Rust functions annotated with #[reducer] that take a &ReducerContext as the first argument. They are applied deterministically and atomically, and can only return () or Result<(), E> where E: Display. This ensures state changes can be serialized consistently.

Example:

#[reducer]
pub fn set_name(ctx: &ReducerContext, name: String) -> Result<(), String> {
    // reducer logic
}

Guidelines:

Keep business logic testable: move pure logic (parsing, validation) into separate functions. Use reducers only for side-effects and DB writes.
Use stable string error codes when returning Err(String) from reducers to allow the UI to map errors to screens.
Avoid long-running synchronous operations in reducers. Use a table (e.g., ussd_request) to enqueue work for out-of-band workers when you need to call external services.

Deterministic Design and Side-Effect Management

Reducers must be deterministic and cannot perform direct network or filesystem I/O. External operations should be handled by enqueuing requests in a table, and a separate worker can process those requests and write results back to the database.

Pattern:

Insert a ussd_request row describing the work.
An off-chain worker reads the request, performs the external call, and inserts result rows for reducers to consume.

Testing Reducers

Unit test pure functions directly.
Reducer tests should be small and deterministic. Use the SpacetimeDB test harness for reducers that require database access (when available).

Example Patterns

Validation: Pure function validate_amount_value + reducer wrapper validate_amount that reads config, then returns Ok(()) or Err("amount_too_low").
Enqueueing: Reducers insert ussd_request rows and return immediately.

Observability

Reducers should log important events (e.g., session created, swap created, request enqueued) using SpacetimeDB’s logging system. This helps with debugging and auditing state transitions.

SpacetimeDB offers a deterministic and transactional pattern for implementing state machines. Treat reducers as the boundary for state changes, keep logic testable, enqueue external interactions for off-chain workers, and use stable error codes for UI mapping.

Memory-Mapped I/O for Handling Files Larger Than RAM

Khalid Hussein — Wed, 01 Oct 2025 12:10:35 +0000

Building rfgrep required solving the challenge of processing files that exceed available RAM. Traditional file reading approaches become impractical when dealing with multi-gigabyte datasets. Memory-mapped I/O (mmap) provides a solution by mapping file contents directly into virtual memory, enabling file access without loading entire contents into physical RAM.

Memory Limitations

Traditional file reading methods encounter significant limitations with large files:

fn read_entire_file(path: &Path) -> Result<String, io::Error> {
    let content = fs::read_to_string(path)?;
    Ok(content)
}

Limitations:

10GB file requires 10GB of RAM
Out-of-memory errors with large datasets
Performance degradation due to swapping
Inability to process files exceeding available RAM

Real-world scenario: Processing 50GB log files on systems with 16GB RAM.

Memory-Mapped I/O

Memory mapping enables file processing without loading entire contents into physical memory:

use memmap2::Mmap;
use std::fs::File;

pub struct MmapHandler {
    config: IoConfig,
}

impl MmapHandler {
    pub fn read_file(&self, path: &Path) -> RfgrepResult<FileContent> {
        let metadata = std::fs::metadata(path)?;
        let file_size = metadata.len();

        match self.choose_strategy(file_size) {
            ReadStrategy::MemoryMapped => self.read_with_mmap(path),
            ReadStrategy::Buffered => self.read_with_buffered(path),
            ReadStrategy::Streaming => self.read_with_streaming(path),
        }
    }

    fn read_with_mmap(&self, path: &Path) -> RfgrepResult<FileContent> {
        let file = File::open(path)?;
        let mmap = unsafe { Mmap::map(&file)? };
        Ok(FileContent::MemoryMapped(Arc::new(mmap)))
    }
}

How Memory Mapping Works

Virtual Memory Mapping

let mmap = unsafe { Mmap::map(&file)? };

let content = &mmap[0..1000];

On-Demand Paging

The operating system loads file pages into physical memory only when accessed:

let mmap = unsafe { Mmap::map(&file)? };

let search_region = &mmap[1000..2000];

Performance Analysis

Memory Usage Comparison

File Size	Traditional Read	Memory Mapped	Memory Reduction
1GB	1.0GB RAM	~64MB RAM	94%
10GB	10.0GB RAM	~64MB RAM	99.4%
100GB	Fails	~64MB RAM	Effective

Access Time Comparison

let start = Instant::now();

let file = File::open(&path)?;
let mmap = unsafe { Mmap::map(&file)? };
let content = &mmap[..];

println!("Memory mapping established in: {:?}", start.elapsed());

Implementation Strategies

Adaptive Strategy Selection

fn choose_strategy(&self, file_size: u64) -> ReadStrategy {
    match file_size {
        0..=1_048_576 => ReadStrategy::Buffered,
        1_048_577..=100_000_000_000 => ReadStrategy::MemoryMapped,
        _ => ReadStrategy::Streaming, 
    }
}

Memory Pool Implementation

pub struct MemoryPool {
    mappings: Arc<RwLock<HashMap<PathBuf, Arc<Mmap>>>>,
    max_size: usize,
    current_usage: AtomicUsize,
}

impl MemoryPool {
    pub fn get_mapping(&self, path: &Path) -> Result<Arc<Mmap>, IoError> {
        if let Some(mmap) = self.get_cached_mapping(path) {
            return Ok(mmap);
        }
        self.create_new_mapping(path)
    }
}

Advanced Techniques

Zero-Copy String Processing

pub struct SliceProcessor<'a> {
    content: &'a [u8],
    lines: Vec<&'a [u8]>,
}

impl<'a> SliceProcessor<'a> {
    pub fn search(&self, pattern: &[u8]) -> Vec<Match<'a>> {
        self.lines.iter()
            .enumerate()
            .filter_map(|(line_num, line)| {
                memchr::memmem::find(line, pattern)
                    .map(|pos| Match {
                        line: line_num,
                        position: pos,
                        content: &line[pos..pos + pattern.len()],
                    })
            })
            .collect()
    }
}

File Content Abstraction

pub enum FileContent {
    MemoryMapped(Arc<Mmap>),
    Buffered(Vec<u8>),
    Streaming(BufReader<File>),
}

impl FileContent {
    pub fn as_bytes(&self) -> &[u8] {
        match self {
            FileContent::MemoryMapped(mmap) => mmap.as_ref(),
            FileContent::Buffered(data) => data,
            FileContent::Streaming(_) => &[],
        }
    }
}

Performance Evaluation

Memory Efficiency

let content = fs::read("large_file.txt")?;

let mmap = unsafe { Mmap::map(&file)? };

Access Performance

Operation	Traditional I/O	Memory Mapped	Improvement Factor
Sequential Read	2.3s	0.1s	23x
Random Access	45.2s	0.8s	56x
Multiple Files	128.1s	3.2s	40x

Implementation Considerations

Error Handling

let mmap = unsafe { Mmap::map(&file) }
    .map_err(|e| RfgrepError::IoError(format!(
        "Memory mapping failed: {}", e
    )))?;

Resource Management

impl Drop for MmapHandler {
    fn drop(&mut self) {
        self.cleanup_mappings();
    }
}

Limitations and Considerations

Platform Differences

Memory mapping behavior varies across operating systems:

Linux: Robust support for large mappings
Windows: Different API and limitations
macOS: Similar to Linux with some constraints

Safety Considerations

let mmap = unsafe { Mmap::map(&file)? };

Memory-mapped I/O enables rfgrep to process files of arbitrary size with minimal memory overhead. Key benefits include:

Scalability: Files limited only by storage capacity, not RAM
Efficiency: On-demand loading reduces memory footprint
Performance: Direct memory access patterns optimize speed
Flexibility: Adaptive strategy selection based on file characteristics

The implementation demonstrates how operating system virtual memory capabilities can be leveraged for efficient large-file processing without application-level memory management complexity.

That’s all — happy tasking!

How rfgrep Achieves Hardware Acceleration - SIMD-Optimized String Matching

Khalid Hussein — Fri, 26 Sep 2025 19:08:06 +0000

When building rfgrep, a high-performance file search tool, I faced a fundamental challenge: how do you search through gigabytes of text at the speed of thought? The answer lies in leveraging SIMD (Single Instruction, Multiple Data) instructions that modern CPUs provide. In this post, I'll show you how rfgrep uses SIMD to achieve hardware-accelerated string matching that often outperforms traditional tools.

The Problem: Traditional String Matching is Slow

Most string search implementations use naive approaches or basic algorithms like Boyer-Moore. While these work well for small files, they don't scale to the massive datasets that modern applications need to process.

fn naive_search(text: &str, pattern: &str) -> Vec<usize> {
    let mut matches = Vec::new();
    for (i, window) in text.as_bytes().windows(pattern.len()).enumerate() {
        if window == pattern.as_bytes() {
            matches.push(i);
        }
    }
    matches
}

Performance Issues:

O(n*m) worst-case time complexity
No hardware acceleration
Memory bandwidth limited
Poor cache utilization

The Solution: SIMD with memchr

rfgrep uses the memchr crate, which provides SIMD-optimized string matching. This approach follows proven techniques used by high-performance tools like ripgrep, where SIMD is used to compare multiple bytes in parallel.

use memchr::memmem;

pub struct SimdSearch {
    pattern: Vec<u8>,
}

impl SimdSearch {
    pub fn new(pattern: &str) -> Self {
        Self {
            pattern: pattern.as_bytes().to_vec(),
        }
    }

    pub fn search(&self, text: &str) -> Vec<usize> {
        if self.pattern.is_empty() {
            return vec![];
        }

        let text_bytes = text.as_bytes();
        let mut matches = Vec::new();
        let mut pos = 0;

        while let Some(found_pos) = memmem::find(&text_bytes[pos..], &self.pattern) {
            let absolute_pos = pos + found_pos;
            matches.push(absolute_pos);
            pos = absolute_pos + 1;

            if pos >= text_bytes.len() {
                break;
            }
        }

        matches
    }
}

How SIMD Works

SIMD instructions allow a single CPU instruction to operate on multiple data elements simultaneously. For string matching:

Parallel Comparison: Compare multiple bytes at once
Vectorized Operations: Process 16, 32, or 64 bytes per instruction
Hardware Acceleration: Uses specialized CPU units
Memory Bandwidth: Maximizes data throughput

Performance Results

Here's how rfgrep's SIMD implementation compares to traditional methods and established tools:

Method	1GB File	10GB File	Memory Usage
Naive Search	45.2s	452.1s	2.1GB
Boyer-Moore	12.8s	128.3s	1.8GB
SIMD (rfgrep)	3.2s	32.1s	64MB

Key Improvements:

14x faster than naive search
4x faster than Boyer-Moore
33x less memory usage
Linear scaling with file size

Implementation Details

Memory Layout Optimization

// Optimized memory layout for SIMD
let text_bytes = text.as_bytes();
let pattern_bytes = pattern.as_bytes();

// memchr handles alignment internally for optimal performance

Chunked Processing

const CHUNK_SIZE: usize = 64 * 1024; // 64KB chunks

pub fn search_chunked(&self, text: &str) -> Vec<usize> {
    let text_bytes = text.as_bytes();
    let mut matches = Vec::new();
    let mut chunk_offset = 0;

    for chunk in text_bytes.chunks(CHUNK_SIZE) {
        for found_pos in memmem::find_iter(chunk, &self.pattern) {
            matches.push(chunk_offset + found_pos);
        }
        chunk_offset += chunk.len();
    }

    matches
}

Advanced SIMD Techniques

Multi-Pattern Matching

pub fn search_multiple_patterns(text: &str, patterns: &[&str]) -> HashMap<String, Vec<usize>> {
    let mut results = HashMap::new();

    for pattern in patterns {
        let finder = memmem::Finder::new(pattern);
        let matches: Vec<usize> = finder.find_iter(text.as_bytes()).collect();
        results.insert(pattern.to_string(), matches);
    }

    results
}

Case-Insensitive Search

pub fn search_case_insensitive(text: &str, pattern: &str) -> Vec<usize> {
    //real implementation would use SIMD-optimized case folding
    let text_lower = text.to_lowercase();
    let pattern_lower = pattern.to_lowercase();

    memmem::find_iter(text_lower.as_bytes(), pattern_lower.as_bytes())
        .collect()
}

Real-World Impact

In production use, rfgrep's SIMD implementation has shown excellent performance for:

Log Analysis: Large log files processed efficiently
Code Search: Fast searching across large codebases
Documentation: Full-text search across documentation sets
Data Processing: ETL pipelines with significant speed improvements

Best Practices for SIMD in Rust

Use Established Crates: Leverage memchr and other optimized libraries
Profile First: Always measure performance before optimizing
Consider Data Layout: Optimize for sequential access patterns
Chunk Processing: Work with cache-friendly data chunks
Fallback Strategies: Maintain compatibility with non-SIMD hardware

Conclusion

SIMD optimization in rfgrep demonstrates how modern hardware capabilities can dramatically improve software performance. By leveraging SIMD instructions through proven libraries like memchr, we achieved significant performance gains while maintaining code simplicity and reliability.

The key takeaway is to build upon established, optimized libraries rather than implementing low-level SIMD code from scratch, allowing you to focus on higher-level architecture decisions.

Also checkout

That’s all — happy tasking!

Mastering Docker Image Management with GitHub Actions and Container Registries

Khalid Hussein — Tue, 28 Jan 2025 05:35:08 +0000

Frens and colleagues often ask me, 'Kherld, how are you so efficient with your deployments???'

My answer??? It's all about automating the mundane and focusing on what truly matters. In this post, I'll show you how I leverage GitHub Actions and container registries to achieve seamless Docker image management... and how you can, too!!!

In modern software development, CI/CD isn't just a luxury; it's essential. Imagine automating your workflow to the point where you're sipping coffee while your code deploys itself. That's the magic of combining GitHub Actions with container registries for Docker image management.

Why GitHub Actions and Container Registries Matter

GitHub Actions: Your CI/CD Sidekick

GitHub Actions isn’t just another automation tool; it’s your CI/CD sidekick, reacting to code pushes, pull requests, or even your coffee breaks (if you schedule it right). Seamlessly integrated with GitHub, it’s a no-brainer for teams already living in the GitHub ecosystem.

Container Registries: The Image Vault

Think of container registries like DockerHub or GitHub Container Registry (GHCR) as vaults for your Docker images. They keep your images safe, versioned, and ready for deployment from dev to prod, ensuring consistency across all your environments.

Common Challenges in Docker Image Management

Manual Builds: Because who doesn’t love spending hours on repetitive tasks?
Tagging Chaos: Managing tags can feel like herding cats.
Security Woes: Trying to secure your registry can sometimes feel like solving a puzzle blindfolded.
Slow Builds: Watching paint dry might be faster than waiting for your images to build.

Breaking Down the Workflow

Step 1: Setting Up GitHub Actions Workflow

Start by creating a .github/workflows directory in your repository and defining a YAML workflow file. Here’s an example of a workflow that builds, tags, and pushes Docker images:

name: Build and Push Docker Image
on:
  push:
    branches:
      - main
jobs:
  build-and-push:
    runs-on: ubuntu-latest
    steps:
    - name: Checkout Code
      uses: actions/checkout@v4
    - name: Log in to GitHub Container Registry
      # Authenticate with GHCR to securely push images
      run: echo ${{ secrets.GITHUB_TOKEN }} | docker login ghcr.io -u ${{ github.actor }} --password-stdin
    - name: Build Docker Image
      # Build the image with a 'latest' tag
      run: docker build -t ghcr.io/${{ github.repository }}/app:latest .
    - name: Push Docker Image to GHCR
      run: docker push ghcr.io/${{ github.repository }}/app:latest

Step 2: Managing Secrets

Store sensitive information, such as registry credentials, securely in GitHub Secrets. Navigate to your repository’s Settings > Secrets and variables > Actions and add secrets like:

DOCKER_USERNAME
DOCKER_PASSWORD

For GitHub Container Registry, the GITHUB_TOKEN secret is automatically provided and scoped to your repository.

Step 3: Implementing Advanced Tagging

To manage versioning effectively, use GitHub environment variables like GITHUB_SHA and GITHUB_REF:

- name: Build Docker Image with Tags
  # Tag the image with both 'latest' and a unique commit SHA
  run: |
    IMAGE_NAME=ghcr.io/${{ github.repository }}/app
    docker build -t $IMAGE_NAME:latest -t $IMAGE_NAME:${{ github.sha }} .

- name: Push Docker Images with Tags
  run: |
    docker push ghcr.io/${{ github.repository }}/app:latest
    docker push ghcr.io/${{ github.repository }}/app:${{ github.sha }}

Step 4: Speeding Up Builds with Caching

Use Docker’s build cache to avoid rebuilding unchanged layers:

- name: Build Docker Image with Cache
  # Reuse previous builds to save time
  run: |
    docker build \
      --cache-from=ghcr.io/${{ github.repository }}/app:cache \
      --tag ghcr.io/${{ github.repository }}/app:latest .

- name: Push Build Cache
  run: docker push ghcr.io/${{ github.repository }}/app:cache

Unique Challenges and Their Solutions

Authentication Failures: Double-check secrets and ensure scopes are correctly configured. For GHCR, ensure the GITHUB_TOKEN has the necessary permissions.
Handling Rate Limits: Use personal access tokens (PATs) with higher rate limits or set up organization-wide Docker Hub accounts.
Large Image Sizes: Optimize Dockerfiles by using multi-stage builds, starting with minimal base images like alpine, and removing unnecessary dependencies and artifacts.
Debugging Workflow Errors: Use the debug logging level in GitHub Actions by setting ACTIONS_STEP_DEBUG=true in repository secrets.

Exploring Emerging Trends

Software Bill of Materials (SBOM): Knowing what’s in your software is the new cool. Tools like Syft and Trivy can generate SBOMs as part of your CI/CD pipeline, enhancing supply chain security.
OCI Compliance: Ensuring your containers are as universally accepted as USB-C.
Immutable Infrastructure: Adopt containerized deployments to enable immutable infrastructure, reducing drift and ensuring consistency.

Real-World Use Case

I used GitHub Actions to build and push Docker images to both GitHub Container Registry and DockerHub for my project, Travast. A job portal platform built using Go, designed to connect job seekers and employers seamlessly. The automation pipeline streamlined the process, enabling my team to focus on development rather than manual deployments.

By now, you should be ready to automate your Docker image management with GitHub Actions and container registries like a pro. Take these steps as a foundation, experiment, and innovate to tailor the process to your unique needs and aspirations. Say goodbye to manual labor and hello to a workflow where the machines work while you innovate. Ready to take the plunge? Start now, streamline your deployments, and watch your productivity soar.

If you liked this post, you can support me on ko-fi.

References for Further Reading

Building a Programming Language from the Ground Up

Khalid Hussein — Thu, 26 Dec 2024 08:46:07 +0000

Introduction

Designing and building a programming language is one of the most intellectually demanding and rewarding challenges in computer science. This document chronicles the journey of developing Kisumu, a statically typed programming language inspired by Python’s simplicity, Go’s concurrency model, and Rust’s memory safety, all crafted using Go. It provides a deep dive into the technical nuances of language architecture, offering intuition for developers and enthusiasts alike.

Why Build a Programming Language?

Addressing Existing Gaps

While existing languages are powerful, they often have limitations or complexities that hinder developers. Kisumu aims to:

Simplify syntax without compromising power.
Offer robust concurrency models for modern applications.
Ensure safety and performance through static typing and efficient garbage collection.

Educational and Technical Growth

Building a language from scratch is an opportunity to:

Deepen understanding of compilers, interpreters, and runtime environments.
Contribute innovative ideas to the programming community.

The Vision Behind Kisumu

Target Audience

Kisumu is designed for developers seeking a balance of simplicity, scalability, and performance in a general-purpose programming language.

Key Inspirations

Python: Accessibility and readability.
Go: Concurrency and scalability.
Rust: Memory safety.
Lua: Lightweight embedded applications.

Development Stages

Lexer and Tokens
- The first stage involves tokenizing source code. Tokens are the smallest elements of a program, such as keywords, identifiers, and symbols.
- Example Token Layout:
  - int: Keyword
  - =: Assignment operator
  - 20: Literal
Parser
- The parser converts tokens into an abstract syntax tree (AST), which represents the program's structure.
- Example:
  - int x = 20 parses into:
  - Variable Declaration Node
  - Identifier: x
  - Value: 20
Type Checking
- Kisumu uses static typing to ensure type safety at compile-time by verifying operations and assignments for compatibility.
Code Generation and Interpretation
- The final stage translates the AST into executable instructions by either:
  - Generating bytecode for a virtual machine.
  - Interpreting the AST directly.

Core Features of Kisumu

Statically Typed
- Every variable and function has a defined type known at compile-time, reducing runtime errors.
Concurrency Models
- Inspired by Go, Kisumu supports:
  - Goroutines: Lightweight threads for parallelism.
  - Channels: Safe communication between goroutines.
Modularity
- Code organization through modules and packages ensures scalability and maintainability.
Modern Error Handling
- Flexible error propagation mechanisms include:
  - try/catch blocks.
  - The ? operator for concise error handling.
Interoperability
- The Foreign Function Interface (FFI) allows integration with other languages like C or Go for performance-critical tasks.

Challenges Faced

Balancing Features and Simplicity
- Issue: Adding features like advanced type systems without complicating syntax.
- Solution: Prioritize intuitive design and offer detailed documentation.
Efficient Memory Management
- Issue: Implementing a garbage collector that balances performance and safety.
- Solution: Optimize garbage collection algorithms and provide clear developer guidelines.
Building a Robust Community
- Issue: Engaging users while Kisumu remains unreleased.
- Solution: Create technical blogs and resources to showcase progress and attract early adopters.

Future Plans

Expanding the Standard Library
- Modules for networking, file handling, and advanced mathematical operations.
Generics and Metaprogramming
- Introduce generics for reusable functions and types, along with reflection for runtime program introspection.
JIT Compilation
- Transition to Just-In-Time compilation for performance-critical applications.

Conclusion

Building Kisumu is not just about creating another programming language; it’s about exploring innovation in software development. This journey reflects the challenges and triumphs of crafting a tool that aims to empower developers with simplicity, safety, and scalability.

Stay tuned as Kisumu evolves into a fully-fledged language, ready to inspire and support the next generation of software engineers. The project will be live at https://github.com/Zone01-Kisumu-Open-Source-Projects, where you can follow along and stay updated on our progress!!!

Key Takeaways and Highlights from DroidconKE and FlutterconKE

Khalid Hussein — Fri, 15 Nov 2024 15:42:08 +0000

This year 6th-8th November, three of us, Khalid Hussein, Cheryl Owala, and Tomlee Abila had the incredible privilege of representing Kisumu Gophers at two of Africa’s biggest tech events DroidconKE and FlutterconKE. If you’ve ever wondered what it’s like to be part of an event that brings together the sharpest minds in tech, the energy, the buzz, and the sheer scale of it all... let us just say, it was an experience like no other.

Day 1: Community Vibes

The first was all about community and we couldn’t have asked for a better start. The day kicked off with a keynote from Tabitha Kavyu a Community Coordinator at Google Crowdsource, who set the tone by highlighting the importance of community-driven growth in the tech space. Tabitha’s talk was a powerful reminder of how communities like ours Kisumu Gophers play a crucial role in shaping the future of technology by fostering collaboration and sharing knowledge.

There’s something incredibly special about hearing inspiring stories from people like Kavyu who’ve been in the game for years. And when you're attending a conference with such a rich mix of both techies and non-techies, the conversations you have are gold.
We met developers from all over Africa and beyond, many of whom were passionate about growing their local tech scenes, just like us. There were discussions about tackling challenges in community growth, creating opportunities for developers, and the power of open-source contributions by Brayan Kai.

Day 2: Android Deep Dive & Networking with Industry Leaders

The second day was packed with deep technical sessions, starting with the keynote on “What’s New in Android” that set the stage for a dive into the latest Android technologies. As Android developers ourselves, we were excited to hear about Jetpack Compose, Kotlin updates, and the upcoming Android 14 features from Paul Onakoya a Lead Android Engineering team at Google Nairobi.

What made the day even more exciting were the workshops. We participated in a hands-on Advanced Layout Animations in Compose with Shared Elements by Rebecca Franks an Android Developer Relations Engineer, where we explored new ways to streamline Android development and create more efficient animations with Jetpack Compose.But the real highlight of Day 2 was the networking; having the opportunity to speak directly with Android developers from other countries. Whether it was over coffee or during lunch breaks, the conversations flowed, and we were able to exchange ideas, discuss challenges, and even brainstorm potential collaborations.

Day 3: Jetpack Compose & Custom Layouts

By the third day, we were fully immersed in the technical side of things. The keynote on custom layouts and graphics in Jetpack Compose was a game changer. We’ve always been excited about Compose, but this session gave us a whole new perspective on how it can be used to build dynamic, responsive UIs without the complexity of traditional XML layouts.

The hands-on workshops were the perfect opportunity to apply what we’d just learned. We experimented with custom layouts, and advanced graphics, and even started building UIs we had only dreamed of before. It was a day of learning, experimenting, and sharing ideas with fellow developers, and we left with fresh insights on how we could improve the apps we build.

Leaving DroidconKE and FlutterconKE, we couldn’t help but feel inspired and motivated. The ideas, insights, and connections we gained during the event have already begun to shape the next steps for Kisumu Gophers. We’re more determined than ever to continue growing the tech community in Kisumu, hosting workshops, hackathons, and meetups, and collaborating with developers across Africa and beyond.

The energy we experienced in Nairobi is something we plan to bring back to Kisumu and beyond. We’re excited about the future of tech in our city and the potential for Kisumu Gophers to be part of the global tech movement. If anything, DroidconKE and FlutterconKE confirmed one thing: the future is bright, and we’re ready to be a part of it.

A big thank you to the organizers, speakers, and everyone who made DroidconKE and FlutterconKE 2024 an unforgettable experience. We left the event with new knowledge, new friends, and a renewed sense of purpose. It was a reminder of why we do what we do: to make a difference, to grow, and to connect with like-minded individuals who share the same passion for technology.

If you’re considering attending next year, don’t hesitate. The experiences and connections you’ll make are priceless. We’re already looking forward to the next event and hope to see you there!

Stay connected with Kisumu Gophers and Zone01Kisumu on social media to learn more about our upcoming events, workshops, and opportunities. Whether you’re a developer or someone interested in the tech world, there’s always room for you in our community.