DEV Community: François Gauthier

Architecture Deep-Dive: Zero-Allocation Hot Paths and Thread-Local Evictionless Dictionaries in C11

François Gauthier — Wed, 12 Aug 2026 07:29:20 +0000

When ingestion pipelines approach tens of millions of events per second on commodity hardware, traditional logging architectures fail at three specific bottlenecks: heap allocation overhead, thread lock contention, and CPU cache thrashing.

Most structured logging implementations rely on dynamically allocated string buffers, lock-based thread synchronization, or complex LRU (Least Recently Used) cache evictions. At line-rate telemetry scale, these patterns introduce non-deterministic latencies and context switching that degrade throughput.

This article details the architectural design of PULP, a native Windows C11 telemetry engine designed to process, precompress, and write high-volume logs to disk with bounded memory usage and zero dynamic memory allocations on the ingestion hot path.

1. The Zero-Allocation Hot Path

The primary rule of the ingestion hot path is zero malloc calls after initialization. Dynamic memory allocations introduce non-deterministic execution times and allocator lock contention across threads.

Instead of stringifying telemetry fields on the fly, incoming structured events are transformed into fixed 32-byte cache-aligned binary records.

Fixed 32-Byte Serialized Records

Here is the exact memory layout used to guarantee that each event fits perfectly into half a standard 64-byte L1 CPU cache line:

|--------------------------------------------------------------------|
|                        sequence (8 Bytes)                          |
|--------------------------------------------------------------------|
|                       timestamp (8 Bytes)                          |
|---------------------------------|----------------------------------|
|         url_idx (4B)            |          ip_idx (4B)             |
|----------------|----------------|----------------|--------|--------|
| http_code (2B) | duration_ms(2B)|resp_size (2B)  |verb(1B)|flag(1B)|
|----------------|----------------|----------------|--------|--------|

typedef struct __declspec(align(32)) {
    uint64_t sequence;
    uint64_t timestamp;
    uint32_t url_idx;
    uint32_t ip_idx;
    uint16_t http_code;
    uint16_t duration_ms;   // 0-65s
    uint16_t response_size; // 0-65KB
    uint8_t  http_verb;
    uint8_t  flags;         // Bits for SSL, cache, etc.
} SerializedEntry;

(Code snippet representing the core logging struct)

Pre-allocation: All memory pools (thread-local queues, compression buffers, staging blocks) are allocated up front during runtime initialization using _aligned_malloc.
Cache Alignment: The struct is strictly forced to a 32-byte boundary via __declspec(align(32)). This eliminates false sharing and maximizes memory bus efficiency during SIMD register loads.

2. Lock-Free, Evictionless Thread-Local Dictionaries

Sharing a global symbol or dictionary table across multiple worker threads introduces lock contention or synchronization overhead (atomics, CAS loops) that severely caps multi-core scaling.

Multi-Hash Cascade Design

Each worker thread maintains its own isolated dictionary table. To handle collisions without costly dynamic resizing, the cache uses a fully unrolled scalar design with a Multi-Hash Cascade:

L1 Probe (CityHash64): The primary bucket is hit first.
L2 Probe (XXH3_64): If the primary probe range is saturated, it breaks primary clustering using a completely different hashing algorithm.
L3 Probe (CityHash ^ XXH3): A combined hash fallback.

Evictionless Architecture (No LRU)

Implementing eviction algorithms like LRU requires maintaining access order pointers or counters, which degrades throughput.
Instead, the dictionary uses a fixed capacity per block lifecycle.
When a dictionary block reaches capacity, the thread flushes the block to disk and soft-resets the local table index, preserving the data for a future entry, thus extending the buffering effect.
The cost of resetting an integer index is O(1).

3. Vectorized (AVX2) In-Line Anonymization

When handling network telemetry (e.g., IPv4/IPv6 logs), regulatory frameworks often require masking IP addresses before writing to persistent storage. Performing ASCII string parsing and manipulation per packet introduces significant CPU overhead.

The engine uses an AVX2-accelerated inline pipeline for IP masking:

Zero Parsing Strategy: By collecting hextet offsets directly during a single-pass scan, the engine performs IPv6 :: expansion and applies configurable bitmasks (up to 8 hextets) without intermediate string allocations.
Vectorized Loading: It operates directly on the binary representation prior to serialization, guaranteeing that IP masking remains a constant-time operation regardless of the ingestion volume.

4. Semantic Precompression Prior to LZ4

LZ4 is extremely fast at byte-level repetition matching, but it is unaware of structure semantics. Applying semantic transformation before handing data to LZ4 significantly improves the compression ratio without sacrificing processing speed.

Dictionary Tokenization: Highly repetitive string values (URIs, IPs) are converted to 32-bit integer handles (url_idx, ip_idx)
Entropy Reduction: By converting messy ASCII strings into dense, structured, binary arrays, the entropy of the byte stream drops dramatically, allowing LZ4 to find longer match sequences in its sliding window

Under adversarial conditions (1.3 million+ unique file paths and URLs crossed with 5,000+ unique IPv4 addresses), this pipeline sustains over 5.23M logs/sec written to physical disk, achieving a 2.6× compression ratio. On structured production workloads, throughput exceeds 20M logs/sec with 3.6×–5.0× compression on a standard development laptop.

Summary

Achieving tens of millions of events per second does not require specialized hardware or complex distributed clusters. By designing around modern CPU cache lines, eliminating dynamic allocations, removing inter-thread locks via thread-local state, and applying vectorization to data pre-processing, C11 on Windows can deliver deterministic, line-rate performance for high-throughput telemetry.

Source code, decoder, and reproducible benchmarks: github.com/superwired-labs/Pulp

Previous article (full benchmark methodology — 250M logs in 11.5s):
Benchmark

Open-sourced: 250M logs to disk in 11.5s with lossless on-the-fly compression and no logs dropped

François Gauthier — Sat, 08 Aug 2026 08:01:00 +0000

PULP (Precompressed Upstream Layer Pipeline) is a native Windows C11 telemetry engine that preprocesses, batches, dictionary‑encodes, and LZ4‑compresses logs at line rate. On a stock ThinkPad, it sustains millions of logs/second (benchmarked up to 28 millions logs/sec with production grade data) written to disk with 3-6× end‑to‑end compression, fully reversible, and with bounded memory.

Today, the entire engine (source code, binary archive format, decoder, and reproducible benchmark suite) is open on GitHub under a dual license: AGPLv3 or Commercial.

Why Another Telemetry Pipeline?

Modern observability pipelines spend most of their cost not on producing logs, but on moving and storing them.
Cloud vendors charge per GB. Log platforms charge per GB ingested. And most of that volume is redundant: repeated URLs, repeated IPs, repeated templates, repeated metadata.

The question behind PULP was simple:

What if you could shrink log volume at the source, before any data leaves the host, without losing a single byte?

PULP is a high‑throughput gateway that sits in front of Datadog, Splunk, or Loki, reducing data volume at line rate so downstream systems ingest less, store less, and cost less.

Dictionary-based Semantic Precompression Before LZ4

Most pipelines look like this:

Raw Log → LZ4 → Disk

LZ4 is fast, but it's blind: it doesn't know URLs, IPs, verbs, or structure. Compression ratios on raw logs are typically 1.5:1 to 2:1.

PULP inserts a critical stage:

Raw Log → Semantic Precompression → LZ4 → Disk

The precompression stage transforms each log into a low‑entropy binary representation:

Dictionary encoding of URLs and IPs (TLS caches with multi‑hash probing)
Structured serialization (fixed‑width fields, aligned, predictable)
Batching (memory aligned buffers that can fit into the CPU caches)

LZ4 then compresses data that is already shaped for maximum compressibility.

Temporal Evictionless Caching

Traditional caches fight a losing battle: they evict entries based on heuristics (LRU, LFU, TTL), hoping to predict what the application will need next. The problem is that eviction is speculative and costly : you're gambling that what you just threw away won't come back in the next few microseconds, and the gambling mechanism hits hard on the performances.

PULP takes a different approach by exploiting a fundamental property of log streams: temporal locality.

Consecutive logs from the same thread share almost all of their structure. The timestamp is within milliseconds, the URL template repeats, the source IP rarely changes between back-to-back requests. The data that matters now is almost always the data that mattered just now.

Instead of maintaining a global shared cache with complex eviction logic and lock contention, PULP uses thread-local dictionaries that never evict but are recycled:

Key properties:

No eviction policy: entries remain cached for the lifetime of the thread's active batch.
No locks on the hot path: each thread owns its dictionary privately
No lookup misses under burst: the working set is tiny and stable within a temporal window.
Natural reset on flush: when a batch is flushed and the buffer rotates, the dictionary context naturally shifts to the next batch's working set.

The result: dictionary lookups become nearly free during sustained bursts (which is exactly when throughput matters most), and there's no eviction overhead fighting against you during quiet periods.

Benchmark Results

All benchmarks are reproducible. The benchmark tool and datasets are included in the repository.

Test Environment

Parameter	Value
Hardware	Lenovo ThinkPad P14s Gen5
CPU	AMD Ryzen 5 Pro
RAM	96 GB
Storage	NVMe SSD
OS	Windows 11 x64
Compiler	MSVC /arch:AVX2
Timing	`QueryPerformanceCounter`
Disk writes	Win32 `WriteFile()` + `FILE_FLAG_WRITE_THROUGH`

Workload

250,000,000 synthetic web‑style logs with randomized numeric fields and realistic URL/IP cardinality (basic set is 5 million+ entries)

Results

Configuration	Threads	Time	Throughput	LZ4-only ratio	End-to-end ratio	RAM	Lost
High-Performance	6 callers	11.52s	21.71M/s	1.5× (67% of original)	5.16× (19.4%)	~105 MB	0
Economy	1 caller	29.52s	8.47M/s	1.5× (67% of original)	4.66× (21.5%)	~16 MB	0

Both configurations: 0 logs lost, 0 backpressure events.

Adversarial Workload

To stress-test the dictionary encoder under worst-case conditions, we used a deliberately hostile dataset:

Resource	Cardinality
Unique filesystem paths & randomly formed URLs	1,300,000 (preloaded in RAM, selected randomly)
Unique endpoint IPs	5,000 (preloaded in RAM, selected randomly)
Other numeric fields	Generated randomly on-the-fly

Results:

Configuration	Threads	Time	Throughput	LZ4-only ratio	End-to-end ratio	RAM	Lost
High-Performance	6 callers	47.81s	5.23M/s	1.54× (65%)	2.60× (38.5%)	~1.5 GB	0
Economy	1 caller	144.12s	1.73M/s	1.54× (65%)	2.14× (46.7%)	~16 MB	0

Both configurations: 0 logs lost, 0 backpressure events.

Even with 1.3M unique URLs hammered randomly across 6 threads (a scenario designed to thrash any dictionary) semantic preprocessing still improves compression by ~40% over raw LZ4 (65% → 38.5% of original size). No magic, just structure.

Key design choices:

Thread‑local caches (URL/IP dictionaries)
Zero‑allocation hot path (PulpWrite never allocates)
Custom batch size (Chose the desired trade-off between speed and compression efficiency based on your needs and data)
Asynchronous compression
Asynchronous disk writes
Optional GDPR‑style IP anonymization (configurable IPv4/IPv6 masking)
Append‑only binary archive format (decoder included)

Try It Yourself

https://github.com/superwired-labs/Pulp

Build with MSVC (Visual Studio 2022)
See README.md for exact instructions

Run the benchmarks in under 5 minutes.

The repository includes:

Full source code (C11, Windows‑native)
Binary archive decoder (pulp-decode)
A full implementation example that doubles as a benchmark suite with test datasets included
API documentation
Dual license (AGPLv3 or Commercial)

What's Next

From the official roadmap:

Scalar (non‑AVX2) fallback for older CPUs
Rust bindings (safe wrapper around pulp.dll)
OTLP / FluentBit exporter (bridge .bin → OpenTelemetry)
Enhanced integration examples (C#, IIS native modules)
Autosized batch/dictionary tuning improvements
Linux reimplementation (to leverage the kernel primitives)

We're Looking for Help on:

Rust bindings (first PR is welcome!)
Compatibility reports with Windows Server 2019/2022
Documentation improvements and integration examples

Contributions welcome, open an issue or start a discussion.

FAQ

Is this faster than spdlog / quill / fmtlog?
Different goals. Those libraries optimize formatting.
PULP optimizes throughput + compression + disk persistence.

Is it production‑ready?
Yes, the core engine is stable and tested.
Today it is Windows‑only (native Win32, MSVC).

What license?
PULP is dual‑licensed:

AGPLv3 is free for open‑source projects (derivative work must share modifications)
Superwired‑Commercial for closed‑source or OEM use

How do I decode a .bin shard?

The PulpReader executable takes two arguments : a path to the archive .bin file, a path to the output text file.

code block

C:\Pulp\x64\Release\PulpReader.exe C:\logs\shard_134.bin C:\logs\shard_134.txt

Built by Superwired‑Labs.
If this project interests you, a ⭐ on the repo helps a lot.
Questions? Issues? reach out via GitHub

🛡️ Loggr: A Real-Time Logging Engine as a Weapon Against DDoS Attacks

François Gauthier — Sat, 01 Nov 2025 09:20:55 +0000

Introduction

Distributed Denial of Service (DDoS) attacks remain one of the most persistent and costly threats in cybersecurity. They overwhelm infrastructures, obscure visibility, and often leave defenders blind at the very moment they need reliable data the most.

The key to detecting, understanding, and countering these attacks lies in something often underestimated: logs.

Traditional logging systems struggle under pressure. They sample, drop events, or rely on approximate timestamps that make it impossible to faithfully reconstruct the timeline of an attack.

This is precisely the challenge that Loggr was designed to address. Loggr is a high‑performance logging engine capable of ingesting hundreds of millions of events in seconds on standard hardware. Beyond raw throughput, it introduces a critical innovation: absolute temporal fidelity, essential for detection, traceability, and post‑mortem analysis in cybersecurity.

🔍 Real-Time Detection

A DDoS attack is defined by a sudden surge of activity: millions of requests flooding in within seconds.

With conventional pipelines, many events are lost or delayed.
With Loggr, ingestion rates push hardware to its limits — tens of millions of logs per second on commodity machines — ensuring that all traffic is captured as long as the system is not saturated.

Result: security teams can spot anomalies instantly, even before downstream SIEMs or dashboards have processed the data. Loggr acts as a first‑line sensor, maximizing visibility.

🧾 Traceability and Forensics

During an attack, every event matters. Who hit the system, when, and how often?

Loggr records all events that the hardware can absorb, without sampling.
Logs are compressed and stored with a predictable footprint, enabling full retention of the attack for later analysis.

This near‑exhaustive capture is critical for:

Compliance (proving what happened).
Forensic investigations (identifying vectors and patterns).
Proactive defense (training detection models on real attack data).

🎞️ Replay and Post-Mortem

Once the attack is over, the post‑mortem begins. Without reliable logs, it is impossible to replay the exact sequence of events.

Loggr stores events in a strictly deterministic order.
Teams can replay the attack event by event, as if watching it unfold again.

This enables:

Identifying bottlenecks.
Understanding attack propagation.
Strengthening defenses for the future.

🕒 Absolute Temporal Fidelity: Beyond Timestamps

Most logging systems rely on timestamps (milliseconds or microseconds). Under heavy load, multiple events share the same timestamp, making it impossible to know which came first.

Loggr takes a radically different approach:

Each event is assigned an atomic inter‑thread sequence number, strictly increasing across all threads.
Even if two events occur in the same microsecond, they are differentiated and ordered.
This guarantees absolute temporal fidelity, without ambiguity.

In practice, this means that during a DDoS, when millions of requests hit simultaneously, Loggr can still reconstruct the exact order of events — as long as throughput remains within hardware capacity. If saturation occurs, losses are possible, but Loggr pushes those thresholds far beyond traditional solutions.

⚙️ Why It Works

Loggr achieves these results through several design choices:

Preprocessing + compression: entropy reduction before LZ4, achieving up to 5× compression without sacrificing speed.
Lock‑free pipelines: eliminating contention, ensuring no bottlenecks even under extreme load.
Predictable footprint: runs on standard hardware, no exotic infrastructure required. The minimal footprint is 20MB (stable), and allow the capture of 1.5 to 8M+ events per second

📌 Positioning in the Security Ecosystem

Loggr is not meant to replace a SIEM or full observability platform. Instead, it acts as an upstream buffer:

Capture: massive, reliable ingestion.
Compression: reducing volume before storage or transfer.
Forwarding: sending data to existing tools (Splunk, Elastic, Datadog, etc.).

By reducing volume at the source, Loggr makes downstream tools more efficient and cost‑effective.

Conclusion

In cybersecurity, visibility is survival. During a DDoS, losing logs means losing the ability to detect, respond, and learn.

With Loggr, no event is lost as long as the hardware holds the load. Detection is immediate, traceability is maximized, and post‑mortems are faithful to reality thanks to absolute temporal fidelity.

This is not just a logging engine: it is a strategic weapon against one of the oldest and most persistent threats in the digital landscape.

As data volumes continue to grow, upstream compression and absolute temporal fidelity will become essential pillars of resilient cybersecurity pipelines.

Architecture overwiew and detailed benchmarks are available here -> benchmarks and overview

How We Built Loggr, a Logging Library That Handles 20M+ Logs/second on a Laptop

François Gauthier — Fri, 31 Oct 2025 10:27:02 +0000

Sharing our work on high-performance log compression - curious about this community's scaling experiences

The Challenge That Started It All

Most logging systems are built for the cloud era — they assume CPU power, RAM, cheap bandwidth and storage.

But what if you're running edge computing, IoT devices, or just want to keep your cloud bills reasonable?

We set out to build something different.

It began with a simple question: how many logs per second can you realistically process on consumer hardware before hitting a wall?

The goal was audacious: create a logging library so efficient it could handle 250 million logs at the highest rate with efficient on-the-fly compression and live statistics exports on a standard developer laptop without breaking a sweat.

Why This Matters More Than You Think

We discovered that typical log data compresses at 4–6× when you apply smart preprocessing.

That means for every terabyte of logs you're storing, you could be storing just 200–250 GB.

Here's the dirty secret of modern logging: you're probably storing the same data dozens of times.

Error messages, API endpoints, user sessions — they follow patterns.

Yet most systems store each instance as if it were unique.

This translates into concrete egress cost cuts and multiplies long-term storage possibilities.

The Breakthrough: On-the-Fly Preprocessing + Compression

Beyond Standard Compression Techniques

Most solutions throw raw logs at a compressor.

We've developed a preprocessing layer that transforms logs into a more compressible format before even applying LZ4.

This step significantly reduces data entropy, allowing compression to achieve on-the-fly 4–6× ratios where standard approaches plateau at 2–3.5×, or require significant overhead using HC algorithms with cumbersome post-processing pipelines.

Lock-Free (Nearly) Everything

We treated contention as the enemy.

The hot path uses MPMC queues and ring buffers so multiple threads can enqueue logs without blocking each other.

It's like a multi-lane highway where cars merge without stopping.

Batch Compression Magic

Instead of compressing each log individually, we batch them and compress entire chunks.

Combined with preprocessing, this gives LZ4 more patterns to work with, dramatically improving ratios.

Our batching strategy pairs with a unique cache approach that handles millions of unique values within tight hardware constraints.

Positioning in the Observability Ecosystem:

Loggr is not designed to replace full-featured platforms like Datadog or Splunk, but to serve as an upstream gateway — compressing logs at the source before transmission to storage or downstream analysis pipelines. This creates a cost-efficient two-stage architecture where Loggr handles the “heavy lifting” of data reduction, dramatically cutting egress and storage costs while maintaining compatibility with existing tools.

The Moment of Truth: 250 Million Log Test

Laptop: Lenovo P14s Gen 5 (Ryzen 5 Pro, 96 GB RAM, NVMe SSD)

Environment: Windows x64, AVX2 enabled

Data: 1,000 unique URLs × 5,000 endpoints, random IP x URL distribution, assorted with other randomly generated fields

Sample Format:

[249328097] [2025-10-17T15:50:20.721988Z] [/forum/thread/12345.html] [172.16.18.116] [506] [SEARCH] [59466ms] [16802b] [174]

The 250M log benchmark isn't just a party trick — it's proof that with careful engineering, we can handle orders of magnitude more data on the same hardware

Results

With 6 caller threads, 2 MB batch size:

🔥 250,000,000 logs processed
⏱️ 11.52 seconds total
🚀 21.71 million logs/second -> to disk
💾 5:1 compression ratio
🖥️ 100% CPU (6 physical cores used)
📊 105 MB RAM footprint (stable)
✅ 0 lost logs & no back-pressure

With 1 caller thread, 512 MB batch size:

🔥 250,000,000 logs processed
⏱️ 27.17 seconds total
🚀 9.2 million logs/second -> to disk
💾 5:1 compression ratio
🖥️ 20% CPU (1 physical core)
📊 1.8 GB RAM footprint (stable)
✅ 0 lost logs & no back-pressure

With 1 caller thread, 500 KB batch size:

🔥 250,000,000 logs processed
⏱️ 40.32 seconds total
🚀 6.2 million logs/second -> to disk
💾 4.6:1 compression ratio
🖥️ 20% CPU (1 physical core)
📊 16 MB RAM footprint (stable)
✅ 0 lost logs & no back-pressure

Detailed Benchmarks : Here

Real-World Implications

Edge computing: Run comprehensive logging on resource-constrained devices without worrying about RAM and CPU limits.
Cost-conscious teams: Reduce log volume manyfold = lower storage costs, lower egress fees, and potentially lower licensing costs.
High-throughput systems: Maintain detailed logging without becoming I/O bound or drowning in storage costs.
Security traceability: Log all events with minimal resources, maintaining absolute temporal order with unique trans-thread atomic IDs.

The Architecture

Producer Threads → [Lock-free Queue] → Batch Builder → [LZ4 Compression] → Disk Writer

Key Design Decisions

Single C DLL (170KB) with no dependencies (plug & play, interoperable with most environments) featuring :

AVX2-optimized code paths
Highly configurable without complex setup
On-the-fly IP anonymization
Custom compression level (or none)
URL param truncation
Unique per-log cross-thread ID
Instant backup write path
Custom memory footprint
Live usage statistics

Optional cryptographic signing for audit trails (on the roadmap).

When You Might Not Need This

Let’s be honest — not every application needs this level of optimization. If you're processing 10,000 logs per second, traditional solutions work fine.

But if you're dealing with:

Hundred of thousand or million of events per second
Bandwidth-constrained environments
Budgets where cloud costs matter
Regulatory requirements for long-term retention ...then host-side compression becomes incredibly valuable.

The Future of Logging

We believe the next frontier in observability isn't collecting more data — it's being smarter about what we keep and how we store it.
By moving compression to the source, we can maintain detailed audit trails without the traditional cost burden.
The 250M log benchmark isn't just a party trick — it's proof that with careful engineering, we can handle orders of magnitude more data on the same hardware.

And in an era where data growth is outpacing budget growth, that might be the most important optimization of all.

Want to run your own tests? For organizations conducting formal technical evaluations, a limited demo DLL is available. We're particularly interested in hearing about edge cases and workloads where our approach does (and doesn't) work well.

Technical specs: Windows x64, AVX2 required, C API (easy bindings for most languages).

Beyond LZ4 Limits, Logging at high speed with on-the-fly compression

François Gauthier — Thu, 30 Oct 2025 11:07:38 +0000

An introduction, full article linked at the bottom.

TL;DR
We built Loggr: a tiny (170 KB, no external dependencies) native C logging library that preprocesses, batches and compresses logs at line rate. On a Lenovo P14s developer laptop (Ryzen 5 Pro, NVMe) we processed 250,000,000 synthetic web-style logs in 11.52 seconds (21.71 million logs/second), achieving roughly 5× end-to-end (to disk) compression (preprocessing + LZ4) while keeping RAM usage low and zero lost logs. This article explains the architecture, test methodology, exact parameters, benchmark data, limitations, and how to reproduce the tests.

How We Achieved on-the-fly 5× Log Compression Where LZ4 Alone Fails

The preprocessing trick that lets fast compression algorithms achieve heavy compression ratios

The Problem

Most logging systems assume cloud-era resources: unlimited CPU, RAM, and cheap storage. But what if you're running edge computing, IoT devices, or just want to keep cloud bills under control?

We started with a simple question: how many logs can you realistically process on consumer hardware before hitting a wall?

The Breakthrough

Instead of throwing raw logs at LZ4, we preprocess them first - transforming logs into a low-entropy format that compressors love.

Key innovations:

Smart preprocessing** reduces entropy before compression
Lock-free queues** handle 21M+ logs/sec without contention
Batch compression** finds longer patterns for better ratios
Temporal caching** leverages natural log patterns

The Numbers Don't Lie

Tested on a stock Lenovo P14s (Ryzen 5 Pro, NVMe SSD, 96GB RAM)

250 Million Logs - Multiple Configurations
6 threads, 2MB batches:
✅ 250M logs in 11.52 seconds
✅ 21.71 million logs/second -> To disk
✅ 5:1 compression ratio
✅ 105MB RAM footprint (stable)
✅ 0 lost logs

1 thread, 500KB batch (economy mode):
✅ 250M logs in 29.52 seconds
✅ 8.47 million logs/second -> To disk
✅ 4.6:1 compression ratio
✅ 16MB RAM footprint (stable)
✅ 0 lost logs

How It Works

The architecture is centered around ring buffers and lock-free queues.
Producer Threads → [Lock-free Queue] → Batch Builder → [LZ4 Compression] → Storage

Core architecture:

170KB C DLL - zero dependencies
AVX2-optimized code paths
Highly configurable memory footprint (20MB to GB+)
Live telemetry and atomic sequencing

Real-World Impact

Edge computing: Full logging on resource-constrained devices
Cost reduction: 80%+ savings on storage and egress fees
High-throughput systems: Maintain detailed logs without I/O bottlenecks
Security: Complete audit trails with minimal resources

The Bottom Line

This isn't just about faster compression - it's about rethinking logging as a data optimization problem. By moving intelligence upstream, we can handle orders of magnitude more data on the same hardware.

For the complete technical deep-dive with full benchmark methodology and API documentation, check out the full article on Medium:

Loggr: Processing 250M Logs in 11.5s on a Laptop with On-the-Fly 5× Compression

What logging challenges are you facing with your high-throughput applications?