PULP (Precompressed Upstream Layer Pipeline) is a native Windows C11 telemetry engine that preprocesses, batches, dictionary‑encodes, and LZ4‑compresses logs at line rate. On a stock ThinkPad, it sustains millions of logs/second (benchmarked up to 28 millions logs/sec with production grade data) written to disk with 3-6× end‑to‑end compression, fully reversible, and with bounded memory.
Today, the entire engine (source code, binary archive format, decoder, and reproducible benchmark suite) is open on GitHub under a dual license: AGPLv3 or Commercial.
Why Another Telemetry Pipeline?
Modern observability pipelines spend most of their cost not on producing logs, but on moving and storing them.
Cloud vendors charge per GB. Log platforms charge per GB ingested. And most of that volume is redundant: repeated URLs, repeated IPs, repeated templates, repeated metadata.
The question behind PULP was simple:
What if you could shrink log volume at the source, before any data leaves the host, without losing a single byte?
PULP is a high‑throughput gateway that sits in front of Datadog, Splunk, or Loki, reducing data volume at line rate so downstream systems ingest less, store less, and cost less.
Dictionary-based Semantic Precompression Before LZ4
Most pipelines look like this:
Raw Log → LZ4 → Disk
LZ4 is fast, but it's blind: it doesn't know URLs, IPs, verbs, or structure. Compression ratios on raw logs are typically 1.5:1 to 2:1.
PULP inserts a critical stage:
Raw Log → Semantic Precompression → LZ4 → Disk
The precompression stage transforms each log into a low‑entropy binary representation:
- Dictionary encoding of URLs and IPs (TLS caches with multi‑hash probing)
- Structured serialization (fixed‑width fields, aligned, predictable)
- Batching (memory aligned buffers that can fit into the CPU caches)
LZ4 then compresses data that is already shaped for maximum compressibility.
Temporal Evictionless Caching
Traditional caches fight a losing battle: they evict entries based on heuristics (LRU, LFU, TTL), hoping to predict what the application will need next. The problem is that eviction is speculative and costly : you're gambling that what you just threw away won't come back in the next few microseconds, and the gambling mechanism hits hard on the performances.
PULP takes a different approach by exploiting a fundamental property of log streams: temporal locality.
Consecutive logs from the same thread share almost all of their structure. The timestamp is within milliseconds, the URL template repeats, the source IP rarely changes between back-to-back requests. The data that matters now is almost always the data that mattered just now.
Instead of maintaining a global shared cache with complex eviction logic and lock contention, PULP uses thread-local dictionaries that never evict but are recycled:
Key properties:
- No eviction policy: entries remain cached for the lifetime of the thread's active batch.
- No locks on the hot path: each thread owns its dictionary privately
- No lookup misses under burst: the working set is tiny and stable within a temporal window.
- Natural reset on flush: when a batch is flushed and the buffer rotates, the dictionary context naturally shifts to the next batch's working set.
The result: dictionary lookups become nearly free during sustained bursts (which is exactly when throughput matters most), and there's no eviction overhead fighting against you during quiet periods.
Benchmark Results
All benchmarks are reproducible. The benchmark tool and datasets are included in the repository.
Test Environment
| Parameter | Value |
|---|---|
| Hardware | Lenovo ThinkPad P14s Gen5 |
| CPU | AMD Ryzen 5 Pro |
| RAM | 96 GB |
| Storage | NVMe SSD |
| OS | Windows 11 x64 |
| Compiler | MSVC /arch:AVX2 |
| Timing | QueryPerformanceCounter |
| Disk writes | Win32 WriteFile() + FILE_FLAG_WRITE_THROUGH
|
Workload
250,000,000 synthetic web‑style logs with randomized numeric fields and realistic URL/IP cardinality (basic set is 5 million+ entries)
Results
| Configuration | Threads | Time | Throughput | LZ4-only ratio | End-to-end ratio | RAM | Lost |
|---|---|---|---|---|---|---|---|
| High-Performance | 6 callers | 11.52s | 21.71M/s | 1.5× (67% of original) | 5.16× (19.4%) | ~105 MB | 0 |
| Economy | 1 caller | 29.52s | 8.47M/s | 1.5× (67% of original) | 4.66× (21.5%) | ~16 MB | 0 |
Both configurations: 0 logs lost, 0 backpressure events.
Adversarial Workload
To stress-test the dictionary encoder under worst-case conditions, we used a deliberately hostile dataset:
| Resource | Cardinality |
|---|---|
| Unique filesystem paths & randomly formed URLs | 1,300,000 (preloaded in RAM, selected randomly) |
| Unique endpoint IPs | 5,000 (preloaded in RAM, selected randomly) |
| Other numeric fields | Generated randomly on-the-fly |
Results:
| Configuration | Threads | Time | Throughput | LZ4-only ratio | End-to-end ratio | RAM | Lost |
|---|---|---|---|---|---|---|---|
| High-Performance | 6 callers | 47.81s | 5.23M/s | 1.54× (65%) | 2.60× (38.5%) | ~1.5 GB | 0 |
| Economy | 1 caller | 144.12s | 1.73M/s | 1.54× (65%) | 2.14× (46.7%) | ~16 MB | 0 |
Both configurations: 0 logs lost, 0 backpressure events.
Even with 1.3M unique URLs hammered randomly across 6 threads (a scenario designed to thrash any dictionary) semantic preprocessing still improves compression by ~40% over raw LZ4 (65% → 38.5% of original size). No magic, just structure.
Key design choices:
- Thread‑local caches (URL/IP dictionaries)
-
Zero‑allocation hot path (
PulpWritenever allocates) - Custom batch size (Chose the desired trade-off between speed and compression efficiency based on your needs and data)
- Asynchronous compression
- Asynchronous disk writes
- Optional GDPR‑style IP anonymization (configurable IPv4/IPv6 masking)
- Append‑only binary archive format (decoder included)
Try It Yourself
https://github.com/superwired-labs/Pulp
Build with MSVC (Visual Studio 2022)
See README.md for exact instructions
Run the benchmarks in under 5 minutes.
The repository includes:
- Full source code (C11, Windows‑native)
- Binary archive decoder (
pulp-decode) - A full implementation example that doubles as a benchmark suite with test datasets included
- API documentation
- Dual license (AGPLv3 or Commercial)
What's Next
From the official roadmap:
- Scalar (non‑AVX2) fallback for older CPUs
- Rust bindings (safe wrapper around
pulp.dll) - OTLP / FluentBit exporter (bridge
.bin→ OpenTelemetry) - Enhanced integration examples (C#, IIS native modules)
- Autosized batch/dictionary tuning improvements
- Linux reimplementation (to leverage the kernel primitives)
We're Looking for Help on:
- Rust bindings (first PR is welcome!)
- Compatibility reports with Windows Server 2019/2022
- Documentation improvements and integration examples
Contributions welcome, open an issue or start a discussion.
FAQ
Is this faster than spdlog / quill / fmtlog?
Different goals. Those libraries optimize formatting.
PULP optimizes throughput + compression + disk persistence.
Is it production‑ready?
Yes, the core engine is stable and tested.
Today it is Windows‑only (native Win32, MSVC).
What license?
PULP is dual‑licensed:
- AGPLv3 is free for open‑source projects (derivative work must share modifications)
- Superwired‑Commercial for closed‑source or OEM use
How do I decode a .bin shard?
The PulpReader executable takes two arguments : a path to the archive .bin file, a path to the output text file.
code block
C:\Pulp\x64\Release\PulpReader.exe C:\logs\shard_134.bin C:\logs\shard_134.txt
Built by Superwired‑Labs.
If this project interests you, a ⭐ on the repo helps a lot.
Questions? Issues? reach out via GitHub
Top comments (0)