Most performance optimization focuses on making each operation faster.
HKD Kernel approaches a different question:
What if most of those operations did not need to execute at all?
I’ve been working on HKD Kernel, a native C library for exact sparse and incremental computation.
The target workload looks like this:
A large computation has already been evaluated.
Only a small subset of the inputs changes.
The dependency structure tells us which results can actually change.
HKD recomputes those affected regions instead of repeating the entire calculation.
The important word is exact. The optimized result must equal the result of full recomputation.
What the benchmark measures
The repository contains reproducible benchmarks comparing full recomputation with the HKD incremental path.
Across the benchmark suite currently documented in the repository, the measured mean speedup is roughly 18,000x.
That requires an important qualification:
This does not mean HKD makes arbitrary programs 18,000x faster.
It means that on workloads with sparse changes and reusable state, avoiding redundant computation can produce extremely large reductions in work.
That distinction is important enough that I built the repository around reproducibility rather than a black-box benchmark claim.
What HKD Kernel is not
HKD Kernel:
does not replace the macOS XNU kernel
does not modify CPU microcode
does not disable SIP
does not change processor ALU hardware
It is a user-space native computation library.
Where I think this model is useful
The workloads I’m most interested in include:
dependency graphs
incremental build systems
large simulations with sparse updates
optimization systems
financial/risk recomputation
logistics and scheduling
cached numerical pipelines
The real question is not “how fast is HKD?”
It is:
How much of your current computation is being repeated even though the inputs affecting it never changed?
I’d especially like developers to try to break the benchmark assumptions or suggest workloads where sparse incremental evaluation should fail.
Source, benchmarks and build instructions:
https://github.com/yangofzeal/hkd-kernel
There is a community edition, and a commercial Business option for production-node licensing.
Feedback on the C API and benchmark methodology is especially welcome.
Top comments (0)