DEV Community: Priyadarshan Singh

How Cilium Cut Restart Times by Reusing Compiled BPF Objects

Priyadarshan Singh — Mon, 02 Nov 2026 13:04:43 +0000

How Cilium Cut Restart Times by Reusing Compiled BPF Objects

If you’ve ever worked with eBPF programs in production, you know that compiling them can be a time-consuming bottleneck, especially during restarts. In Cilium, a popular networking and security project, the agent used to recompile all BPF programs from scratch on every restart. This led to delays stretching into minutes and even caused missed regeneration deadlines in continuous integration (CI) environments.

In this article, we’ll explore the root cause of this inefficiency, the technical details of the caching solution introduced, and the broader engineering lesson about build artifact reuse.

The Problem: Unnecessary Recompilation on Every Restart

Cilium’s agent manages a set of BPF programs that it compiles using clang. Before the change, each time the agent restarted, it triggered a full recompilation of all BPF programs. This was painfully slow — in some cases, endpoint template compilations took up to 553 seconds.

Why did this happen? Because the agent had no way to detect if the BPF source code or its build environment had changed since the last compilation. Without such detection, it always assumed it needed to recompile everything.

This behavior caused two main issues:

Long startup delays: Developers and CI pipelines had to wait for lengthy recompilations even if nothing changed.
Missed deadlines: In CI, where quick feedback loops matter, these delays led to missed regeneration deadlines and slowed down testing.

Root Cause: Lack of Input Hashing and Cache

The core reason for the redundant recompilation was the absence of a mechanism to track build inputs and reuse previously compiled artifacts.

The agent’s compilation process depends on multiple inputs:

BPF source files
Clang compiler configuration and environment
Go build ID (which can affect generated headers)
Generated headers and other build artifacts

Before this change, none of these inputs were hashed or checked for differences. The compilation was always triggered unconditionally, causing expensive clang invocations even if the inputs were identical.

The Solution: Input Hashing and Caching Compiled Objects

The fix introduced a robust caching mechanism keyed by a hash that uniquely identifies the compilation inputs. Here’s how it works in detail:

Hashing all relevant inputs: The system computes a single hash that combines the contents of all BPF source files, the clang configuration and environment variables, the Go build ID, and any generated headers. This produces a unique fingerprint representing the exact compilation context.
Cache lookup on agent restart: When the agent starts, it calculates this hash and checks if compiled BPF objects for this hash already exist on disk.
Reuse or recompile:
- If a matching cached object is found, the agent loads it directly, skipping the expensive compilation.
- If not, it proceeds with a normal compilation and then caches the resulting objects under the new hash.

This caching logic is primarily implemented in pkg/datapath/loader/cache.go. To ensure correctness, new tests were added in cache_test.go and hash_test.go verifying that the hashing accurately reflects input changes and that cached objects are reused appropriately.

This approach ensures that recompilation only happens when something truly changes, drastically reducing restart times.

Takeaway: Hashing Inputs to Cache Build Artifacts Saves Time and Resources

Build systems and runtime components often depend on multiple inputs that rarely change between runs. Without tracking these inputs, systems waste time recompiling or rebuilding unnecessarily.

By hashing all relevant inputs and caching build artifacts keyed by this hash, you can:

Avoid redundant expensive work
Speed up restarts and deployments
Reduce CPU and memory usage during builds
Improve developer and CI pipeline productivity

In Cilium’s case, this change turned restart times from minutes down to seconds, enabling faster iteration and more reliable CI.

If you’re building systems that compile code or generate artifacts dynamically, consider implementing a similar input hashing and caching mechanism.

This improvement came from the pull request titled "loader: Reuse compiled BPF objects across agent restarts" in the cilium/cilium repository.

Read the full breakdown on nossletter.tech.

Originally published on Nossletter.

Enhancing ggml with Fused Sparse-Attention Masks and Precision Introspection in v0.28.0

Priyadarshan Singh — Wed, 28 Oct 2026 21:26:05 +0000

Why Fused Sparse-Attention Masks Matter for ML Performance

Sparse attention mechanisms have become a cornerstone of efficient transformer implementations, enabling models to scale by focusing computation on the most relevant parts of the input. However, supporting these sparse patterns efficiently in low-level ML compute libraries requires careful design of operations and data flow. The recent ggml v0.28.0 release tackles this challenge head-on by introducing a fused sparse-attention mask operation, alongside precision introspection APIs and cleaner mechanisms for skipping computation slots. These improvements not only simplify backend implementations but also unlock performance gains across CPU, CUDA, and Metal platforms.

The Problem: Inefficient Sparse-Attention Support and Skipping Logic

Prior to this release, ggml lacked a dedicated operation to fuse sparse-attention masks with top-k selection, forcing users to rely on separate steps or less efficient workarounds. This fragmented approach increased overhead and complexity in model implementations that rely heavily on sparse attention.

Additionally, the way ggml handled skipping matrix slots was problematic. Instead of an explicit sentinel value, invalid or skipped matrix IDs were treated as out-of-range indices, which complicated backend logic. This meant that the compute engines had to interpret "magic" out-of-range values as signals to skip computations, leading to brittle and error-prone code.

Moreover, introspecting the precision of compute nodes was limited, which made it harder for developers and optimizations to adapt based on the exact precision used during computation.

The Solution: Introducing GGML_OP_TOP_K_MASK and Cleaner APIs

Fused Sparse-Attention Mask Operation

The centerpiece of this release is the new GGML_OP_TOP_K_MASK operation. This operator fuses the top-k selection with the application of a sparse-attention mask into a single kernel. By doing so, it reduces intermediate memory usage and kernel launch overhead, leading to more efficient execution.

Implementations of this op have been provided across all major backends:

CPU: Optimized multi-threaded code paths leverage SIMD instructions where possible.
CUDA: GPU kernels utilize efficient parallel reductions and masking to minimize warp divergence.
Metal: Metal shaders are crafted to exploit Apple GPUs' architecture for sparse attention.

This cross-platform consistency ensures that models using sparse attention can benefit regardless of hardware.

Explicit Skip Index with -1

To simplify logic around skipping matrix slots, the release introduces support for -1 as a sentinel skip index in the MUL_MAT_ID and ADD_ID operations. This explicit value signals the backend to zero out the corresponding output row and skip computation cleanly.

This change removes the ambiguity of out-of-range indices and allows backends to implement skipping more robustly and efficiently. For example, instead of checking for invalid indices and guessing their meaning, the backend can directly branch on -1 and avoid unnecessary work.

Precision Introspection APIs

Two new APIs, ggml_prec_get_acc() and ggml_prec_get_src(), have been added to query the precision of accumulation and source data on compute nodes. This enables more adaptive optimizations and debugging, as developers can programmatically determine if a node is using FP16, FP32, or other precisions.

Additional Improvements

Scale and sum operations have been relaxed to support non-contiguous inputs, increasing flexibility in tensor layouts.
Various throughput improvements and bug fixes have been applied to all backends, further enhancing stability and performance.

Why This Matters: Takeaways for ML Library Design

This update to ggml highlights several important engineering principles for ML compute libraries:

Fused operations for common patterns: Sparse attention is ubiquitous in modern transformers. Providing fused ops tailored to these patterns reduces overhead and improves maintainability.
Explicit sentinel values simplify backend logic: Using a dedicated skip index makes the code clearer and less error-prone than relying on out-of-range or magic indices.
Precision introspection aids optimization: APIs that expose compute node precision help downstream tooling and adaptive kernels tune performance dynamically.
Cross-platform consistency: Implementing new ops across CPU, CUDA, and Metal ensures broad hardware support and consistent behavior.

Together, these changes make ggml a more robust and performant foundation for ML workloads involving sparse attention and complex tensor operations.

Credit for this deep-dive to the original PR in the ggml-org/ggml repository. Read the full breakdown on nossletter.tech.

Originally published on Nossletter.

Breaking Down Milvus QueryView Coordinator for Better Modularity and Maintainability

Priyadarshan Singh — Tue, 27 Oct 2026 15:53:14 +0000

Why Modular Design Matters in Complex Systems

In large-scale distributed systems like Milvus, coordination components often grow complex as they accumulate diverse responsibilities. The QueryView coordinator in Milvus, responsible for managing load, scheduling, schema lifecycle, and readiness, had become a tightly coupled monolith. This complexity made it challenging to maintain, extend, or even reason about individual parts independently — a common pain point in production systems where scalability and agility are critical.

This article dives into how the Milvus team tackled this challenge by extracting distinct, immutable components within the QueryView coordinator. This change not only improved modularity but also paved the way for more robust scheduling and schema management with clear, stable interfaces.

The Problem: A Monolithic Coordinator

Originally, the QueryView coordinator bundled several responsibilities together:

Load snapshot handling: Capturing the state of data load and replica distribution.
Replica allocation and scheduling: Deciding where and how data replicas should be placed and balanced.
Scheduling cost calculations: Using metrics to reconcile the placement of data efficiently.
Schema lifecycle management: Tracking schema versions and ensuring consistent reads.
Readiness protocols: Managing component readiness and synchronization progress.

All these concerns lived in one coordinator module, leading to complex interdependencies. For example, load snapshots were mutable and intertwined with scheduling logic, making it hard to isolate bugs or introduce new scheduling strategies without risking side effects. Similarly, schema management was tightly coupled with readiness checks, complicating schema evolution.

The Fix: Extracting Immutable Components and Interfaces

The core idea behind the fix was to split the monolithic coordinator into well-defined, immutable components with clear interfaces. This separation enables each part to evolve independently and be tested in isolation.

Immutable Load Snapshots and Placement Management

Instead of mutable shared state, load information is now captured as immutable snapshots. These snapshots represent a consistent view of the cluster load and replica placements at a given point in time. By treating them as immutable, the system avoids side effects and race conditions during scheduling.

Scheduling with Cost-Based Reconciliation Using LogicalSize Weights

Scheduling was enhanced by introducing cost calculations based on immutable LogicalSize weights. This abstraction allows the scheduler to evaluate placement decisions quantitatively, facilitating more efficient and balanced replica allocation. The cost-based reconciliation process compares current and desired placements, driving incremental adjustments.

Schema Lifecycle Management with Certified Read Schemas

Schema handling was extracted into its own component that manages certified read schemas and snapshot bindings. This ensures that schema versions are tracked consistently and that reads always operate against a known-good schema snapshot. It decouples schema evolution from load and scheduling concerns, improving reliability.

Readiness Protocols with Scoped Loading and Progress Fences

Readiness checks now use scoped loading and progress fences to track component states precisely. This mechanism helps coordinate startup and recovery sequences without conflating readiness with other logic.

New Interfaces to Decouple Dependencies

To glue these components together while maintaining loose coupling, the PR introduces interfaces like LoadInfoCatalog and PrepareDataViewSchema. These interfaces abstract data access and preparation logic, allowing components to interact via contracts rather than concrete implementations.

Maintaining Legacy Support While Removing Unreleased Formats

The change preserves legacy recovery paths to ensure backward compatibility with existing deployments. At the same time, it removes support for unreleased or experimental formats, reducing maintenance overhead and potential confusion.

Lessons Learned: The Power of Clear Boundaries

This refactoring highlights the importance of splitting complex coordination logic into immutable, well-defined components. Clear interfaces enable teams to reason about each part independently, facilitate testing, and reduce the risk of unintended side effects.

For engineers working on distributed systems or large coordination modules, this approach encourages incremental integration of new features without disrupting existing workflows. It also paves the way for future optimizations, such as more advanced scheduling algorithms or schema versioning strategies.

Conclusion

The Milvus team's extraction of QueryView coordinator components exemplifies how thoughtful modularization can tame complexity in distributed systems. By embracing immutability, cost-based scheduling, and interface-driven design, they improved maintainability and extensibility — a valuable blueprint for similar challenges elsewhere.

Credit to the original PR in the milvus-io/milvus repository. Read the full breakdown on nossletter.tech.

Originally published on Nossletter.

Optimizing JSON Column Storage in ClickHouse with SHARED Typed Paths

Priyadarshan Singh — Mon, 26 Oct 2026 14:51:04 +0000

Why JSON Storage Efficiency Matters in ClickHouse

ClickHouse is a high-performance analytical database designed for fast queries on large datasets. JSON columns, which can be deeply nested and have many paths, are increasingly common as applications ingest semi-structured data. However, storing each JSON typed path as separate files quickly becomes a bottleneck, especially on object storage systems where thousands of small files cause high request overhead and degrade performance.

This article explores a recent improvement in ClickHouse's JSON column storage model that introduces a SHARED modifier for typed paths. This change significantly reduces the number of files and object storage requests while preserving efficient read access.

The Problem: File Explosion with JSON Typed Paths

Prior to this change, each typed path in a JSON column was stored as its own set of files within MergeTree parts. For example, a JSON column with hundreds of typed paths would generate thousands of files per data part. This file explosion leads to several issues:

High object storage request overhead: Object stores like S3 charge per request, so thousands of small files cause increased costs and latency.
Performance degradation: Managing and opening many small files slows down query execution.
Operational complexity: Large numbers of files complicate maintenance and increase metadata management overhead.

The root cause is that the storage model treats each typed path as an independent data stream, resulting in many small files instead of consolidated storage.

The Solution: Introducing SHARED Typed Paths

To address this, the PR adds a SHARED modifier for JSON typed paths. Instead of creating separate files for each path, all SHARED paths within a data part are grouped into a constant set of shared files.

How It Works

Shared files per part: All SHARED typed paths write their data into the same set of files rather than separate files per path.
Marks file for indexing: A marks file stores offsets within granules that index the data for each path. This allows the system to locate the data for a particular path efficiently.
Efficient partial reads: Because the marks file provides offsets, queries can read only the relevant portions of the shared files, preserving fine-grained access.
Compatibility: The approach maintains compatibility with existing features such as ALTER operations, ensuring seamless integration.

Technical Details

The marks file acts as a directory mapping each path's data offsets within the shared files. When a query requests data for a specific path, ClickHouse uses the marks to seek to the correct position in the shared files and reads only the necessary bytes. This avoids loading all paths' data and keeps query performance high.

This design cleverly balances the trade-off between file count reduction and read efficiency. By consolidating many small data streams into shared files indexed by offsets, it reduces overhead without sacrificing the ability to perform selective reads.

Takeaway: Consolidate Small Data Streams to Improve Storage Efficiency

This improvement highlights an important engineering principle when dealing with many small data substreams: consolidating storage into shared files indexed by offsets can drastically reduce file system or object storage overhead.

Especially in cloud environments where object storage request costs and latencies matter, reducing file counts minimizes operational costs and improves performance. At the same time, maintaining an index of offsets preserves the ability to read data selectively and efficiently.

This balance is crucial for production-grade systems handling complex data types like JSON, which can easily generate many small data streams.

Conclusion

The addition of SHARED typed paths in ClickHouse's JSON type storage model is a practical and elegant solution to the problem of excessive file counts for wide JSON columns. By grouping paths into shared files and indexing offsets with a marks file, ClickHouse reduces object storage overhead and improves query performance without losing fine-grained read capabilities.

This change exemplifies how thoughtful storage design can address real-world scalability challenges in analytical databases.

Credit: This article is based on the merged PR "Add SHARED typed paths to the JSON type" in the ClickHouse repository.

Read the full breakdown on nossletter.tech.

Originally published on Nossletter.

How Python's ssl Module Gained Support for Encrypted Client Hello (ECH) to Enhance TLS Privacy

Priyadarshan Singh — Sun, 25 Oct 2026 08:40:22 +0000

Why Encrypted Client Hello Matters for TLS Privacy

Transport Layer Security (TLS) is the backbone of secure communication on the internet. However, until recently, a critical piece of the TLS handshake—the ClientHello message—was sent in plaintext, exposing metadata like the server name and client preferences to network observers. Encrypted Client Hello (ECH) is an emerging TLS extension designed to encrypt this initial handshake message, significantly improving user privacy and preventing passive surveillance or censorship.

Python's standard library ssl module is widely used for secure communications in countless applications. Yet, before this change, it lacked support for ECH, limiting developers who wanted to leverage this privacy enhancement in their Python programs.

This article dives into the technical challenges and implementation details behind adding ECH support to Python's ssl module, illustrating how evolving cryptographic protocols require careful integration across language boundaries.

The Problem: Missing ECH Support in Python's ssl Module

The TLS ClientHello message initiates the handshake by communicating client capabilities and the desired server name. Traditionally, this message is unencrypted, allowing observers to see which sites a client is connecting to, even if the rest of the session is encrypted.

Encrypted Client Hello (ECH) aims to fix this by encrypting the entire ClientHello message, hiding sensitive metadata from on-path adversaries. This is especially important in environments where network censorship or traffic analysis is a concern.

Despite ECH's growing adoption and draft standardization, Python's ssl module and its underlying C extension _ssl did not support ECH. This meant Python applications could not benefit from this privacy feature, putting them at a disadvantage compared to other TLS implementations that had begun adopting ECH.

The root cause was that the _ssl C extension lacked the necessary hooks and logic to construct, send, and process the ECH extension data during the TLS handshake. Additionally, the Python-level ssl.py interface did not expose any means to configure or enable ECH.

The Fix: Extending _ssl and Python APIs to Handle ECH

Adding ECH support required coordinated changes at multiple layers:

1. Enhancing the _ssl C Extension

The _ssl module, implemented in C, interfaces directly with OpenSSL to perform TLS operations. To support ECH, it needed to:

Accept ECH configuration data (e.g., public keys and parameters) from Python code.
Integrate with OpenSSL's APIs for constructing and encrypting the ClientHello message with ECH extensions.
Handle server-side decryption of the ECH payload when acting as a TLS server.

This involved adding new data structures and logic to parse and include ECH extension bytes in the handshake messages.

2. Updating the Python ssl.py Interface

On the Python side, the ssl module was extended to expose new APIs allowing developers to specify ECH configurations. This includes:

Methods to load or specify ECH keys and configurations.
Options to enable or disable ECH for client connections.

By exposing these options in pure Python, developers can now opt into ECH without dealing with low-level C code.

3. Adding Tests and Documentation

Supporting a complex and evolving standard like ECH requires robust testing. The PR introduced comprehensive tests that:

Verify correct construction and encryption of ClientHello messages.
Ensure server-side decryption works as expected.
Test interoperability with different ECH configurations.

Additionally, documentation was updated to guide users on how to use the new ECH features.

Handling the Evolving Draft Specification

ECH is still evolving, and the implementation tracks the latest draft standards. This means the code is designed to be adaptable, with clear pathways to update cryptographic parameters and message formats as the standard matures.

Lessons Learned: Integrating Emerging TLS Features is a Cross-Layer Effort

Adding support for new TLS extensions like ECH is far from trivial. It requires:

Deep integration in the underlying TLS library bindings (_ssl in Python).
Thoughtful API design to expose new features in a user-friendly way.
Rigorous testing to ensure security and correctness.

This change exemplifies how evolving cryptographic standards necessitate collaboration between C-level system code and high-level language interfaces.

For Python developers, this means more privacy-preserving TLS connections out of the box. For the broader community, it highlights the importance of keeping standard libraries up to date with security innovations.

This work was originally contributed in the PR titled "gh-89730: Implement support for Encrypted Client Hello (ECH) in ssl" in the python/cpython repository.

Read the full breakdown on nossletter.tech.

Originally published on Nossletter.

Enhancing Distributed Streaming in Milvus: Remote TransformLog Subscriptions and MVCC Barriers

Priyadarshan Singh — Fri, 23 Oct 2026 17:23:24 +0000

Enhancing Distributed Streaming in Milvus: Remote TransformLog Subscriptions and MVCC Barriers

Distributed systems that rely on streaming replication face a delicate balance between efficiency, consistency, and fault tolerance. Milvus, a popular vector database, recently tackled a subtle but critical issue in its streaming replication mechanism. By introducing remote TransformLog subscriptions with ordered, resumable delivery and multi-version concurrency control (MVCC) barriers, Milvus now supports more reliable recovery and consistent query progress tracking.

In this article, we'll explore the problem Milvus faced, the technical mechanisms introduced in the fix, and the broader lessons for anyone implementing distributed streaming with strong consistency guarantees.

The Problem: Inefficient Recovery and Inconsistent Streaming Progress

Before this change, Milvus’s streaming replication lacked support for remote TransformLog subscriptions that could deliver entries in order and resume from interruptions. This gap led to two main issues:

Inefficient recovery: Without resumable cursors and ordered delivery, replaying logs during recovery was unreliable and potentially redundant.
Inconsistent query progress tracking: The system did not preserve MVCC barriers properly, which are essential for snapshot isolation during queries. This caused checkpoints to advance prematurely and queries to see inconsistent or stale data.

Additionally, the backpressure mechanism was simplistic — it relied on a scanner mode rather than actual cache occupancy, which failed to regulate load effectively and maintain stable delivery rates.

These issues combined to create a streaming replication system that was fragile under failure conditions and complicated to maintain strong consistency guarantees.

The Fix: Remote Subscriptions, MVCC Barriers, and Cache-Aware Backpressure

Remote TransformLog Subscriptions with Resumable Cursors

The core innovation was introducing a new SubscribeTransform RPC mechanism that supports both local and remote owner routing. This subscription model includes a handshake protocol to establish and close subscriptions gracefully. Key features include:

Ordered Entry and SyncUp delivery: Ensuring that TransformLog entries arrive in the correct sequence, which is crucial for consistent replay and state updates.
Resumable cursors: If a connection drops or is interrupted, the subscription can resume from the last known position, avoiding redundant processing and speeding up recovery.
Backoff and reconnect strategies: Failed subscription attempts back off exponentially, and reconnects are suspended when there are no active subscriptions, conserving resources.
Idle stream reclamation: Streams that are no longer in use are cleaned up promptly, preventing resource leaks.

Aligning Recovery with WAL Consumer

Recovery logic was aligned with the Write-Ahead Log (WAL) Consumer by filtering replay events before they reach data-only handlers. This ensures that replayed events are consistent and that recovery proceeds in a controlled manner.

MVCC Barrier Coordination

MVCC barriers are preserved and scoped progress is published in WAL order without advancing durable checkpoints prematurely. This means:

Volatile barriers are queued until RecoveryStorage confirms prefix completeness: This prevents queries from seeing inconsistent snapshots.
Durable checkpoints advance only after safe conditions: Avoiding premature checkpoint advancement maintains snapshot isolation and query correctness.

Cache-Aware Backpressure

The previous backpressure mechanism based on scanner mode was replaced with a more sophisticated cache-occupancy-based admission control:

WAB cache uses capacity and TTL eviction: This ensures that the cache does not overflow and that stale entries are evicted timely.
RecoveryStorage consumption constraints: These constraints help regulate admission based on actual resource usage.
Configurable slowdown and rejection thresholds: When the cache occupancy crosses certain thresholds, the system slows down or rejects new entries to maintain stability.

Additional Improvements

WALSummary reads reuse verified chunk footers: This optimization prevents repopulation of retired entries, improving efficiency.
Support for independent reader cancellation and GC snapshot protection: This enhances the robustness of concurrent reads and garbage collection.

Takeaway: Combining Resumable Cursors, MVCC Barriers, and Backpressure for Strong Consistency

Implementing distributed streaming subscriptions with strong consistency is challenging. Milvus’s approach highlights several key principles:

Resumable cursors and handshake protocols are essential for robust, ordered, and fault-tolerant delivery.
MVCC barriers must be carefully coordinated to preserve snapshot isolation and prevent premature checkpoint advancement.
Backpressure should be cache-aware and resource-sensitive, not simplistic or mode-based, to maintain stability under load.

Together, these mechanisms enable efficient recovery, consistent query progress tracking, and reliable streaming replication.

For engineers building distributed streaming systems, this case study underscores the importance of integrating consistency controls with resource management and connection resilience.

This article is based on the merged PR in the milvus-io/milvus repository titled "feat: add remote TransformLog subscriptions and MVCC barriers." Read the full breakdown on nossletter.tech.

Originally published on Nossletter.

How Milvus Improved Distributed Resource Loading with Streaming and Centralized Management

Priyadarshan Singh — Thu, 22 Oct 2026 13:34:36 +0000

Why Resource Loading Matters in Distributed Query Systems

In distributed systems like Milvus, efficient and reliable resource management is critical. Analyzer files—key resources used during query processing—must be fetched, validated, and kept consistent across multiple components. Failures or stale data in this process can cause query errors, inconsistent results, or even system instability. This article explores a recent Milvus improvement that overhauled how analyzer file resources are prepared, moving from legacy RPCs and local caches prone to transient failures, to a robust streaming client with centralized metadata management.

The Problem with Legacy Resource Loading

Previously, Milvus fetched analyzer file resources using older RPC mechanisms and local caching strategies. These legacy approaches suffered from several issues:

No batch fetching: Resources were requested individually, increasing overhead and latency.
Lack of centralized metadata: Without a single source of truth, metadata could become inconsistent or stale.
Poor concurrency coordination: Multiple components might redundantly download the same resource without coordination.
Inadequate retry and expiration handling: Transient IO errors often caused failures, and caches lacked time-to-live (TTL) policies, leading to stale data.

These limitations led to validation failures in the query coordinator and runners, inconsistent resource states, and unreliable query execution.

The New Streaming Client and AnalyzerResourceManager

The core of the fix is introducing a streaming client that supports a new batch FetchFileResources RPC. This API can fetch multiple resource metadata and content items by their IDs or names in a single request, dramatically improving efficiency.

Centralized Metadata Management

The AnalyzerResourceManager acts as the central authority for managing resource metadata. It:

Fetches metadata in batches via the streaming client.
Coordinates concurrent download requests to avoid duplicate work.
Implements shared cancellation so that if one component cancels, others depending on the same resource are notified.

Robust Download and Caching

Downloads are retried locally on transient failures, reducing the chance of query errors due to momentary network glitches. Files are cached with TTL-based expiration, ensuring that stale data is refreshed proactively.

Validation Integration

Validation logic is routed through the streaming client, which verifies resource integrity against the fetched metadata. This ensures that corrupted or incomplete resources are detected early, with precise status errors returned to the caller.

Runner Lifecycle Improvements

Runners—components that execute queries—now pin exact resource bindings and include schema IDs in their identity. This guarantees that the runner's resource environment is stable and consistent throughout its lifecycle.

Additionally, if a runner is unavailable, it can be prepared on demand using retry logic after releasing its admission scope. This ensures that resources are always available during write-ahead log (WAL) replay and query execution.

Lessons Learned

This change highlights several important principles for distributed resource management:

Batching and streaming reduce overhead: Fetching multiple resources in a single streaming RPC is more efficient than many individual requests.
Centralized metadata prevents inconsistency: A single source of truth for resource metadata avoids stale or conflicting information.
Coordinated concurrency avoids redundant work: Managing concurrent downloads and cancellations reduces wasted network and computation resources.
Explicit lifecycle management improves stability: Pinning resources to runner lifecycles and including schema IDs ensures consistent execution environments.
Robust retry and caching policies reduce transient failures: Handling IO errors locally and using TTL caches minimizes query disruptions.

Together, these improvements make Milvus's distributed query execution more reliable and performant.

This article is based on the PR "fix: prepare analyzer file resources through streaming client" merged into the milvus-io/milvus repository. Read the full breakdown on nossletter.tech.

Originally published on Nossletter.

Building a Core Storage Library for Primary Key Indexing in Milvus

Priyadarshan Singh — Wed, 21 Oct 2026 13:54:07 +0000

Why a Core Storage Library Matters for Primary Key Indexing

When working with large-scale vector databases like Milvus, efficient and consistent primary key indexing is crucial. Primary keys serve as the backbone for data retrieval, and mishandling their storage can lead to fragmented logic, duplicated code, and performance bottlenecks. Before this enhancement, Milvus lacked a unified core storage library for primary key indexes, resulting in scattered implementations and maintenance challenges.

This article dives into how introducing a dedicated core storage library for primary key indexes addresses these issues, streamlines storage lifecycle management, and sets the stage for incremental improvements without disrupting existing workflows.

The Problem: Fragmented Storage Logic for Primary Keys

In Milvus’s architecture, primary key indexes are essential for locating vector data efficiently. However, prior to this enhancement, there was no shared, core storage library dedicated to managing these indexes. Instead, primary key handling was embedded in various nodes or test-only memory engines.

This fragmentation led to several issues:

Duplicated Storage Logic: Different components implemented their own versions of primary key storage and retrieval, increasing maintenance overhead.
Inconsistent Behavior: Without a unified approach, lifecycle management of storage layers, such as LSM-tree generations, was handled inconsistently.
Limited Efficiency: Embedding logic in nodes or memory engines limited the ability to optimize write paths and manage storage transitions smoothly.

The root cause was clear: the absence of a dedicated, shared library meant no single source of truth or standardized mechanism for primary key storage.

The Solution: Introducing `internal/pkindex` Core Library

To tackle these challenges, the PR introduces a new core storage library named internal/pkindex. This library encapsulates all primary key index storage logic, providing a consistent, efficient, and reusable foundation.

Key Components of the Library

Order-Preserving Encoding of Primary Keys

The library implements an encoding scheme that preserves the order of primary keys. This is critical for range queries and efficient lookups, ensuring that the storage engine can maintain sorted keys without additional overhead.

SSTable Read/Write via Pebble

The library leverages Pebble (version 1.1.5), a performant key-value store inspired by LevelDB, to manage SSTables (Sorted String Tables). By integrating Pebble, the library handles persistent storage on local files, enabling robust and efficient disk-backed storage.

Engine Managing Write Paths and Lifecycle Transitions

One of the standout features is an engine within the library that manages the write path and lifecycle transitions of LSM (Log-Structured Merge) tree generations. Crucially, it does so without introducing write pauses, enabling smooth compaction and generation transitions.

Error Categorization and Backward Compatibility

The library categorizes errors to facilitate better error handling and debugging. It also maintains backward-compatible error message formatting, ensuring existing components that rely on previous formats continue to function without disruption.

How It Works in Practice

Imagine a policy that needs to write primary key data as part of indexing. Instead of each node or engine implementing its own storage logic, it now calls into internal/pkindex. The library encodes the keys preserving their order, writes them to SSTables using Pebble, and manages the underlying lifecycle transitions automatically. This separation allows for incremental adoption; components can start using the library without needing to overhaul their entire storage logic.

Lessons Learned: The Power of Separation and Incremental Adoption

This enhancement underscores the importance of isolating core storage logic into a dedicated library. By doing so, Milvus achieves:

Consistency: Uniform handling of primary key storage and lifecycle management across components.
Maintainability: Reduced code duplication and clearer separation of concerns.
Performance: Efficient write paths and seamless lifecycle transitions without write pauses.
Flexibility: The ability to incrementally adopt the new library across the system without breaking existing behavior.

In complex distributed systems, especially those dealing with storage and indexing, having a well-defined core library not only simplifies development but also lays a strong foundation for future improvements.

This article is based on the recent pull request merged into the Milvus repository. Read the full breakdown on nossletter.tech.

Originally published on Nossletter.

Enhancing Node.js Module Privacy: Introducing --experimental-strip-private-modules

Priyadarshan Singh — Wed, 21 Oct 2026 10:46:58 +0000

Why Module Privacy Matters in Node.js

In the Node.js ecosystem, managing dependencies often means dealing with nested node_modules directories, where public and private packages coexist. Ensuring that private packages are handled correctly is crucial, especially when it comes to stripping types or other metadata during builds. Without careful controls, private package information can leak or public packages can be inadvertently altered, leading to subtle bugs or security concerns.

This article explores a recent improvement in Node.js that introduces a new CLI flag to better manage private packages during type stripping, preventing unintended side effects and privacy leaks.

The Problem: Overbroad Type Stripping and Privacy Smuggling

Previously, Node.js applied type stripping broadly to all packages inside node_modules. This approach had two main issues:

Unintended Removal of Types from Public Packages: Since type stripping wasn’t selective, it could remove type information even from public packages that should retain their typings.
Privacy Smuggling via Nested package.json Files: Private package metadata could be hidden or "smuggled" inside nested node_modules directories of non-private packages, bypassing intended privacy boundaries.

The root cause was the lack of granularity in the stripping logic. The system did not distinguish between private and public packages effectively, nor did it enforce strict checks on package privacy metadata in nested directories. Internal modules like package_json_reader.js and node_modules.cc needed more precise logic to identify private packages correctly.

The Fix: Introducing --experimental-strip-private-modules

To address these issues, the Node.js team introduced a new CLI flag: --experimental-strip-private-modules. This flag changes the behavior of type stripping in several key ways:

Selective Type Stripping: When enabled, type stripping applies only to packages explicitly marked as private in their package.json files. This prevents public packages from losing their type information inadvertently.
Enhanced Privacy Detection: Updates to package_json_reader.js and node_modules.cc improve the detection of private packages. The logic now carefully parses the package.json metadata, ensuring that private flags are respected even in deeply nested node_modules folders.
Improved TypeScript Handling: The internal module internal/modules/typescript.js was updated to align with the new privacy-aware stripping logic, maintaining consistency in how TypeScript types are processed.
Comprehensive Testing: New tests in test/es-module/test-typescript.mjs validate that type stripping behaves as expected under the new flag, covering scenarios involving nested packages and privacy boundaries.

How It Works Internally

The enhanced privacy check involves reading the package.json for each package encountered during module resolution. If the private field is set to true, and the CLI flag is enabled, the type stripping logic activates for that package.

For example, a policy like this might check:

if (packageJson.private === true && cliFlags.experimentalStripPrivateModules) {
  stripTypes(packagePath);
} else {
  preserveTypes(packagePath);
}

This targeted approach ensures that only intended packages are affected.

Additionally, by tightening the logic in package_json_reader.js, the system prevents scenarios where a private package's package.json could be nested inside a public package without proper recognition, closing a loophole that previously allowed privacy smuggling.

Takeaway: Enforce Explicit Boundaries and Use Targeted Flags

This change highlights an important lesson in package management and build tooling:

Avoid Broad Application of Transformations: Applying transformations globally without considering package metadata can lead to unintended consequences.
Explicitly Enforce Privacy Boundaries: Tools should respect package privacy metadata, especially in complex nested dependency trees.
Use Feature Flags for Experimental Behavior: Introducing a CLI flag allows developers to opt-in to new behavior, reducing risk and enabling gradual adoption.

By carefully respecting package privacy and controlling transformation scope, Node.js improves module safety and developer experience.

This article is based on the changes introduced in the Node.js repository PR titled "module: add --experimental-strip-private-modules".

Read the full breakdown on nossletter.tech.

Originally published on Nossletter.

Seamless Model Discovery and Download from HuggingFace Hub in llama.cpp UI

Priyadarshan Singh — Mon, 19 Oct 2026 05:58:00 +0000

Seamless Model Discovery and Download from HuggingFace Hub in llama.cpp UI

Integrating external model repositories directly into machine learning tooling can drastically improve user experience and workflow efficiency. For users of the popular llama.cpp project, managing GGUF models—especially quantized ones with associated sidecar files—was previously a manual, error-prone process. This article dives into a recent enhancement that brings HuggingFace Hub model discovery and synchronized downloads into the llama.cpp UI, making model management intuitive and reliable.

The Problem: Manual and Fragmented Model Downloads

Before this change, llama.cpp users who wanted to leverage GGUF format models hosted on HuggingFace Hub faced several challenges. The UI lacked any way to browse or search the Hub’s model repository, forcing users to manually locate and download models outside the application.

More critically, quantized models often come with additional "draft" sidecar files—like dflash or mtp—that optimize or complement the main model. Users had to manually track and download these sidecars separately, with no visual indication or download controls in the UI. This led to inconsistent states, missed files, and a cumbersome workflow.

At the root, the existing model manager was not integrated with HuggingFace Hub’s APIs and did not model the relationship between main models and their sidecars. Download logic was isolated and unaware of these dependencies, making coordinated downloads and resumable controls impossible.

The Solution: Discover Models View and Coordinated Downloads

This PR introduces a comprehensive Discover Models view within the llama.cpp UI that directly queries the HuggingFace Hub for GGUF models. Users can now search, filter, and view detailed metadata about available models without leaving the app.

UI Enhancements

Discover Models View: Presents a live list of GGUF models from HuggingFace Hub with search and filtering capabilities.
Download Chips: Each model entry features interactive download buttons that reflect real-time download states (e.g., queued, in progress, paused).
Draft Sub-lines: When a model has associated draft sidecar files, these appear as sub-items under the main model entry, clearly indicating their relationship.
Settings Toggles: Users can enable or disable the Discover integration and filter out unsupported decision models right from the settings.

Backend Download Logic

The download subsystem, particularly in common/download.cpp, was extended to recognize and handle tagged sidecar files linked to a main quantized model. This means:

When a user initiates a download of a quantized model, the system automatically identifies and queues downloads for its draft sidecars.
Download controls such as pause, resume, and cancel apply synchronously to both main and sidecar files, ensuring consistent states.
Manual downloads of drafts are also supported, giving users flexibility.

By explicitly modeling these file sets as coordinated groups, the download manager maintains robust state tracking and error handling.

Why This Matters: Lessons for Integrating Complex External Resources

This enhancement highlights key considerations when integrating external repositories with complex dependency relationships into an application UI:

Model Related Files as Groups: Treat main models and their auxiliary files as a single logical unit in both backend and frontend to maintain consistency.
Synchronized Controls: Download operations and UI states must be coordinated across all related files to avoid user confusion and partial states.
User-Centric Discovery: Embedding search and filtering capabilities directly into the UI reduces friction and encourages adoption.
Configurability: Providing users toggles to enable/disable integrations or filter content respects diverse workflows and resource constraints.

For projects dealing with large, multipart downloads or external resource catalogs, these principles can greatly improve usability and reliability.

This article is based on the PR "ui : Discover & Download Models from HF Hub" merged into the llama.cpp repository.

Read the full breakdown on nossletter.tech.

Originally published on Nossletter.

Extending llama.cpp to Support Reflection Beam-1.0-501B-A23B: Handling Complex MoE and Attention Architectures

Priyadarshan Singh — Sat, 17 Oct 2026 13:30:31 +0000

Why Supporting Specialized Large Models Matters

The rapid evolution of large language models (LLMs) has introduced increasingly complex architectures, often combining mixture-of-experts (MoE) routing, novel attention mechanisms, and specialized input/output formats. For open-source projects like llama.cpp, which aim to provide lightweight, efficient inference, adding support for these models is a non-trivial challenge. This article dives into how the Reflection Beam-1.0-501B-A23B model, with its intricate MoE and attention layers, was integrated into llama.cpp — revealing key engineering lessons for anyone extending LLM frameworks to handle cutting-edge architectures.

The Challenge: Why Reflection Beam Wasn't Supported Initially

Before this change, llama.cpp did not support the Reflection Beam-1.0-501B-A23B model. This model stands out for several reasons:

Mixture-of-experts (MoE) with top-8 expert selection: Unlike simpler MoE implementations, this model uses a specialized routing mechanism that selects the top 8 experts per token, combined with sigmoid-based routing biases.
Interleaved attention layers: It alternates sliding-window attention with NoPE (no positional encoding) attention layers, applying rotary positional embeddings (RoPE) selectively.
Power-law query scaling: An optional feature that scales query vectors in attention layers with a power-law function to improve performance.
New chat parser format: The model uses a unique tool-call chat format requiring a dedicated parser.

These architectural and format differences meant that simply loading weights or running inference with existing llama.cpp code was impossible.

How the PR Tackled the Complexity

The solution involved multiple coordinated enhancements across the codebase:

1. New Model Implementation (src/models/beam.cpp)

A dedicated model file was created to encapsulate the unique MoE and attention logic:

MoE FFN with grouped expert selection: The code implements a top-8 expert selection per token, using a sigmoid routing mechanism that biases the expert scores. This differs from simpler MoE layers by requiring grouped computation and careful routing to efficiently select and combine expert outputs.
Interleaved attention layers: The model alternates between sliding-window attention (which focuses on local context windows) and NoPE attention (which omits positional embeddings). The implementation ensures that RoPE is applied only where appropriate, preserving the intended positional information.
Power-law query scaling: This feature is implemented as an optional step in llama-graph.cpp, where query vectors in attention layers are scaled by a power-law function before computing attention scores. This subtle modification can improve the model's ability to focus on relevant tokens.

2. Converter Script (conversion/beam.py)

To facilitate loading the Beam model weights into llama.cpp, a new converter script was added. This script translates the official Beam model weights into the GGUF format compatible with llama.cpp, including handling the MoE-specific parameters and attention layer weights.

3. Harmony-XML Chat Parser (common/parsers/harmony-xml.cpp)

The Beam model uses a specialized chat format that includes tool-call markers. To support this, a new parser was introduced that detects these markers and correctly interprets the chat messages, enabling seamless interaction with the model's tool-calling capabilities.

4. Integration and Testing

The new components were wired into the existing llama.cpp architecture with appropriate tests to ensure correctness. GGUF format support was extended to accommodate the Beam-specific model parameters, guaranteeing that loading and generation behave as expected.

Lessons Learned: Modularity and Format Awareness Are Key

This integration highlights several important takeaways for LLM framework maintainers and contributors:

Modular architecture enables extensibility: By isolating the Beam model logic in its own source file and adding specialized parsers and converters, the codebase remains maintainable and adaptable to future models with unique architectures.
Attention to routing and attention details is critical: Complex MoE routing and interleaved attention mechanisms require careful implementation to preserve model semantics and performance.
Input/output format support matters: Supporting new chat formats and tool-call protocols is as crucial as supporting the model architecture itself, especially for interactive or multi-modal models.
Testing and format compatibility: Adding tests and ensuring compatibility with standard formats like GGUF helps maintain robustness and user confidence.

Conclusion

Supporting a model as specialized as Reflection Beam-1.0-501B-A23B in llama.cpp required extending the model graph logic, adding new routing and attention mechanisms, and implementing format-aware parsers. This work not only enables users to run this advanced model efficiently but also sets a foundation for integrating future complex architectures. For developers working on LLM frameworks, this case underscores the importance of modular design and comprehensive format support when adapting to the evolving landscape of large models.

This article is based on the merged PR "model: add Beam (Reflection)" in the ggml-org/llama.cpp repository. Read the full breakdown on nossletter.tech.

Originally published on Nossletter.

How Hugging Face's Transformers Added UltraBERT with Layer-Specific Rotary Embeddings

Priyadarshan Singh — Fri, 16 Oct 2026 16:29:53 +0000

Why UltraBERT Integration Matters

The Hugging Face Transformers library is a cornerstone for NLP research and applications, providing a unified interface for countless transformer architectures. But as models evolve, they often introduce nuanced architectural features that challenge existing abstractions. UltraBERT is one such model, distinguished by its dual rotary embedding scheme tailored for local and global attention layers. Integrating UltraBERT into Transformers not only unlocks new research possibilities but also highlights best practices for managing complex, layer-specific parameters in large model codebases.

The Problem: Handling Distinct Rotary Embeddings Per Layer

Rotary embeddings are a powerful technique for injecting relative position information into transformer models. Most models use a single rotary embedding parameter set applied uniformly across layers. UltraBERT, however, innovates by employing two different rotary embedding theta values: one for local attention windows and another for global attention. This dual setup requires the model to maintain and apply separate rotary embeddings depending on the layer's attention type.

Before this PR, the Transformers library did not support multiple rotary embeddings coexisting and being selectively applied per layer. The existing code assumed a single rotary embedding instance, making UltraBERT's unique requirement incompatible with the library's design.

The Solution: Modular Rotary Embeddings with ModuleDict

To address this, the PR introduced UltraBERT as a new model class within Transformers, building on the modular design of existing models like ModernBERT and EuroBERT. This reuse minimized code duplication and leveraged proven architectures.

The key technical innovation was the use of a ModuleDict to store rotary embeddings keyed by their type—specifically, separate entries for "local" and "global" rotary embeddings. This design allows the model to initialize, store, and access multiple rotary embedding parameters cleanly and efficiently.

In modeling_ultrabert.py, the ModuleDict is set up during model initialization to contain two rotary embedding modules, each initialized with the appropriate theta values for local and global attention. During the forward pass, UltraBERT applies the correct rotary embedding by selecting from the ModuleDict based on the layer's attention configuration.

This approach avoids duplicating entire model implementations for each rotary embedding variant. Instead, it provides a flexible container that elegantly manages layer-specific parameter variations.

The PR also integrated UltraBERT's configuration, pretrained model weights, and added support for common heads like Masked Language Modeling and Sequence Classification, making the model fully functional within the Transformers ecosystem.

Key Takeaway: Embrace Modular Containers for Layer-Specific Parameters

UltraBERT's integration underscores a broader engineering lesson: when dealing with models that require distinct parameters per layer or per functional unit, designing modular and flexible parameter containers is crucial. Structures like PyTorch's ModuleDict enable clean separation and management of multiple parameter sets without bloating the codebase or introducing fragile conditionals.

This pattern not only improves maintainability but also facilitates future extensibility. New variants with additional parameter groups or attention mechanisms can be integrated by extending the container rather than rewriting core model logic.

For practitioners extending large model libraries, this approach balances code reuse with the flexibility needed to accommodate cutting-edge research innovations.

Credit to the Hugging Face Transformers team for the original UltraBERT integration PR.

Read the full breakdown on nossletter.tech.

Originally published on Nossletter.

DEV Community: Priyadarshan Singh

How Cilium Cut Restart Times by Reusing Compiled BPF Objects

How Cilium Cut Restart Times by Reusing Compiled BPF Objects

The Problem: Unnecessary Recompilation on Every Restart

Root Cause: Lack of Input Hashing and Cache

The Solution: Input Hashing and Caching Compiled Objects

Takeaway: Hashing Inputs to Cache Build Artifacts Saves Time and Resources

Enhancing ggml with Fused Sparse-Attention Masks and Precision Introspection in v0.28.0

Why Fused Sparse-Attention Masks Matter for ML Performance

The Problem: Inefficient Sparse-Attention Support and Skipping Logic

The Solution: Introducing GGML_OP_TOP_K_MASK and Cleaner APIs

Fused Sparse-Attention Mask Operation

Explicit Skip Index with -1

Precision Introspection APIs

Additional Improvements

Why This Matters: Takeaways for ML Library Design

Breaking Down Milvus QueryView Coordinator for Better Modularity and Maintainability

Why Modular Design Matters in Complex Systems

The Problem: A Monolithic Coordinator

The Fix: Extracting Immutable Components and Interfaces

Immutable Load Snapshots and Placement Management

Scheduling with Cost-Based Reconciliation Using LogicalSize Weights

Schema Lifecycle Management with Certified Read Schemas

Readiness Protocols with Scoped Loading and Progress Fences

New Interfaces to Decouple Dependencies

Maintaining Legacy Support While Removing Unreleased Formats

Lessons Learned: The Power of Clear Boundaries

Conclusion

Optimizing JSON Column Storage in ClickHouse with SHARED Typed Paths

Why JSON Storage Efficiency Matters in ClickHouse

The Problem: File Explosion with JSON Typed Paths

The Solution: Introducing SHARED Typed Paths

How It Works

Technical Details

Takeaway: Consolidate Small Data Streams to Improve Storage Efficiency

Conclusion

How Python's ssl Module Gained Support for Encrypted Client Hello (ECH) to Enhance TLS Privacy

Why Encrypted Client Hello Matters for TLS Privacy

The Problem: Missing ECH Support in Python's ssl Module

The Fix: Extending _ssl and Python APIs to Handle ECH

1. Enhancing the _ssl C Extension

2. Updating the Python ssl.py Interface

3. Adding Tests and Documentation

Handling the Evolving Draft Specification

Lessons Learned: Integrating Emerging TLS Features is a Cross-Layer Effort

Enhancing Distributed Streaming in Milvus: Remote TransformLog Subscriptions and MVCC Barriers

Enhancing Distributed Streaming in Milvus: Remote TransformLog Subscriptions and MVCC Barriers

The Problem: Inefficient Recovery and Inconsistent Streaming Progress

The Fix: Remote Subscriptions, MVCC Barriers, and Cache-Aware Backpressure

Remote TransformLog Subscriptions with Resumable Cursors

Aligning Recovery with WAL Consumer

MVCC Barrier Coordination

Cache-Aware Backpressure

Additional Improvements

Takeaway: Combining Resumable Cursors, MVCC Barriers, and Backpressure for Strong Consistency

How Milvus Improved Distributed Resource Loading with Streaming and Centralized Management

Why Resource Loading Matters in Distributed Query Systems

The Problem with Legacy Resource Loading

The New Streaming Client and AnalyzerResourceManager

Centralized Metadata Management

Robust Download and Caching

Validation Integration

Runner Lifecycle Improvements

Lessons Learned

Building a Core Storage Library for Primary Key Indexing in Milvus

Why a Core Storage Library Matters for Primary Key Indexing

The Problem: Fragmented Storage Logic for Primary Keys

The Solution: Introducing internal/pkindex Core Library

Key Components of the Library

How It Works in Practice

Lessons Learned: The Power of Separation and Incremental Adoption

Enhancing Node.js Module Privacy: Introducing --experimental-strip-private-modules

Why Module Privacy Matters in Node.js

The Problem: Overbroad Type Stripping and Privacy Smuggling

The Fix: Introducing --experimental-strip-private-modules

How It Works Internally

Takeaway: Enforce Explicit Boundaries and Use Targeted Flags

Seamless Model Discovery and Download from HuggingFace Hub in llama.cpp UI

Seamless Model Discovery and Download from HuggingFace Hub in llama.cpp UI

The Problem: Manual and Fragmented Model Downloads

The Solution: Introducing `internal/pkindex` Core Library