DEV Community: ZhengZhiCong

VictoriaMetrics Stream Aggregation: A Three-Year Retrospective (2026)

ZhengZhiCong — Sun, 07 Jun 2026 22:56:28 +0000

It's been exactly three years since the first article in this series was published in March 2023. The VictoriaMetrics ecosystem has changed dramatically since then. Let's revisit the problems we laid out, see what the official project has resolved, and assess where our custom stream-metrics-route gateway stands today.

I. The Problems We Identified in 2023

Here's a quick recap of the core issues from the original post:

#	Problem	2023 Status
P1	Collection gap inflation	Network jitter or performance issues cause time gaps that inflate stream aggregation deltas
P2	Single-node compute limits	Stream aggregation has no historical state, fast but single-instance bottlenecked
P3	Distributed task allocation	Which compute node should each sample be assigned to?
P4	Out-of-order discarding for same-dimension metrics	Multiple nodes computing the same dimension with different time windows causes later values to be discarded
P5	Resource balancing	Uneven load across distributed compute nodes
P6	Task ID dimension explosion	Stream aggregation inserts node IDs into aggregated time series — the label cardinality grows with every node you add

To address these, we built stream-metrics-route, a Go-based distributed stream aggregation gateway.

II. Three Years Later — What the Official Project Has Done

I reviewed VictoriaMetrics changelogs from v1.86 through v1.138.0 and the official documentation. Here's the scorecard.

✅ Perfectly Resolved

Issues P3, P5: Distributed Task Allocation & Resource Balancing

Official solution: vmagent now natively supports -remoteWrite.shardByURL with consistent hashing sharding.

Starting from v1.86, basic shardByURL was introduced. v1.138.0 (March 2026) was the real milestone — it upgraded the data distribution algorithm from round-robin to consistent hashing, which significantly reduces data redistribution ratios during node changes.

The architecture evolution looks like this:

┌─────────────────┐     ┌─────────────────┐
│ Prometheus      │     │ Prometheus      │
│ Agent 1         │     │ Agent 2         │
└────────┬────────┘     └────────┬────────┘
         │ remote write          │ remote write
         ▼                       ▼
┌─────────────────────────────────────────┐
│           vmagent Cluster               │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐│
│  │vmagent-0 │  │vmagent-1 │  │vmagent-2 ││
│  └─────┬────┘  └─────┬────┘  └─────┬────┘│
│        │             │             │     │
│        └─────────────┼─────────────┘     │
│                      ▼                   │
│           ┌──────────────────┐           │
│           │ Consistent Hash │           │
│           └────────┬─────────┘           │
└────────────────────┼─────────────────────┘
                     │ shard by hash
         ┌───────────┼───────────┐
         ▼           ▼           ▼
   ┌──────────┐ ┌──────────┐ ┌──────────┐
   │vmstorage │ │vmstorage │ │vmstorage │
   │    -0    │ │    -1    │ │    -2    │
   └──────────┘ └──────────┘ └──────────┘

The VictoriaMetrics blog provides specific algorithm recommendations. Combined with VictoriaMetrics Operator, you can manage shards via shardCount.

Issue P2: Single-Node Compute Scaling

vmagent now supports horizontal scaling natively with replicas + shardCount, including HA support — see Issue #5573.

Out-of-Order / Delayed Data Accuracy (P1 — Partial Mitigation)

v1.112.0 (February 2025) was a key release, adding Aggregation Windows — dual-window buffering for histogram and rate calculations. Instead of flushing immediately, the output is delayed by a samples_lag window, which significantly improves accuracy for late-arriving data. The tradeoff: roughly doubled memory usage (maintaining two aggregation windows simultaneously).

How it works:

Collector          vmagent           VictoriaMetrics
   │                  │                    │
   │  sample1 @T0    │                    │
   ├─────────────────►│  Write to          │
   │                  │  Window A (current)│
   │                  │                    │
   │  sample2 @T1    │                    │
   │  (delayed)      │                    │
   ├─────────────────►│  Write to          │
   │                  │  Window B (previous)│
   │                  │                    │
   │                  │  Aggr result A @T2 │
   │                  ├────────────────────►│
   │                  │  Aggr result B @T3 │
   │                  ├────────────────────►│

See the official docs on streaming aggregation windows.

❌ Still Unresolved

True Distributed Stream Aggregation Coordination

vmagent's stream aggregation is single-instance. There is no coordination mechanism between instances — if two vmagent instances aggregate the same metric, you get duplicate or conflicting output. The official recommendation is to use without/by label clauses to divide responsibility between instances, rather than providing a cross-instance coordination protocol.

Task ID Dimension Explosion (P6)

Official vmagent still inserts internal labels (such as _aggr-related labels) into aggregated time series, but lacks a stream_task_id pre-marking plus dimension control design.

III. stream-metrics-route: Current Status and Value

Core Code Architecture

File	Role
`router.go`	Routing core — filters metrics based on relabel rules
`remotecluster.go`	Dual hashmod scheduling core
`remotewrite.go`	Remote write HTTP client
`kafka.go`	Kafka producer

The Core Algorithm (from `remotecluster.go`)

// Dual hashmod scheduling
hash := sortLabelsHashKey(ts.Labels)
dime := hashMod(r.dimension, hash) // First hashmod → task partition ID

ts.Labels = append(ts.Labels, prompb.Label{
    Name:  "stream_task_id",
    Value: strconv.Itoa(dime), // Insert stream_task_id label
})

hashnode = sortLabelsHashKey(filterLabels) // Second hashmod → node selection
tmpch := hashMod(r.uplen, hashnode)       // Which backend writer to send to

Is It Still Needed in 2026?

Yes — but with an adjusted role. The positioning should shift from "full stream aggregation gateway" to "metric distribution routing gateway + Kafka integration layer." The core differentiated value:

Dual hashmod scheduling + stream_task_id pre-injection — tags metrics at the gateway layer, so all downstream nodes route consistently by this ID. This solves dimension control earlier in the pipeline than the official approach.
Multi-backend async distribution — supports async distribution to both Kafka and remote write, solving the "synchronous forwarding blocks the time window" problem from the original post.
Native Prometheus relabeling integration — works with standard Prometheus relabel configs, no custom syntax to learn.

IV. Recommended 2026 Hybrid Architecture

┌─────────────────┐   ┌──────────────────┐
│  Prometheus     │   │  Business System │
│  Agent Cluster  │   │  Metrics (Kafka) │
└────────┬────────┘   └────────┬─────────┘
         │                     │
         ▼                     ▼
    ┌──────────────────────────────┐
    │   stream-metrics-route       │
    │   (Routing Layer)            │
    │   - Dual hashmod scheduling  │
    │   - stream_task_id injection │
    │   - Relabeling               │
    └──────┬──────┬──────┬─────────┘
           │      │      │
    task=0 │ task=1│task=2│
           ▼      ▼      ▼
    ┌─────────────────────────┐
    │   vmagent Cluster       │
    │  (v1.112.0+ with        │
    │   aggregation windows)  │
    └──────────┬──────────────┘
               │
               ▼
    ┌──────────────────┐      ┌────────────┐
    │  VictoriaMetrics │      │   Kafka    │
    │  (Storage)       │      │   (Topic)  │
    └──────────┬───────┘      └────────────┘
               │
               ▼
    ┌──────────────────┐
    │  vmalert         │
    │  Grafana         │
    └──────────────────┘

Key Configuration

vmagent version requirement: >= v1.112.0, with aggregation windows enabled:

# stream aggregation config
- match: 'http_request_duration_seconds_bucket'
  interval: 5m
  without: [instance]
  enable_windows: true   # Critical! Enables dual-window buffering
  outputs: [rate_sum]

V. Evolution Recommendations

Short-term

Action	Description
Upgrade vmagent to >= v1.112.0	Enable `enable_windows: true` to improve histogram aggregation accuracy
Evaluate stream-metrics-route necessity	If you have no Kafka requirement or high-cardinality `stream_task_id` control need, consider migrating away

Medium-term

Action	Description
Retain stream-metrics-route as front-end routing only	Keep hashmod task allocation + Kafka distribution; remove aggregation responsibility
Disable raw metric persistence	Write only stream-aggregated results to storage to reduce volume
Add metadata management module	The `ruler-handle-process` from the original post (dynamic Record Rule by dimension) is worth building or contributing

Long-term

Action	Description
Contribute `stream_task_id` dimension control upstream	If the dual hashmod design proves out in production
Improve monitoring metrics	Add business-level metrics — queue depth per routing rule, distribution latency, etc.

Putting It All Together

Dimension	Assessment
Problem resolution rate	~50% — 2 of 4 core problems resolved via official upgrades; 2 still need custom solutions
Is stream-metrics-route still needed?	Yes — repositioned as "metric distribution routing gateway + Kafka integration layer"
Recommended architecture	Prometheus → stream-metrics-route → vmagent v1.112.0+ → VictoriaMetrics Storage

Three years is a long time in the observability space. The VictoriaMetrics ecosystem has matured significantly — consistent hashing, aggregation windows, and native sharding all address problems that required custom tooling in 2023. But the hard problems around true distributed stream aggregation coordination and dimension control at the gateway layer remain open.

If you're running a similar stack, the hybrid approach — letting the official project handle what it's good at (single-node aggregation, storage) while keeping custom routing for what it isn't (distributed coordination, dimension pre-injection, Kafka bridging) — has proven to be the right call for us.

Originally published on my blog

Scaling VictoriaMetrics Stream Aggregation: A Deep Dive into Distributed Design（2023）

ZhengZhiCong — Sun, 07 Jun 2026 22:50:31 +0000

VictoriaMetrics' stream aggregation is a powerful feature for reducing metric cardinality in real time. But when you push it to millions of time series across a distributed fleet, the native implementation starts to show cracks.

This post walks through what I found analyzing the stream aggregation source code, the real-world problems that emerged at scale, and the distributed gateway we built to solve them.

Community VM Stream Aggregation — Capability Analysis

Stream aggregation was integrated into vmagent starting from version 1.86 (GitHub issue #3460). Let's look at what it actually does under the hood.

Core Computation: The `pushSample` Function

The heart of stream aggregation lives in the pushSample function. Here's the simplified logic:

func (as *totalAggrState) pushSample(inputKey, outputKey string, value float64) {
    currentTime := fasttime.UnixTimestamp()
    deleteDeadline := currentTime + as.intervalSecs + (as.intervalSecs >> 1)

again:
    v, ok := as.m.Load(outputKey)
    if !ok {
        v = &totalStateValue{
            lastValues: make(map[string]*lastValueState),
        }
        vNew, loaded := as.m.LoadOrStore(outputKey, v)
        if loaded {
            v = vNew
        }
    }
    sv := v.(*totalStateValue)
    sv.mu.Lock()
    deleted := sv.deleted
    if !deleted {
        lv, ok := sv.lastValues[inputKey]
        if !ok {
            lv = &lastValueState{}
            sv.lastValues[inputKey] = lv
        }
        d := value
        if ok && lv.value <= value {
            d = value - lv.value
        }
        if ok || currentTime > as.ignoreInputDeadline {
            sv.total += d
        }
        lv.value = value
        lv.deleteDeadline = deleteDeadline
        sv.deleteDeadline = deleteDeadline
    }
    sv.mu.Unlock()
    if deleted {
        goto again
    }
}

The core idea: each incoming sample's value is compared against the last seen value for that time series. The delta is accumulated into a running total. This works well for counters that monotonically increase, but there's a critical detail — the time window logic is simple periodic checking with no sophisticated handling for delayed or out-of-order arrivals.

What Stream Aggregation Looks Like in Practice

In theory, stream aggregation should cleanly reduce high-cardinality metrics down to manageable summaries. In practice, the picture is messier:

Ideal model: every sample arrives on time, windows align perfectly, aggregation is lossless
Reality: samples arrive late, retries flood old data, gaps appear from network or service issues

Common Issues with Native Stream Aggregation

The Collection Gap Problem

Collection gaps are inevitable. Network blips, service restarts, GC pauses — any of these can cause a gap in metric collection. For high-precision stream aggregation, gaps create a specific failure mode:

When a counter's last tracked value is 1000, and a gap causes the next received value to be 5000, the delta is 4000 — which may span an unknown number of actual increments. If the gap occurred within a single aggregation window, the inflated value corrupts that window's result. If the gap crosses window boundaries, you get compounding errors in downstream calculations.

Distributed Computing Challenges

Stream aggregation doesn't persist historical data, so it's fast — but even the fastest service has single-node limits. When you need to scale horizontally, a cascade of new problems appears:

Is vmagent's built-in collection viable at scale?

In testing, enabling vmagent shard + replica collection with real-time stream aggregation caused significant resource spikes. At very large scales, collection gaps became more frequent, which amplified the calculation errors from gaps rather than mitigating them.

Which compute node should each sample be assigned to?

Without a consistent routing strategy, the same metric dimension gets split across nodes, producing partial results that can't be safely combined.

How do you handle the same dimension set being computed by multiple nodes with different values?

When multiple nodes produce results for the same dimension within the same time window, VictoriaMetrics triggers its out-of-order handling logic — which discards later values. You lose data silently.

How do you balance resources in distributed computation?

Uneven distribution means some nodes are overloaded while others sit idle.

What about the new dimensions introduced by routing?

If you insert a task ID to differentiate compute nodes, you've just added a new dimension that grows with every node you add — defeating part of the purpose of aggregation.

Design and Implementation: A Distributed Stream Aggregation Gateway

After analyzing these problems, it became clear that a frontend module was needed to address them at the entry point. Since vmgateway is an enterprise component, we built our own: vm-receive-route, a distributed stream aggregation gateway.

Key Insight from Source Code

Two aspects of the native implementation are particularly relevant:

Time window range:

currentTime := fasttime.UnixTimestamp()
deleteDeadline := currentTime + as.intervalSecs + (as.intervalSecs >> 1)

The window has a 50% grace period (intervalSecs >> 1) beyond the configured interval. This is the only protection against late-arriving data — and it's quite generous, which means stale data can still influence results within that extended window.

Calculation logic:

lv, ok := sv.lastValues[inputKey]
if !ok {
    lv = &lastValueState{}
    sv.lastValues[inputKey] = lv
}
d := value
if ok && lv.value <= value {
    d = value - lv.value
}
if ok || currentTime > as.ignoreInputDeadline {
    sv.total += d
}

The aggregation only checks if a previous value exists and is smaller — it doesn't handle the case where a late-arriving sample with a lower value (after a restart, for example) creates a negative delta that's simply ignored. There's no deduplication, no gap detection, no intelligent merging.

What the Gateway Solves

1. Asynchronous Processing

Most remote write adapters (like prometheus-kafka-adapter) do synchronous forwarding — they wait for the downstream (Kafka, etc.) to acknowledge before accepting the next batch. Stream aggregation has window constraints: if the write pipeline blocks and samples arrive late, they miss their window and calculations drift.

The gateway decouples ingestion from forwarding using an internal buffer. The remote write endpoint returns immediately, and samples are forwarded asynchronously to the stream aggregation backend. This prevents back-pressure from creating cascading delays.

2. Time Window Filtering

Since stream aggregation already computes deltas between successive values, there's no need for complex out-of-order handling at this layer. The gateway simply cooperates with the aggregation window:

Samples arriving within the window → forward normally
Samples arriving outside the window → discard

This solves two problems at once:

Prometheus retries that send large backlogs of old samples no longer corrupt real-time results
The resource overhead of processing those retries is eliminated at the gateway level — the aggregation backend never sees them

3. Dimension Control

The stream aggregation component inserts a node ID into each aggregated time series to distinguish labels across compute nodes. But as nodes scale horizontally, the cardinality of that label scales too. You need a way to:

Control dimension growth from node identities
Route time series by dimension to the correct node

We designed a dual hashmod scheduling algorithm:

The gateway assigns a hash-based task ID to each time series based on its stable dimensions (not the node identity)
The same series always routes to the same compute node, regardless of which gateway instance processed it
The task ID dimension is bounded by the number of unique dimension combinations, not the number of gateway nodes

By moving the task ID labeling to the gateway layer, we eliminated the unbounded dimension growth that horizontal scaling would otherwise cause.

4. Backend Service Migration for Failures

When a compute node fails, its in-memory aggregation state is lost. The gateway detects failures via health checks and reroutes traffic to healthy nodes. Since the dual hashmod ensures consistent routing, the remaining nodes can immediately pick up the work, though there will be a brief period of incomplete aggregation until the state rebuilds.

Record Rule Dimension Task Generator

Stream aggregation is great for reducing cardinality of single metrics. But real-world monitoring scenarios require combining multiple metrics with functions — which is where Prometheus Record Rule comes in.

The Problem with Record Rules at Scale

Consider an HTTP request metric with a req_path dimension:

Before stream aggregation:

a_http_req_total{zone="bj", src_svr="192.168.1.2", src_port="30021", dis_svr="192.168.2.3", code="202", req_path="/api/foo?abc=xyz"}
a_http_req_total{zone="bj", src_svr="192.168.1.2", src_port="30023", dis_svr="192.168.2.3", code="202", req_path="/api/bar?abc=def"}
a_http_req_total{zone="bj", src_svr="192.168.1.2", src_port="10021", dis_svr="192.168.2.3", code="202", req_path="/api/baz?abc=ghi"}
...

After stream aggregation (dropping req_path and src_port):

agg_a_http_req_total{zone="bj", src_svr="192.168.1.2", dis_svr="192.168.2.3", code="202"}
agg_a_http_req_total{zone="bj", src_svr="192.168.1.2", dis_svr="192.168.2.3", code="500"}
agg_a_http_req_total{zone="bj", src_svr="192.168.1.2", dis_svr="192.168.2.3", code="400"}

But what you actually want to display is the success rate per target:

sum by (dis_svr) (
    rate(a_http_req_total{code=~"2.*"}[5m])
)
/
sum by (dis_svr) (
    rate(a_http_req_total{}[5m])
)

You can use Record Rule to precompute this:

groups:
  - name: a_http_req_total:sum:rate:5m
    rules:
      - expr: sum by (src_svr, dis_svr, code) (rate(a_http_req_total{}[5m]))
        record: a_http_req_total:sum:rate:5m

The problem? Record Rule loads all dimensions of the metric into memory. When dimension counts reach critical thresholds, it triggers OOM. Even below that threshold, higher cardinality means slower computation.

In production, istio_requests_total QPS could be delayed by 20 minutes at high dimension counts. After applying stream aggregation to reduce from tens of millions of time series down to tens of thousands, the delay dropped to 1-2 minutes — better, but still far from real-time.

Dynamic Dimension-Split Record Rules

The issue is that Record Rule evaluates one query per group, loading everything into memory. But if you split the query by specific dimension values, you can process them concurrently.

The static approach after stream aggregation:

groups:
  - name: agg_a_http_req_total:sum:rate:5m-2xx
    rules:
      - expr: sum by (src_svr, dis_svr, code) (rate(agg_a_http_req_total{code=~"2.*"}[5m]))
        record: agg_a_http_req_total:sum:rate:5m
  - name: agg_a_http_req_total:sum:rate:5m-4xx
    rules:
      - expr: sum by (src_svr, dis_svr, code) (rate(agg_a_http_req_total{code=~"4.*"}[5m]))
        record: agg_a_http_req_total:sum:rate:5m
  - name: agg_a_http_req_total:sum:rate:5m-5xx
    rules:
      - expr: sum by (src_svr, dis_svr, code) (rate(agg_a_http_req_total{code=~"5.*"}[5m]))
        record: agg_a_http_req_total:sum:rate:5m

But the problem is that in production, dimension labels are dynamic. You can't hardcode splits for every dimension value. You need a label metadata management system that watches dimension combinations and dynamically generates split queries.

The `ruler-handle-process` Component

We built a small metadata watch and rule builder that automates this. Its configuration looks like:

recode_rules:
  - interval: 5m
    recode_to: istio_requests_total:sum:rate:5m
    metric_name: istio_requests_total
    aggr_type: sum
    vector_type: rate
    vector_range: 5m
    group_by_and_filter:
      - source_workload
      - destination_workload
      - cluster
      - namespace
    group_by:
      - response_code
      - namespace
      - source_workload_namespace
      - destination_workload_namespace
      - destination_service_name
      - cluster
      - reporter
    filter_by:
      cluster: "k8s-hw-bj-xxxxxx"
    with_out:
      source_workload: "ingressgateway-workflows"

The component watches the actual dimension combinations under the metric name istio_requests_total and generates a set of Record Rule configurations — one per unique dimension combination. Combined with Prometheus's Rule component for concurrent evaluation, this reduced computation latency from minutes to seconds.

The generated rules look like:

groups:
  - name: istio_requests_total:sum:rate:5m-7218756fe8a0bc327e818812cefb02f7
    rules:
      - expr: sum by (...) (rate(istio_requests_total{cluster="k8s-hw-bj-1-prod", destination_workload="skyaxe-778-flink", ...}[5m]))
        record: istio_requests_total:sum:rate:5m
  - name: istio_requests_total:sum:rate:5m-8e30244048f8d5519a6332f309578ed4
    rules:
      - expr: sum by (...) (rate(istio_requests_total{cluster="k8s-hw-bj-1-prod", destination_workload="t-bean-portal", ...}[5m]))
        record: istio_requests_total:sum:rate:5m

Each unique dimension combination gets its own rule group, enabling true concurrent computation.

Architecture Combinations for Different Scales

Using community open-source components alongside our custom gateway and rule builder, we assembled a tiered architecture that handles everything from small deployments to massive fleets.

Small Scale: Tens of Thousands of Single Metrics

Minimal setup — vmagent with built-in stream aggregation. No gateway needed. The single-node limits aren't reached yet.

Medium Scale: Tens of Thousands of Multi-Metrics

Add Record Rules for computed metrics. Use the dimension-split approach to keep computation fast. Stream aggregation reduces cardinality before Record Rule evaluation.

Large Scale: Millions+ of Single Metrics

This is where the distributed gateway comes in:

Multiple vmagent instances behind the gateway for horizontal ingestion
Dual hashmod scheduling ensures consistent routing
Time window filtering at the gateway prevents retries from polluting results
Asynchronous forwarding prevents back-pressure from creating cascading delays

Large Scale: Scenario-Based Computational Aggregation for Millions of Metrics

The full stack:

Stream aggregation — first pass cardinality reduction at ingestion
Distributed gateway — routing, filtering, and load distribution
Dynamic rule builder — watches dimension metadata and generates concurrent Record Rule groups
Rule engine — evaluates split queries in parallel

Summary

VictoriaMetrics' stream aggregation is a solid foundation, but it was designed for single-node operation. When you need to scale horizontally, you encounter a series of interconnected problems — collection gaps, dimension explosion, record rule performance bottlenecks — that the native implementation doesn't address.

The solutions aren't particularly complex individually (async processing, time window filtering, hash-based routing, dimension-aware rule generation), but they need to work together as a coherent system. The gateway pattern — intercepting the remote write path before it reaches the aggregation layer — proved to be the right abstraction point for injecting these capabilities without modifying upstream code.

The ruler-handle-process component was the second key insight: rather than fighting Prometheus Record Rule's single-query-per-group limitation, we embraced it by dynamically splitting queries by dimension and running them concurrently. This turned a 20-minute computation into a seconds-level operation.

If you're running VictoriaMetrics at scale and hitting these issues, the patterns described here should be applicable regardless of your specific stack. The gateway approach is generic enough to work with any Prometheus-compatible remote write backend.

Originally published on my blog

MiBeeNvr: A Lightweight Home NVR System I Built

ZhengZhiCong — Sun, 07 Jun 2026 09:27:47 +0000

I have several cameras at home — a few Xiaomi cameras, some DIY ESP32 cameras, and multiple Raspberry Pi CSI cameras. I'd been using cloud storage solutions, but I was never comfortable with them: vendor lock-in, network dependency, and the costs add up. So I decided to build my own NVR system, called MiBeeNvr.

Why Build MiBeeNvr

To be honest, I was never satisfied with existing cloud storage solutions. Take Xiaomi cameras, for example. By default, you can only view them through the Mi Home app. Recordings are either stored on an SD card (limited capacity, frequent plugging/unplugging) or in the cloud. Cloud storage costs tens of dollars per month, and there's the privacy concern — you never know when the manufacturer might use your video data for AI training or sell it to third parties. Not to mention vendor lock-in — switching platforms is nearly impossible.

ESP32 cameras have a similar problem. I built several ESP32 cameras, storing recordings on SD cards, but viewing and playback were inconvenient. I needed a unified management platform.

I also tried other open-source solutions: ZoneMinder requires a LAMP stack — installing and deploying it is more complex than my entire project; Shinobi's configuration is a nightmare; and some smaller projects are basically unmaintained. Frigate is nice but primarily focused on AI detection and depends on Docker — too heavy.

In short, I wanted something that is:

A single binary file — download and run
Lightweight enough to run on a Raspberry Pi
Supports multiple camera types, especially Xiaomi's proprietary protocol
Clean Web interface without frontend complexity
Auto-cleanup of old recordings, won't fill up the disk

After searching around, none of the existing solutions fit. So I wrote my own.

What is MiBeeNvr

MiBeeNvr is a lightweight NVR system written in Go, designed to solve local storage for home cameras.

Overall Architecture

The system has three layers:

Camera End — multiple camera types connect through different protocols:

Xiaomi cameras use the proprietary "miss" protocol (encrypted, multi-layer)
ESP32 cameras send HTTP JPEG/MJPEG streams directly
Raspberry Pi CSI cameras output standard RTSP H.264 via MediaMTX

Protocol Bridge Layer — proprietary protocols get converted to standard RTSP:

go2rtc handles Xiaomi's miss protocol decryption and transcodes it to RTSP
MediaMTX converts Raspberry Pi CSI interface video to RTSP

MiBeeNvr Core — handles the recording logic:

REST API receives all video streams
Recording engine segments incoming video into MP4 files
SQLite stores metadata (pure Go, no CGO, no separate DB installation)
Auto-cleanup daemon removes old recordings per retention policy
HLS live streaming for real-time viewing (up to 4 concurrent)

Access Methods — users interact through:

Built-in Web UI (Svelte 5 SPA, embedded in the single binary)
WebDAV (read-write) and FTP for file-level access
Prometheus metrics for system monitoring

Result: cameras → protocol bridge → MiBeeNvr core → user access, all in a single binary.

Recording Pipeline

Video processing follows a straightforward pipeline:

Input — RTSP streams are handled via gortsplib, HTTP JPEG streams are periodically grabbed as frames
Decode & Mux — RTP packets are depacketized via pion/rtp, then muxed into MP4 via go-mp4
Storage — video is segmented into files (configurable: 30s or 10m intervals), SQLite tracks metadata, files go to disk

The frontend uses Svelte 5 — the entire SPA is compiled to static assets and embedded into the Go binary. Deployment is a single file, no separate Web server needed.

Backend tech stack:

Go 1.26 + modernc.org/sqlite (pure Go, no CGO dependency)
chi routing library, clean and efficient
gortsplib for RTSP/RTP protocol
pion/rtp for real-time streaming

SQLite was chosen because it's single-file, pure Go, performs well enough for home use, supports concurrent access, and most importantly, doesn't require a separate database installation.

Design Philosophy

The entire project's design philosophy is "simple and straightforward:"

Single binary file, no external dependencies
Supports cross-compilation, runs on AMD64/ARM64
YAML configuration, intuitive
Built-in Web interface, open browser to use
Minimal resource usage, runs smoothly on Raspberry Pi 4

Key Features

Supports multiple camera protocols: RTSP (H.264/H.265), HTTP JPEG
Built-in Web interface with dark/light theme switching
Chinese/English bilingual support
WebDAV (read-write), FTP, REST API
MQTT-triggered recording, ideal for smart home integration
Prometheus monitoring metrics
Per-camera independent retention policies
MP4 segmented recording, auto-cleanup of old files
Supports HLS live streaming (up to 4 concurrent)

My Actual Deployment

I run it on an ARM64 mini host with 512MB RAM and 2GB storage. The system runs very stably — basically set it and forget it.

Connected 4 cameras, each with its own characteristics:

Raspberry Pi CSI Camera — RTSP bridge via MediaMTX, converting CSI interface video to standard RTSP. Configured as rtsp_h264.
ESP32-S3 Camera — DIY, running MJPEG stream via HTTP protocol. Configured as http_jpeg.
Xiaomi Camera (Balcony) — Protocol conversion via go2rtc (Xiaomi proprietary → RTSP), 2K resolution, configured as rtsp_h265.
Xiaomi Camera (Living Room) — Same as above, 1080P.

Configuration is 30-second segment recording with 1-day retention. This interval is a trade-off: too short creates too many files, too long makes it inconvenient to look up incidents. WebDAV (read-write) and FTP are enabled for convenient phone viewing and backup.

The Web interface is clean — camera management and recording lists are straightforward.

Settings page:

Configuration File

The complete configuration file looks like this — YAML format, clear at a glance:

server:
  listen: ":9090"

storage:
  root_dir: "/mnt/data/nvr"
  segment_duration: "30s"

cameras:
  - id: "rpi-csi-cam"
    name: "RPi CSI Camera"
    protocol: "rtsp_h264"
    url: "rtsp://10.0.1.100:8554/stream"
    enabled: true

  - id: "esp32-cam"
    name: "ESP32-S3 Camera"
    protocol: "http_jpeg"
    url: "http://10.0.1.101/capture"
    enabled: true

  - id: "xiaomi-balcony"
    name: "Xiaomi Camera"
    protocol: "rtsp_h265"
    url: "rtsp://10.0.1.102:8554/xiaomi_stream"
    enabled: true

cleanup:
  retention_days: 30
  disk_threshold_percent: 95

auth:
  username: "admin"
  password_hash: "Use mibee-nvr hash-password command to generate"

webdav:
  enabled: true
  path_prefix: "/dav"
  read_write: true

ftp:
  enabled: true
  port: 2121

Xiaomi Camera Integration

Xiaomi camera protocol is a major headache. It uses its proprietary "miss" (Mi Secure Streaming) protocol with multi-layer encryption, without a standard RTSP interface. Even if you know the camera's IP, you can't pull a stream with VLC.

Fortunately, there's go2rtc, a lifesaver. The integration works in four steps:

Account auth & key exchange — go2rtc logs into your Xiaomi account, retrieves device lists and encryption keys from Xiaomi Cloud
Establish P2P connection — go2rtc initiates a miss protocol handshake with the camera, establishing a peer-to-peer link
Video stream relay — the camera sends its encrypted miss stream to go2rtc continuously; go2rtc decrypts and transcodes it into a standard RTSP stream (H.265); MiBeeNvr receives this as a normal RTSP camera and records segmented MP4 files
Independent access — once set up, you no longer need the Mi Home app or Xiaomi cloud subscription; all recordings are accessible via MiBeeNvr's Web UI, WebDAV, or FTP

The entire process requires no firmware flashing, no camera disassembly, and no Xiaomi cloud storage subscription. go2rtc handles all the protocol conversion.

go2rtc Deployment

The easiest way to deploy go2rtc is with Docker:

# Create configuration file
cat > go2rtc.yaml << EOF
streams:
  xiaomi_balcony:
    - xiaomi://your_account:cn@10.0.1.100?did=your_camera_did&model=isa.camera.hlc7
  xiaomi_living_room:
    - xiaomi://your_account:cn@10.0.1.101?did=your_camera_did&model=isa.camera.mj200

rtsp:
  listen: ":8554"
EOF

# Run container
docker run -d --name go2rtc \
  -p 8554:8554 \
  -p 1984:1984 \
  -v $(pwd)/go2rtc.yaml:/config.yaml \
  alexxit/go2rtc

Key points:

The xiaomi:// protocol requires Xiaomi account and password authentication
did is the device's unique identifier, model is the device model (can be found in the Mi Home app)
go2rtc automatically handles P2P connection and miss protocol decryption
The final standard RTSP stream is exposed on port 8554, and MiBeeNvr connects to it like any normal camera

Then point to go2rtc in MiBeeNvr's config:

cameras:
  - id: "xiaomi-balcony"
    name: "Xiaomi Camera"
    protocol: "rtsp_h265"
    url: "rtsp://localhost:8554/xiaomi_balcony"
    enabled: true

Pitfalls Encountered

Xiaomi camera integration has several pitfalls:

First-time network connection: Xiaomi cameras must be able to reach the internet for key exchange with Xiaomi servers. After connection is established, subsequent transmission is over LAN.
Device ID acquisition: Each camera's did is unique. Use go2rtc's WebUI (port 1984) for auto-discovery, or dig through the Mi Home app.
Not all models are supported: go2rtc maintains a compatibility list — check before buying a camera.
H.265 vs H.264: Newer Xiaomi cameras mostly use H.265. MiBeeNvr supports both codecs, but H.265 saves storage space.

ESP32 Camera Projects

While working on MiBeeNvr, I also built several ESP32 camera firmware projects. ESP32 cameras had their share of pitfalls, but were also quite interesting.

I built three firmware projects with different positioning, all designed as upstream capture endpoints for MiBeeNvr — cameras handle video capture, MiBeeNvr handles unified storage and management.

MiBeeCam — ESP32-S3-A10 Solution

GitHub · MIT License

This is the most successful solution. ESP32-S3-A10 dev board + OV2640 camera (8225N module), 16MB Flash, ESP-IDF v5.4.3 development. Features include MJPEG stream, frame-differencing motion detection, Web config interface, Prometheus metrics. Having an LCD screen makes debugging much easier.

AI Thinker ESP32-CAM — Classic Solution

GitHub · MIT License

Entry-level choice, AI Thinker ESP32-CAM dev boards are widely available for around $10-15. 4MB Flash + 4MB PSRAM, runs MJPEG stream without issues. Highlights include SD card storage and NAS upload (WebDAV/HTTP), plus adaptive dark scene detection — automatically switches to infrared mode at night. Downside: no screen, 4MB Flash is limited.

MiBeeHomeCam — XIAO ESP32-S3 Sense

GitHub · GPL v3.0

The most advanced solution. XIAO ESP32-S3 Sense board is compact and refined, with dual camera support (OV2640/OV3660), 8MB Octal PSRAM. Highlights include AVI segmented recording (real video recording, not just snapshots), FTP/WebDAV dual-protocol upload, watchdog anti-freeze, chip temperature monitoring, batch file management. Suitable for long-term stable operation.

Selection Guide

Beginners: Choose AI Thinker ESP32-CAM — cheap with plenty of resources
Daily use: Choose MiBeeCam — LCD screen makes debugging convenient
Maximum features: Choose XIAO ESP32-S3 Sense — most powerful

System Service Configuration

For stable operation, I use systemd to manage MiBeeNvr:

[Unit]
Description=MiBee NVR
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=nvr
ExecStart=/mnt/data/nvr/bin/mibee-nvr -config /mnt/data/nvr/mibee-nvr.yaml
WorkingDirectory=/mnt/data/nvr
Restart=on-failure
RestartSec=5

# Security hardening
NoNewPrivileges=true
ProtectSystem=strict
ReadWritePaths=/mnt/data/nvr
PrivateTmp=true

[Install]
WantedBy=multi-user.target

Save to /etc/systemd/system/mibee-nvr.service, then systemctl enable --now mibee-nvr. Auto-start on boot, auto-restart on failure.

Open Source

MiBeeNvr is open source, stars and contributions welcome:

MiBeeNvr: https://github.com/Mi-Bee-Studio/MiBeeNvr (MIT License)
MiBeeCam: https://github.com/Mi-Bee-Studio/luatos-esp32s3-a10-camera (MIT)
AI Thinker ESP32-CAM: https://github.com/Mi-Bee-Studio/ai-thinker-esp32-cam (MIT)
MiBeeHomeCam: https://github.com/Mi-Bee-Studio/seeed-esp32s3-cam (GPL v3.0)

Documentation is comprehensive, with detailed deployment and configuration instructions.

Closing Thoughts

To be honest, I built this project mainly because I was dissatisfied with all the existing solutions. Cloud storage is too expensive, open-source solutions are too heavy, and commercial products are too closed. Building my own was just right: lightweight, free, and fully under my control.

Oh, about the name MiBeeNvr — "Mi" stands for me (Mickey), "Bee" stands for... let's keep that a secret, and "Nvr" is naturally Network Video Recorder. Simple, memorable, and a bit meaningful.

If you also have home camera needs or ideas about NVR systems, feel free to reach out. Issues are welcome on GitHub.

Originally published on my blog.

DEV Community: ZhengZhiCong

VictoriaMetrics Stream Aggregation: A Three-Year Retrospective (2026)

I. The Problems We Identified in 2023

II. Three Years Later — What the Official Project Has Done

✅ Perfectly Resolved

Issues P3, P5: Distributed Task Allocation & Resource Balancing

Issue P2: Single-Node Compute Scaling

Out-of-Order / Delayed Data Accuracy (P1 — Partial Mitigation)

❌ Still Unresolved

True Distributed Stream Aggregation Coordination

Task ID Dimension Explosion (P6)

III. stream-metrics-route: Current Status and Value

Core Code Architecture

The Core Algorithm (from remotecluster.go)

Is It Still Needed in 2026?

IV. Recommended 2026 Hybrid Architecture

Key Configuration

V. Evolution Recommendations

Short-term

Medium-term

Long-term

Putting It All Together

Scaling VictoriaMetrics Stream Aggregation: A Deep Dive into Distributed Design（2023）

Community VM Stream Aggregation — Capability Analysis

Core Computation: The pushSample Function

What Stream Aggregation Looks Like in Practice

Common Issues with Native Stream Aggregation

The Collection Gap Problem

Distributed Computing Challenges

Design and Implementation: A Distributed Stream Aggregation Gateway

Key Insight from Source Code

What the Gateway Solves

1. Asynchronous Processing

2. Time Window Filtering

3. Dimension Control

4. Backend Service Migration for Failures

Record Rule Dimension Task Generator

The Problem with Record Rules at Scale

Dynamic Dimension-Split Record Rules

The ruler-handle-process Component

Architecture Combinations for Different Scales

Small Scale: Tens of Thousands of Single Metrics

Medium Scale: Tens of Thousands of Multi-Metrics

Large Scale: Millions+ of Single Metrics

Large Scale: Scenario-Based Computational Aggregation for Millions of Metrics

Summary

MiBeeNvr: A Lightweight Home NVR System I Built

Why Build MiBeeNvr

What is MiBeeNvr

Overall Architecture

Recording Pipeline

Design Philosophy

Key Features

My Actual Deployment

Configuration File

Xiaomi Camera Integration

go2rtc Deployment

Pitfalls Encountered

ESP32 Camera Projects

MiBeeCam — ESP32-S3-A10 Solution

AI Thinker ESP32-CAM — Classic Solution

MiBeeHomeCam — XIAO ESP32-S3 Sense

Selection Guide

System Service Configuration

Open Source

Closing Thoughts

The Core Algorithm (from `remotecluster.go`)

Core Computation: The `pushSample` Function

The `ruler-handle-process` Component