Tracing a 3 Memory Blow-Up in Grafana's Time Comparison

SACHIN INDWAR — Fri, 07 Aug 2026 15:15:46 +0000

While contributing to Grafana, I picked up a memory issue in the Time Comparison feature — a follow-up to earlier performance work I had done in the same area.

A comparison panel was consuming significantly more memory than expected. The interesting part: the extra memory wasn't coming from real data.

This post covers how I traced it to the root cause and fixed it.

Background

Time Comparison overlays an earlier period onto the current one — for example, this week vs. last week. The comparison data is fetched from the earlier window and shifted forward before rendering:

Query → DataFrame → Prepare frame → Shift → Render
                         │
                         └─ Gap filling

The important detail: gap filling ran before the comparison frame was shifted.

The Problem

I reproduced the issue with:

Parameter	Value
Series	500
Window	6h
Interval	20s
Compare offset	24h

A single-period panel contained roughly 540,000 points, so a comparison panel should be about 2× the baseline.

Instead, the compare frame contained 3,240,500 points — ~6× the baseline — and consumed 76.4 MB.

The question was: where did the extra points come from?

Investigation

I first verified the baseline to rule out the query returning unexpected data. It was correct.

Then I used a reproducible browser harness and a heap snapshot to inspect the extra memory. Most of it was null rows introduced during gap filling — not real samples, not copies.

Following the frame through the preparation pipeline revealed why. When gap filling ran, the compare frame still represented data 24 hours in the past, but the gap-filler was using the current time range as its reference:

   Compare frame                         Current range
   [===== 6h =====]                      [===== 6h =====]
          └───────────────  24h  ───────────────┘
                gap-filler reads this offset as one gap

At a 20-second interval, 24 hours is:

24 × 60 × 60 / 20 = 4,320 intervals

So up to 4,320 null positions per series were introduced purely because the frame hadn't been shifted yet. The frame was then shifted forward, leaving most of that padding outside the visible range — where nothing draws it.

Root Cause

The issue was the reference range gap filling used. It evaluated the compare frame against the current range instead of the range the frame actually represented:

Before                         After
──────                         ─────
Compare frame                  Compare frame
     ↓                              ↓
Gap-fill vs current range      Use frame's effective range
     ↓                              ↓
Shift by 24h                   Gap-fill
                                    ↓
                               Shift by 24h

The Fix

I changed the gap-filling logic to account for the compare frame's own offset when determining its effective time range. This means:

genuine gaps are still filled
non-comparison frames are unchanged
the unnecessary null padding is eliminated

Results

Scenario	Before	After
Compare points	3,240,500	1,081,000
Compare heap	76.4 MB	22.8 MB
12× dashboard heap	822.5 MB	225.0 MB

The compare frame went from roughly 6× to 2× the baseline — consistent with the actual amount of data being rendered. I also verified it in the browser: the reproduced panel dropped from 144.2 MB to 84.4 MB.

What I Learned

1. Performance bugs can be data-flow bugs. The problem wasn't an expensive algorithm — it was a correct operation running against the wrong state.

2. Profile before optimizing. The heap snapshot showed the extra memory was synthetic null data, which immediately narrowed the search.

3. Large-codebase bugs often live between components. This one emerged from the interaction between time comparison, frame preparation, gap filling, and time shifting — not from a single isolated function.

Closing

This reinforced a lesson I keep coming back to while contributing to large open source projects:

When debugging performance, follow the data — not just the allocations.

The fix is up as a pull request against Grafana:

TimeComparison: Stop padding compare frames across the compare offset #129934

sAchin-680 posted on Aug 03, 2026

Follow-up to the memory investigation in #129681.

Compare frames are gap-filled while still sitting in their own earlier window - the panel only shifts them onto the current range afterwards. Measuring them against the unshifted current range reads the whole compare offset as a gap, so each frame gets padded with a null row per interval across it. Those rows then travel forward with the shift and land outside the visible range, where nothing can see them.

The fix measures a compare frame against the window it actually covers. Genuine gaps inside a frame still fill, and non-compare frames are untouched.

Measured with @drew08t's benchmark from #129806 (500 series, 6h window, 20s interval, 24h offset):

Scenario	Prepared points	Heap
Compare — before	3,240,500	76.4 MB
Compare — after	1,081,000	22.8 MB
Dashboard 12x compare — before	38,886,000	822.5 MB
Dashboard 12x compare — after	12,972,000	225.0 MB

Compare now reports 2.00x the points of a single-period baseline instead of 6.00x — two periods of 500 series should cost exactly twice one period, and it now does.

Also confirmed in a browser on the four-panel harness, toggling the fix in one session: the compare panel goes from 144.2 MB to 84.4 MB, with compare frames back to their true 1080 points. It now sits alongside an equivalent 1000-series non-compare panel (81.7 MB), the remaining ~3 MB being genuine compare overhead.

Note the benchmark in #129806 is still a draft, so those numbers aren't reproducible from main until it lands.

Ref #125104

View on GitHub

DEV Community: SACHIN INDWAR