While contributing to Grafana, I picked up a memory issue in the Time Comparison feature — a follow-up to earlier performance work I had done in the same area.
A comparison panel was consuming significantly more memory than expected. The interesting part: the extra memory wasn't coming from real data.
This post covers how I traced it to the root cause and fixed it.
Background
Time Comparison overlays an earlier period onto the current one — for example, this week vs. last week. The comparison data is fetched from the earlier window and shifted forward before rendering:
Query → DataFrame → Prepare frame → Shift → Render
│
└─ Gap filling
The important detail: gap filling ran before the comparison frame was shifted.
The Problem
I reproduced the issue with:
| Parameter | Value |
|---|---|
| Series | 500 |
| Window | 6h |
| Interval | 20s |
| Compare offset | 24h |
A single-period panel contained roughly 540,000 points, so a comparison panel should be about 2× the baseline.
Instead, the compare frame contained 3,240,500 points — ~6× the baseline — and consumed 76.4 MB.
The question was: where did the extra points come from?
Investigation
I first verified the baseline to rule out the query returning unexpected data. It was correct.
Then I used a reproducible browser harness and a heap snapshot to inspect the extra memory. Most of it was null rows introduced during gap filling — not real samples, not copies.
Following the frame through the preparation pipeline revealed why. When gap filling ran, the compare frame still represented data 24 hours in the past, but the gap-filler was using the current time range as its reference:
Compare frame Current range
[===== 6h =====] [===== 6h =====]
└─────────────── 24h ───────────────┘
gap-filler reads this offset as one gap
At a 20-second interval, 24 hours is:
24 × 60 × 60 / 20 = 4,320 intervals
So up to 4,320 null positions per series were introduced purely because the frame hadn't been shifted yet. The frame was then shifted forward, leaving most of that padding outside the visible range — where nothing draws it.
Root Cause
The issue was the reference range gap filling used. It evaluated the compare frame against the current range instead of the range the frame actually represented:
Before After
────── ─────
Compare frame Compare frame
↓ ↓
Gap-fill vs current range Use frame's effective range
↓ ↓
Shift by 24h Gap-fill
↓
Shift by 24h
The Fix
I changed the gap-filling logic to account for the compare frame's own offset when determining its effective time range. This means:
- genuine gaps are still filled
- non-comparison frames are unchanged
- the unnecessary null padding is eliminated
Results
| Scenario | Before | After |
|---|---|---|
| Compare points | 3,240,500 | 1,081,000 |
| Compare heap | 76.4 MB | 22.8 MB |
| 12× dashboard heap | 822.5 MB | 225.0 MB |
The compare frame went from roughly 6× to 2× the baseline — consistent with the actual amount of data being rendered. I also verified it in the browser: the reproduced panel dropped from 144.2 MB to 84.4 MB.
What I Learned
1. Performance bugs can be data-flow bugs. The problem wasn't an expensive algorithm — it was a correct operation running against the wrong state.
2. Profile before optimizing. The heap snapshot showed the extra memory was synthetic null data, which immediately narrowed the search.
3. Large-codebase bugs often live between components. This one emerged from the interaction between time comparison, frame preparation, gap filling, and time shifting — not from a single isolated function.
Closing
This reinforced a lesson I keep coming back to while contributing to large open source projects:
When debugging performance, follow the data — not just the allocations.
The fix is up as a pull request against Grafana:
TimeComparison: Stop padding compare frames across the compare offset
#129934
Follow-up to the memory investigation in #129681.
Compare frames are gap-filled while still sitting in their own earlier window - the panel only shifts them onto the current range afterwards. Measuring them against the unshifted current range reads the whole compare offset as a gap, so each frame gets padded with a null row per interval across it. Those rows then travel forward with the shift and land outside the visible range, where nothing can see them.
The fix measures a compare frame against the window it actually covers. Genuine gaps inside a frame still fill, and non-compare frames are untouched.
Measured with @drew08t's benchmark from #129806 (500 series, 6h window, 20s interval, 24h offset):
| Scenario | Prepared points | Heap |
|---|---|---|
| Compare — before | 3,240,500 | 76.4 MB |
| Compare — after | 1,081,000 | 22.8 MB |
| Dashboard 12x compare — before | 38,886,000 | 822.5 MB |
| Dashboard 12x compare — after | 12,972,000 | 225.0 MB |
Compare now reports 2.00x the points of a single-period baseline instead of 6.00x — two periods of 500 series should cost exactly twice one period, and it now does.
Also confirmed in a browser on the four-panel harness, toggling the fix in one session: the compare panel goes from 144.2 MB to 84.4 MB, with compare frames back to their true 1080 points. It now sits alongside an equivalent 1000-series non-compare panel (81.7 MB), the remaining ~3 MB being genuine compare overhead.
Note the benchmark in #129806 is still a draft, so those numbers aren't reproducible from main until it lands.
Ref #125104
Top comments (1)
the heap snapshot made the cause clear, synthetic null rows from the wrong time window. a useful regression test could vary the compare offset, interval, and series count, then assert both point count and bytes before rendering. i would also keep one test for real gaps inside the compare window, so the fix does not remove valid padding. this gives the data flow a clear contract.