How do you deduplicate a stream you don't control?

Field notes, part 1 of 3. Data plumbing from an AI engineer's desk.*

My job title says AI. A meaningful share of my week is data.

Not the glamorous part — no model architecture, no eval harness. Just the plumbing that decides whether the model ever sees the right thing at the right time. And lately I've been stuck on a shape of problem I now see everywhere: something upstream is chatty, and something downstream is expensive.

The expensive thing varies: re-embedding a document, an LLM enrichment call, a search index that must be re-committed. The arithmetic doesn't. Upstream emits fifty events about one entity in twenty seconds, and downstream bills you fifty times for an answer that changed once.

Looking for how this is solved at scale, I landed on a PyCon DE talk by Mirano Tuk and Filip Bacic — How to Search Through 800 Billion Records in Real Time. Their scale is far past mine, but the pattern is small and ported straight onto my problems. These notes are me working it through on my own data.

Deduplicating a stream means collapsing repeated events about one key into a single unit of downstream work. A per-batch set mostly fails, because a batch is a time window whose width you don't control. A TTL buffer decouples that window from throughput — it cut my simulated stream from 904,298 messages to 200,398, a 4.51× reduction.

Why doesn't a per-batch set work?

First instinct: dedupe within each batch. You're already polling in batches, so throw the keys in a set and you're done.

for batch in consumer:
    for key in {msg.key for msg in batch}:
        process(key)

I simulated a stream shaped like the real thing — 200,000 distinct keys, each emitting a burst of updates over a few seconds, 904,298 messages across an hour:

Strategy	Effective window	Downstream work	Reduction
No dedup	—	904,298	1.00×
Per-batch set, 1,000	4.0s	548,524	1.65×
Per-batch set, 10,000	39.8s	260,396	3.47×
Per-batch set, 50,000	199.0s	212,122	4.26×
TTL buffer, 30s	30s	215,364	4.20×
TTL buffer, 60s	60s	200,398	4.51×
TTL buffer, 300s	300s	200,000	4.52×

Read the window column before the reduction column. That's the thing I got wrong when I first ran this.

A batch of 1,000 messages at 251 msg/s is a four-second window. A batch of 50,000 is a 199-second window. Per-batch dedup isn't a different technique from TTL dedup — it's the same technique with a window you didn't choose. Its width is batch_size / throughput.

Which means it moves in exactly the wrong direction. Traffic doubles, your window halves. The moment duplicates are most abundant is the moment your dedup window is narrowest. That's not a knob, it's a trapdoor.

The TTL buffer's entire contribution is decoupling the window from throughput. That's it. That's the idea.

Can a plain dict work as the buffer?

Here's the part that delighted me, because I expected to need a real data structure.

Since Python 3.7, regular dicts preserve insertion order — that's a language guarantee, not a CPython implementation detail. So a plain dict is a FIFO queue. The oldest entry is the first one you iterate. Which means expiry checking is O(1) amortised: look at the front, and if it hasn't expired, nothing behind it has either.

class Deduplicator:
    def __init__(self, ttl, on_evict):
        self.ttl = ttl
        self.on_evict = on_evict
        self.buf = {}                      # key -> expiry, in insertion order

    def add(self, key, now):
        self.expire(now)
        if key in self.buf:
            return                          # already scheduled, drop it
        self.buf[key] = now + self.ttl

    def expire(self, now):
        while self.buf:
            key = next(iter(self.buf))      # oldest entry
            if self.buf[key] > now:
                break                       # nothing behind it can be older
            del self.buf[key]
            self.on_evict(key)

That's the whole thing. No heap, no external store, no Redis.

I did wonder about Redis, and the answer is a nice piece of reasoning: if your key is also your partition key, every event for a given key lands on the same partition, and therefore the same consumer replica. The buffer is correctly local. Reaching for a shared store would add a network hop to enforce an invariant the partitioner already gives you for free.

The bug I'd have shipped

Look again at expire(). The processing happens on eviction — when the key leaves the buffer — not when it arrives.

My first version did the obvious thing and processed on arrival, treating the buffer purely as a "have I seen this?" filter. It looks equivalent. It isn't, and the gap is enormous:

Variant	Work	Keys stale at the end	% stale
ttl=5s, process on insert	425,508	58,239	29.1%
ttl=5s, process on evict	425,508	0	0.0%
ttl=30s, process on insert	215,364	141,879	70.9%
ttl=30s, process on evict	215,364	0	0.0%
ttl=60s, process on insert	200,398	155,650	77.8%
ttl=60s, process on evict	200,398	0	0.0%

Same amount of work. Wildly different correctness.

Processing on insert commits the first version of the record and then discards every update that arrives inside the window. If the last update for a key lands during its own TTL — which, given that updates arrive in bursts, is the common case, not the edge case — you have permanently stale data and no error anywhere to tell you.

At a 60-second TTL, 77.8% of keys ended the run holding a version that was not the latest one.

Flipping to process-on-eviction fixes it completely. You can't know which message is the last one for a key, but if you wait out the TTL and then read the current state, you get the state after the last message in that window. It's an approximation of "process the final update" that costs nothing extra.

The tell is that the work column is identical. This isn't a trade-off. The insert version is just wrong.

How long should the TTL be?

The TTL is latency you are choosing to add. A 60-second buffer means data becomes visible up to a minute after it arrives.

The returns die fast. Going 30s → 60s buys 4.20× → 4.51×. Going 60s → 300s buys 4.51× → 4.52×, for five times the latency. Almost all of the available win is in the first half-minute, because that's the width of the bursts. Set the TTL to roughly the width of your upstream's burst, and stop.

There's a floor you can't cross: 200,000 keys means at least 200,000 units of work. At a 300s TTL the buffer is doing literally nothing but adding delay.

What I'm taking to my own work

The reason I chased this down: the same shape sits underneath a lot of AI infrastructure, and I'd been solving it badly with cron jobs.

A document store where every edit triggers re-embedding. A user-activity stream where every event triggers a profile refresh through an LLM. Both are chatty-upstream, expensive-downstream, and both are places I'd previously reached for "just batch it hourly" — which is a TTL buffer with the worst possible TTL and no correctness story at all.

• Process on eviction, never on insert. Same cost, and it's the difference between fresh and silently stale.

• A batch is a window you didn't choose. If dedup matters, choose it explicitly.

• Set the TTL from your burst width, not from a latency budget someone made up.

TL;DR

• A per-batch set is a TTL buffer with a window of batch_size / throughput — it narrows exactly when traffic spikes and duplicates matter most.

• A TTL buffer cut 904,298 messages to 200,398 units of work (4.51×) in my simulation, against a hard floor of 200,000.

• Process on eviction, not insertion. At a 60s TTL, processing on insert left 77.8% of keys holding stale data for identical cost.

• Since Python 3.7 a plain dict is a FIFO queue, so the buffer needs no heap and no Redis — the partitioner already guarantees a key's events reach one replica.

Reproduce this

python 3.13 · stdlib only · seed 20260810
200,000 keys · mean 4 updates each · 8s burst spread · 1 hour horizon

Next: Why committing Kafka offsets out of order loses data — the version of this buffer I'd have shipped drops 4,456 messages on the floor the first time it restarts.

---

Credit where it's due: the pattern in this note comes from Mirano Tuk and Filip Bacic's PyCon DE 2026 talk. The simulation, the numbers, and any mistakes are mine.

👉 Follow me: LinkedIn | GitHub

---

Originally published on ZyVOP

💡 For more articles like this, subscribe to the ZyVOP newsletter!