Why tracking "frequent lately" breaks standard patterns

If you've ever had to build high-scale rate limiting, telemetry sampling, or hot-key detection, you've likely run into a very specific, painful problem.

The requirement always boils down to one question: "How often has this thing been happening lately?"

And the system must answer this question in nanoseconds, for millions of distinct things, using a fixed amount of memory, accessed by many threads concurrently, forever.

To understand why this is so hard to build, let's look at a real-world analogy.

The Bouncer Analogy

Imagine you run the door at an enormous, very busy club. Thousands of faces walk up to the door every minute. Your job is to spot the regulars of the last hour. You don't care about the person who came every night in 2019 and stopped, and you don't care about the person who just showed up once tonight. You only care about who is showing up a lot, recently.

Now, add the engineering constraints:

1. You cannot write anything down per person. There are millions of possible faces, but your notepad is one single page, fixed size, bought once. (Fixed Memory)
2. You must answer instantly. The line cannot stop while you think. (Nanosecond Latency)
3. Memory must fade on its own. Last hour's regular who stopped coming must automatically stop counting as a regular. You don't get a nightly "erase the notepad" break, because the club never closes. (Intrinsic Decay)
4. Several of you work the door at once. You can't huddle to compare notes between guests. (Lock-free Concurrency)

If you look closely, this is actually three questions fused into one:

1. How often? (Frequency)
2. How recently? (Freshness / Decay)
3. Have I seen this one in the current shift yet? (First-sighting diversity)

Why the obvious solutions fail

When we try to solve this in software, we usually reach for standard tools. But under extreme scale, they all break down:

1. "Just use a hashmap of counters" (map[key]count)

• Memory is unbounded. Every new key (new IP, new log template) grows the map.
• No notion of time. To fix this, you have to bolt on timestamps and run a background cleanup sweep. Now your cleanup thread is racing your writer threads, which means you need locks, which kills your latency.

2. "Count in fixed windows and reset"

• The window boundary is a cliff. If you wipe the counters every minute, a key that fired 10,000 times at 11:59:59 looks completely new at 12:00:00.
• Global pauses. Wiping the data is a global mutation that fights concurrent writers.

3. "Use a probabilistic sketch (Count-Min Sketch)"

This gets us close—it gives us fixed memory and fast counts! But:

• CMS has no clock. Counts only ever grow. "Frequent lately" is not a question it can answer natively. You have to bolt on a background decay loop, which brings back the synchronization nightmares.
• It errs in the dangerous direction. Hash collisions in a CMS only ever inflate a count. If you use this for rate limiting, inflating a rare key's count means you accidentally punish a well-behaved user.

The Pattern

If you look at the standard answers, they all share the same structural challenge: Frequency and recency are treated as two separate facts requiring two separate systems.

Every standard approach ends up maintaining "time" as a separate mechanism from "counting" (via sweeps, window rotations, or halving loops). And the synchronization between the two is exactly where the memory blow-ups, the pauses, and the race conditions come from.

What's next?

I've been obsessing over this problem space recently. Is it possible to design a mechanism where recency is an intrinsic property of reading the state—requiring no background mutation, no reset pause, and no cleanup sweep—all while remaining lock-free?

I'm currently in the experimentation phase of designing a new primitive to solve this exact intersection of constraints. It's too early to share the exact mechanics before the math proves out in simulation, but I'll be posting my findings and (hopefully) some open-source code as I progress.

Have you come across such problems in your own systems? How did you solve them? Did you have to make a tough memory vs. speed tradeoff? Let me know in the comments!