Fail-Open Cache with Batched Writes and Exponential Backoff

They say caching is a weapon of last resort, most of the time, it’s a weapon of mass destruction.

Most of Discord bot development is just an elaborate exercise in managing other people’s garbage. My latest project, Azalea, involves a circus of X (formerly Twitter) media links and FFmpeg transcoding — a deep dive into video engineering I’m reasonably sure I’m not qualified for. The bot sits in Discord servers, waiting for someone to mention the bot with a link to a tweet containing video. Then it downloads that video, possibly transcodes it to meet Discord’s upload limits, and reuploads it as a native embed. Simple enough in theory. In practice, it’s a distributed systems problem masquerading as a chat bot.

I could have just slapped Redis into a Docker container and called it a day. We’re all caching everything anyway, aren’t we? Redis is the default choice, the safe bet, the thing you put on your resume. But for Azalea, I wanted something more targeted, something that lived inside the process but wouldn’t vanish if the server decided to trip over its own power cord. I wanted durability without operational complexity, consistency without network partitions, and failure modes I could reason about in a single codebase.

Photo by Saied Ashour on Unsplash

The core problem is deceptively simple. When someone mentions the bot with an X link in Discord, the bot needs to download, possibly transcode, and reupload that media. But Discord is chatty. Multiple people might post the same viral tweet within seconds. In a busy server, a single trending video might get posted ten times in a minute. Without deduplication, you’re running FFmpeg ten times in parallel, burning CPU, hammering X’s CDN, and probably hitting rate limits on all fronts.

The naive solution is a simple HashSet protected by a mutex. Check if the ID exists, if not, insert it and proceed. But this is async Rust with Tokio. We're dealing with futures, not threads, and the gap between "check" and "insert" is an await point where anything can happen. Two tasks can simultaneously see an empty cache, both decide to process the same tweet, and both spawn FFmpeg processes before either completes. This is the thundering herd: cache miss, parallel execution, resource exhaustion.

Worse, the processing itself involves multiple stages with different timeouts. Downloading a video from X might take 30 seconds if their CDN is slow. FFmpeg transcoding a 4K video to 720p might take two minutes. Uploading to Discord’s CDN might take another 30 seconds. The total window of vulnerability, the time between “we decided to process this” and “we finished processing this” is measured in minutes, not milliseconds.

I ended up with a three-tier approach:

1. In-flight deduplication (Moka, seconds TTL)

2. Processed results (Moka, hours/days TTL)

3. Disk persistence (redb, batched async writes)

Each layer serves a distinct purpose. The in-flight cache prevents concurrent duplicate work. The processed cache avoids re-processing recent tweets after restart. The disk persistence survives process restarts.

Racing to Claim Work

The in-flight cache uses Moka, a Rust concurrent cache library that provides async-aware APIs and automatic value coalescing. The TTL isn’t arbitrary, it’s calculated based on worst-case processing time. I take the download timeout, add the FFmpeg timeout, add the upload timeout, then double the FFmpeg component because that thing is unpredictable, then add a 60-second margin because I don’t trust computers. For typical configurations, this results in a 5–7 minute TTL.

The value stored is an Arc, which lets threads race to claim work atomically:

let marker = self
    .inflight
    .get_with(key, async { Arc::new(AtomicBool::new(false)) })
    .await;

if marker
    .compare_exchange(false, true, Ordering::SeqCst, Ordering::SeqCst)
    .is_err()
{
    return false;
}

The get_with is crucial here. Moka guarantees that concurrent calls on the same key coalesce into one initialization. Even if fifty users spam the same link simultaneously, only one creates the initial Arc. The rest receive clones of the same atomic. Then the compare_exchange acts as a turnstile: the first caller sees false, swaps it to true, and proceeds. All others see true and back off.

This pattern, coalesced initialization followed by atomic claim is a general solution for “exactly once” execution in distributed systems. The difference here is the timeframe. Most systems deal with milliseconds. We’re dealing with minutes of processing time, which makes the window of vulnerability much larger and the correctness requirements more stringent.

If the claim succeeds, the task proceeds to download and transcode. If it fails, the task waits, but not by blocking. Instead, it polls the processed cache periodically, waiting for the in-flight marker to disappear and the result to appear. This avoids holding a task slot during long-running work.

The processed cache is simpler: straightforward Moka with LRU eviction and TTL expiration. When a tweet finishes processing, we insert its ID with an empty value. The empty value is a memory optimization — we don’t need to store the result, just the fact of completion. The real result is the uploaded Discord message, which exists outside our system.

The Persistence Mess

The persistence layer is where things get messy and where I had to embrace the “fail-open” philosophy. I wanted durability across restarts without the operational burden of a separate service. Redis would require running another container, managing connections, handling network partitions. SQLite would work, but I wanted something more modern, with better Rust integration and MVCC for concurrent reads.

I reached for redb, a pure-Rust embedded key-value store. It’s ACID-compliant, uses B-trees, has MVCC for concurrent reads, and compiles to a single static library.

But redb is synchronous. It uses standard filesystem APIs that block the calling thread. In an async Rust application, blocking the executor thread is heresy! it prevents other tasks from making progress and can cascade into latency spikes across the entire system. The solution is spawn_blocking, Tokio's mechanism for running CPU-bound or blocking I/O work on a separate thread pool.

Writes are batched and asynchronous. When a tweet finishes processing, we don’t immediately hit the disk, that would murder throughput. Instead, we queue to a VecDeque protected by an async RwLock:

let should_flush = {
    let mut pending = self.pending_writes.write().await;
    pending.push_back(PendingWrite {
        key: db_key,
        timestamp,
    });
    self.enforce_pending_cap(&mut pending);
    pending.len() >= self.batch_size
};

if should_flush {
    self.flush().await;
}

The enforce_pending_cap is a safety valve. If the queue grows beyond ten times the batch size, we start dropping the oldest writes. This is heresy in some circles—you're losing data!—but consider the alternative: unbounded memory growth until the OOM killer arrives. In a media processing pipeline, stale deduplication entries are less dangerous than the process dying. The cache degrades to in-memory-only mode, which is still correct, just not durable.

In CAP theorem sense, we’re choosing availability over consistency when under pressure. The cache doesn’t become unavailable when the disk fills up; it becomes less consistent across restarts. For a deduplication cache, this is the right trade-off. We’d rather process a duplicate after restart than stop processing entirely.

Exponential Backoff and Graceful Degradation

The flush itself runs in spawn_blocking because redb is synchronous. But what happens when the disk is full? Or when redb encounters corruption? The first instinct is to propagate the error and crash, but that's the wrong move for a cache. A cache should never be the reason your application stops working.

Instead, I implemented exponential backoff:

fn compute_backoff_secs(&self, failures: usize) -> u64 {
    let exponent = failures.saturating_sub(1).min(8);
    let backoff = FLUSH_BACKOFF_BASE_SECS.saturating_mul(1u64 << exponent);
    backoff.min(FLUSH_BACKOFF_MAX_SECS)
}

After the first failure, we wait 1 second before retrying. After the second, 2 seconds. Then 4, 8, 16, up to a maximum of 300 seconds (5 minutes). This prevents tight loops of failure that spam the logs and waste CPU.

After five consecutive failures, we permanently disable persistence and log an error. The system keeps running on memory alone. This is the “fail-open” part: when the cache’s durability mechanism breaks, the cache itself doesn’t become a liability. It degrades gracefully rather than catastrophically.

Observability Without Overhead

The metrics subsystem shares this DNA. It’s also backed by redb, tracking stage durations and error counts across the pipeline. But unlike the deduplication cache, metrics are purely best-effort. We never want metrics collection to slow down media processing.

The Tracker uses atomics for hot-path updates—Ordering::Relaxed because we don't need sequential consistency for statistics, we just need them to be roughly correct eventually:

pub fn record_stage_duration(&self, stage: Stage, duration_ms: u64) {
    if !self.inner.enabled {
        return;
    }
    let idx = stage as usize;
    if let (Some(sum), Some(count)) = (
        self.inner.stage_duration_sum_ms.get(idx),
        self.inner.stage_count.get(idx),
    ) {
        sum.fetch_add(duration_ms, Ordering::Relaxed);
        count.fetch_add(1, Ordering::Relaxed);
    }
}

Note the early return if disabled. This prevents the atomic operations entirely when metrics are turned off, which matters when you’re recording thousands of events per second.

The stages are an enum mapped to array indices: Resolve, Download, Optimize, Upload. This fixed mapping keeps the hot path allocation-free. No hash maps, no dynamic dispatch, just array indexing.

The flush resets counters after persistence, which means averages are calculated over the flush interval rather than all-time. This is a deliberate trade-off; I care more about recent performance trends than historical accuracy. If the pipeline got slower after a deployment, I want to see that in the next flush, not buried under months of historical data.

Error tracking uses a DashMap for concurrent updates without locking the entire map. We cap the number of distinct error keys at 128 to prevent unbounded growth from unique error messages. When the cap is reached, new error kinds are logged but not counted.

Resolver Caching and Negative Caching

The resolver caching follows similar patterns but with different constraints. When someone posts an X link, we need to resolve it to actual media URLs. This involves API calls to VxTwitter or spawning yt-dlp, both of which are slow and rate-limited. We cache both successes and failures: positive cache for media metadata, negative cache for “this tweet doesn’t exist or is private.”

The negative cache is dangerous. You don’t want to cache a transient 503 and permanently block a valid link. So there’s logic to detect which errors are cacheable:

fn should_negative_cache(error: &ResolveError) -> bool {
    match error {
        ResolveError::HttpStatus(code) => *code == 404,
        ResolveError::ParseFailed(_) => true,
        ResolveError::ProcessFailed { stderr, .. } => {
            let lower = stderr.to_lowercase();
            if lower.contains("timed out")
                || lower.contains("timeout")
                || lower.contains("rate limit")
                || lower.contains("429")
                || lower.contains("temporar")
                || lower.contains("server")
            {
                return false;
            }
            // ... durable error detection
        }
    }
}

Notice the string matching on “temporar” — a lazy substring check that catches “temporary” and “temporarily” because yt-dlp’s error messages aren’t standardized. “Temporary server error” and “temporarily unavailable” are different strings but the same intent.

The negative cache has a shorter TTL than the positive cache — typically 5 minutes versus 24 hours. This limits the damage from a false positive (caching a transient error as permanent) while still protecting against repeated expensive lookups of actually-deleted tweets.

The resolver chain also implements fallback logic. We try VxTwitter first because it’s fast and lightweight. If that fails with a potentially-transient error, we fall back to yt-dlp, which is slower but more robust. Only if both fail do we consider caching the negative result. This creates a hierarchy of reliability: fast path, slow path, cached rejection.

Rate Limiting with Moka

The rate limiter uses Moka differently, as a TTL-based counter rather than a key-value store. Each user ID maps to an Arc, and we increment with relaxed ordering because exact precision isn't worth the synchronization cost:

pub async fn check<Marker>(&self, id: Id<Marker>) -> bool {
    if self.max_requests == 0 {
        return true;
    }

    let user_key = id.get();
    let counter = self
        .cache
        .get_with(user_key, async { Arc::new(AtomicU32::new(0)) })
        .await;
    let current = counter.fetch_add(1, Ordering::Relaxed);
    current < self.max_requests
}

When the count exceeds the threshold, requests are rejected. The TTL provides the windowing automatically; when a user’s entry expires, they get a fresh counter. It’s approximate, Moka’s TTL isn’t millisecond-precise but for “30 requests per minute,” being off by a few seconds is acceptable.

This is a fixed-window rate limiter, which has known issues with burst traffic at window boundaries. A user could make 30 requests at 11:59:59 and another 30 at 12:00:00, effectively doubling their allowed rate. A sliding window implementation would be more accurate but would require storing timestamps for every request, not just counts. For Discord bot usage patterns, the fixed window is sufficient and much more memory-efficient.

Why Not Redis?

Could I have used Redis? Sure. Then I’d handle connection failures, cluster topology changes, serialization overhead, and the operational burden of another service. Redis is fast, but it’s another moving part. It can fail independently of your application. It requires network calls that add latency to every operation.

With this architecture, the cache is a library dependency. It compiles into the binary. Works the same in development and production. And when things go wrong as they certainly will, it degrades gracefully rather than catastrophically.

Photo by GOETZ Jean-Pierre on Unsplash

The trade-off here is scale. This architecture works for a single process on a single machine. If I needed to run multiple Azalea instances behind a load balancer, I’d need external coordination for deduplication. But for a Discord bot, vertical scaling goes surprisingly far. A single machine can handle thousands of concurrent downloads. By the time you need horizontal scaling, you’ve probably outgrown Discord’s API limits anyway.

The Philosophy of Fail-Open

This architecture embodies a specific philosophy: components should fail in the direction of reduced functionality, not total failure. When the persistence layer breaks, we don’t crash; we degrade to in-memory caching. When the cache is disabled, we don’t refuse to process media; we just process everything (and potentially duplicate work). When rate limiting is misconfigured, we default to allowing requests rather than blocking everything.

This is the opposite of “fail-safe” or “fail-closed” systems, which stop operating when they detect anomalies. Fail-closed is appropriate for security systems, if you can’t verify a cryptographic signature, you shouldn’t proceed. But for a cache, which is purely an optimization, fail-closed is inappropriate. The cache is not the product; the product is media processing. The cache exists to make it faster and cheaper. If the cache becomes a liability, discard it.

This philosophy extends to the operational design. There are no critical alerts for cache flush failures. They get logged at WARN level, and after five failures they become ERROR when persistence is disabled. But the service keeps running. On-call doesn’t get paged because a disk filled up.

The degradation is visible in metrics, cache hit rate drops to zero after restart, memory usage climbs as entries accumulate, but the user-facing functionality continues.