DEV Community: Yehezkiel Dio Sinolungan

When Drop is Your Last Line of Defense

Yehezkiel Dio Sinolungan — Fri, 20 Feb 2026 02:23:05 +0000

Spawning a child process is trivial. Ensuring it actually dies along with its entire lineage, when your application panics, cancels a task, or drops a future is notoriously difficult.

This is the second article on a series of documenting Azalea’s architecture, a Discord bot that downloads, transcodes, and reuploads X media. The first covered building a fail-open cache with batched persistence; this one covers the darker art of subprocess management. In Azalea, external binaries are black boxes: FFmpeg for transcoding, yt-dlp for extraction (sometimes). They can hang, flood pipes with gigabytes of data, or spawn daemonized child trees that outlive their welcome.

The problem with async Rust subprocess management is that you’re playing three-dimensional chess against the operating system. The kernel is synchronous. Your runtime is cooperative. Your requirements are “kill everything immediately when I say so, but also don’t leak if I forget to say so.” These are incompatible desires, and resolving them requires accepting that your last line of defense is a synchronous destructor running on a potentially-dead async runtime.

A Problem of Temporal Mismatch

Here’s the fundamental issue: Rust’s Drop trait is synchronous. It cannot .await. It runs on whatever thread happens to be dropping the value, which might be the async runtime's worker thread, might be a spawn_blocking pool, might be the main thread during panic unwinding. You don't know. You can't know. And you certainly can't perform async I/O inside it.

Photo by Djim Loic on Unsplash

Meanwhile, your subprocess is an OS resource with its own lifecycle. You hold a tokio::process::Child handle, which is a thin wrapper around a file descriptor and some metadata. The handle is async-friendly, you can .await its completion, which internally calls waitpid or WaitForSingleObject depending on your platform. But if your task gets cancelled, if your future is dropped mid-await, that handle vanishes. The kernel doesn't know you lost your reference. The process keeps running.

Worse, some media tools can spawn child trees. yt-dlp, for example, may invoke helper subprocesses depending on extractor/path. FFmpeg is often thread-heavy rather than process-tree-heavy, but treating subprocesses as process groups is still a robust default when wrappers or helpers are involved. Kill only the parent, and surviving descendants can outlive your task.

The naive solution is to wrap everything in Arc> and use some kind of global registry with a background cleanup task. It works until it doesn't, well until you have a thousand entries in your registry, until the cleanup task itself gets cancelled, until you're holding locks across await points and deadlocking your runtime. Global state is a trap. We can do better.

RAII? I barely know her!

The solution is a guard struct that treats the subprocess as a loan from the kernel, collateralized by a cached PID. Here’s the core insight: Child::id() returns None after the process exits. The kernel reclaims the PID immediately upon termination, and Tokio reflects this by clearing the stored ID. If you rely on the handle to provide the PID during cleanup, you're racing the reaper.

So we cache it at spawn time, when success is guaranteed:

let child = command.spawn()?;
let pid = child.id().map(|id| id as i32); // Cache now, or forever hold your peace

The guard holds three things: the Child handle, the cached Option, and an armed boolean. The armed flag is the state machine. It starts true. If you successfully await the child's completion, we set armed = false. If the guard drops while armed is still true, we assume the worst like task cancellation, panic, early return, and perform synchronous cleanup.

This is the linear type pattern encoded in a boolean. armed = true means "I have not witnessed completion." armed = false means "the kernel has confirmed this process is done." There's no third state. The boolean makes the protocol explicit and costs exactly one byte, which the compiler probably packs into padding anyway.

The Drop implementation is where we accept our limitations. We cannot .await. We cannot gracefully shutdown. We issue SIGKILL to the entire process group and immediately reap the zombie with WNOHANG. It's violent. It's impolite. It doesn't give the subprocess time to flush buffers or clean up temp files. But it is guaranteed to run, guaranteed to complete, and guaranteed not to block the async runtime.

Or rather, it shouldn’t block. More on that later. In Azalea, this Drop path is a fallback safety net; timeout and output-limit paths proactively terminate subprocess groups asynchronously via kill_process_group(guard.child_mut()).await before Drop is reached.

Unix or Death

Unix process management has a taxonomy that seems designed to confuse. Sessions, process groups, foreground/background, controlling terminals. Most of this is historical baggage from the 1970s PDP-11 era that we’re still paying for. But process groups are genuinely useful for our purposes.

When you spawn a command with process_group(0), you're calling setpgid(0, 0) in the child. This creates a new process group with the child's PID as the group ID. Any children that child spawns inherit this group ID by default. Suddenly your "single" subprocess is an addressable collective.

The cleanup sequence in Drop exploits this:

killpg(pid, SIGKILL) — Terminate every process in the group simultaneously
kill(pid, SIGKILL) — Fallback for the leader, in case the group was already orphaned
waitpid(pid, WNOHANG) — Reap the zombie without blocking

The ordering matters. We try the group kill first because it’s the big hammer. We fallback to individual kill because race conditions exist, the leader might have exited between our check and our signal. We reap because un-reaped zombies accumulate and eventually exhaust the PID space.

Also, we're ignoring errors. This is deliberate. The process might already be dead. We might not have permission to signal it. The PID might have been recycled (though our short window makes this unlikely). In a destructor, you cannot propagate failure. You cannot retry. You cannot allocate. You attempt the cleanup, absorb any failure, and proceed. Panicking in Drop aborts the entire program, which is almost always worse than a leaked process.

The Windows path is simpler because Windows doesn’t have process groups in the Unix sense. Job objects would be the “correct” abstraction, but that’s complexity we don’t need for this use case. We call start_kill() and hope for the best.

Orphans of the Foreground

By moving your child into a new process group with process_group(0), it is no longer part of the terminal's foreground group. It will "ignore" a Ctrl-C sent to the main Rust app because the terminal driver only forwards SIGINT to the foreground process group. Your child is now in its own group, not the foreground group, so it doesn't get the signal.

This is “correct” behavior by Unix standards, which is to say it’s surprising and annoying. If you want ^C to propagate, you have to manually catch the signal in Rust and forward it to the negative PID (the process group ID). I don’t do this in Azalea because the guard will kill them in Drop anyway when tasks are cancelled during shutdown. The process group isolation just means they don’t receive the initial SIGINT, they get SIGKILL later when the guard drops.

But if you’re building an interactive tool where ^C should mean “stop everything immediately,” you’ll need signal handling.

I don’t include this in the guard itself because it requires global tracking of active process groups, which brings us back to the registry problem we were trying to avoid. Choose your poison: isolated children that might outlive your process, or global state that might deadlock your runtime.

No Time for Politeness

The problem is the “wait a grace period” part. In a synchronous destructor, you cannot wait. You cannot std:🧵:sleep because that blocks the async runtime. You cannot tokio::time::sleep because you can't .await. You could spawn a detached thread to do the escalation, but now you're managing thread lifetimes in a destructor, and if the main process exits before your thread finishes, you leak the escalation thread or leave the child running.

We send SIGKILL immediately. No grace period. No escalation. If FFmpeg was halfway through writing a frame, that frame is truncated. If yt-dlp was downloading a fragment, that fragment is partial. This is the "fail-fast" philosophy: partial output is detectable and retryable; leaked processes are not.

For Azalea specifically, this is fine. Discord uploads are atomic, either the entire file arrives or it doesn’t. A truncated transcode fails validation and gets retried. But if you’re building something where partial writes are dangerous (database compaction, log rotation), you’ll need a different architecture. Don’t put that in Drop. Use a proper supervisor with graceful shutdown timeouts in your main loop, and accept that panics will still bypass it.

Blocking Drop Handler

I said waitpid with WNOHANG doesn't block. This is mostly true. But "mostly" is doing a lot of work here.

WNOHANG returns immediately if the child hasn't exited yet. But if the child is in an uninterruptible sleep state, say, waiting on NFS or a FUSE filesystem, waitpid itself might block on kernel locks.

The fix is to not reap in Drop at all. Just send the signals, set a flag, and let a background task handle reaping. But that requires global state. Or you accept that WNOHANG is "non-blocking enough" for your workload and you monitor for edge cases.

The Leaderless Group

For this case, child process exits extremely quickly, faster than the kernel can set up the process group, it might die before setpgid completes in the child. The parent sees a successful spawn, caches the PID, but the child is already a zombie. When we call killpg, we're targeting a PGID that was never established, or the child's PID which is now a zombie not associated with any group.

The child also might have called setpgid successfully, then immediately exited, leaving the process group empty but technically existing. killpg on an empty group returns success (no processes to signal), but waitpid on the leader might fail if we've already reaped it elsewhere.

There’s a deeper issue: if the group leader dies but children survive, they become “orphaned” and reparent to init (PID 1). They’re still in the process group, but the group has no leader. killpg still works on them. the kernel doesn't require a living leader to signal a group, but some Unix variants handle this differently. (Solaris anyone?)

Defensive? Defensive!

Subprocess output is untrusted input. This seems obvious when you say it out loud, but production code could stream subprocess stdout into an unbounded Vec because "it's just a few kilobytes of JSON." Then someone points it at a malicious or merely broken program that outputs infinite newlines, and suddenly your "few kilobytes" is consuming gigabytes and the OOM killer is selecting victims.

We allocate an 8KB stack buffer and loop:

let remaining = limit.saturating_sub(data.len());
if read <= remaining {
    if let Some(slice) = buffer.get(..read) {
        data.extend_from_slice(slice);
    }
} else {
    if remaining > 0
        && let Some(slice) = buffer.get(..remaining)
    {
        data.extend_from_slice(slice);
    }
    exceeded = true;
    if let Some(tx) = notify.take() {
        // Best-effort signal to cancel the owning process.
        let _ = tx.try_send(());
    }
}

The saturating_sub is defensive against integer underflow, though data.len() should never exceed limit by construction. The slice bounds are checked. The memory footprint is provably bounded: limit + 8KB maximum, where the 8KB is the temporary read buffer.

But here’s the subtle part: once exceeded becomes true, we keep reading. We don't break the loop. We discard the data, but we keep issuing read() calls until we get EOF. Why? Because if we stop reading, the subprocess blocks on write(). It might be stuck in a write() syscall waiting for us to drain the pipe, and if we're not draining, we can't kill it cleanly. The pipe buffer fills, and the subprocess can block in write(). We still send SIGKILL immediately, which usually terminates the process promptly; however, in pathological kernel-level I/O states (for example uninterruptible sleep), teardown visibility can lag. Keeping the pipe drained is still a practical defensive measure to reduce backpressure and improve shutdown reliability.

A process could be “killed” but still in ps. The guard's Drop ran but the PID is still occupied. The solution is to keep the pipe empty so the process remains killable. We pay CPU cycles to read data we'll throw away, in order to guarantee that the process can actually die.

The notification channel is a one-shot circuit breaker. When we first exceed the limit, we take() the Optionmpsc::Sender<()> and try_send(()). The take() ensures we signal exactly once. The try_send() is non-blocking we're already in truncation mode, so we can't afford to wait for channel capacity. The parent can use this signal to tear down the entire pipeline, not just this one read.

Why Not Higher-Level Abstractions?

Could I have used tokio::process::Command directly? I did, initially. Just like with any cases, it's fine until it isn't, until you need process groups, until you need guaranteed cleanup, until you're debugging why your app has ten thousand PIDs and no apparent processes.

There’s a process pool or supervisor. Sure. Then I’d handle pool exhaustion, queue management, and the complexity of returning results from pooled workers. For Azalea’s workload, bursty, short and long-running, heterogeneous commands, a pool is overkill. We want per-task isolation and cleanup, not resource sharing.

The guard is the minimal abstraction that provides the guarantees we need. It doesn’t require global state. It doesn’t need configuration. It composes with any async code that can hold a struct until completion.

But minimal doesn’t mean simple. The guard encodes assumptions about Unix process semantics that are not portable, not guaranteed, and not even consistently documented, at least from what I can gather.

Hopes and Prayers

Defensive programming (and by extension systems programming) is the art of assuming failure. Not handling it gracefully, sometimes that’s impossible but preventing it from spreading.

I like elegant code, though I’ll grant that elegance is in the eye of the beholder. This, however, is not it. Elegant code would have async destructors and structured concurrency and graceful shutdown protocols. We don’t have those things. We have synchronous cleanup running on dying threads, sending violent signals to process groups we hope still exist, reaping zombies we hope are ours to reap.

The guard pattern encodes a protocol in the type system. The armed boolean is a promise: either we witnessed completion, or we will clean up. The PID cache is a hedge against kernel timing. The bounded read is a denial-of-service prevention mechanism. Each piece addresses a specific failure mode discovered through production pain.

But the guard is also a collection of trade-offs I’m not fully comfortable with. The terminal control issue means ^C doesn’t work intuitively. The lack of graceful shutdown means truncated files. The blocking Drop handler means potential runtime stalls. These are not bugs to be fixed. They're inherent tensions in the problem space. You can move them around, but you can't eliminate them.

It is the guarantees that hold when grace is impossible. The guard doesn’t trust you to clean up. It doesn’t trust Tokio to unwind cleanly. It doesn’t trust the subprocess to exit politely. It trusts only the kernel’s promise that SIGKILL is unblockable and waitpid will eventually reap.

Just remember: when you move a process into its own group, you become its parent, its supervisor, and its executioner. The terminal won’t help you. The runtime won’t help you. It’s just you, a cached PID, and hopes and prayers.

Fail-Open Cache with Batched Writes and Exponential Backoff

Yehezkiel Dio Sinolungan — Mon, 09 Feb 2026 22:46:47 +0000

They say caching is a weapon of last resort, most of the time, it’s a weapon of mass destruction.

Most of Discord bot development is just an elaborate exercise in managing other people’s garbage. My latest project, Azalea, involves a circus of X (formerly Twitter) media links and FFmpeg transcoding — a deep dive into video engineering I’m reasonably sure I’m not qualified for. The bot sits in Discord servers, waiting for someone to mention the bot with a link to a tweet containing video. Then it downloads that video, possibly transcodes it to meet Discord’s upload limits, and reuploads it as a native embed. Simple enough in theory. In practice, it’s a distributed systems problem masquerading as a chat bot.

I could have just slapped Redis into a Docker container and called it a day. We’re all caching everything anyway, aren’t we? Redis is the default choice, the safe bet, the thing you put on your resume. But for Azalea, I wanted something more targeted, something that lived inside the process but wouldn’t vanish if the server decided to trip over its own power cord. I wanted durability without operational complexity, consistency without network partitions, and failure modes I could reason about in a single codebase.

Photo by Saied Ashour on Unsplash

The core problem is deceptively simple. When someone mentions the bot with an X link in Discord, the bot needs to download, possibly transcode, and reupload that media. But Discord is chatty. Multiple people might post the same viral tweet within seconds. In a busy server, a single trending video might get posted ten times in a minute. Without deduplication, you’re running FFmpeg ten times in parallel, burning CPU, hammering X’s CDN, and probably hitting rate limits on all fronts.

The naive solution is a simple HashSet protected by a mutex. Check if the ID exists, if not, insert it and proceed. But this is async Rust with Tokio. We're dealing with futures, not threads, and the gap between "check" and "insert" is an await point where anything can happen. Two tasks can simultaneously see an empty cache, both decide to process the same tweet, and both spawn FFmpeg processes before either completes. This is the thundering herd: cache miss, parallel execution, resource exhaustion.

Worse, the processing itself involves multiple stages with different timeouts. Downloading a video from X might take 30 seconds if their CDN is slow. FFmpeg transcoding a 4K video to 720p might take two minutes. Uploading to Discord’s CDN might take another 30 seconds. The total window of vulnerability, the time between “we decided to process this” and “we finished processing this” is measured in minutes, not milliseconds.

I ended up with a three-tier approach:

In-flight deduplication (Moka, seconds TTL)
Processed results (Moka, hours/days TTL)
Disk persistence (redb, batched async writes)

Each layer serves a distinct purpose. The in-flight cache prevents concurrent duplicate work. The processed cache avoids re-processing recent tweets after restart. The disk persistence survives process restarts.

Racing to Claim Work

The in-flight cache uses Moka, a Rust concurrent cache library that provides async-aware APIs and automatic value coalescing. The TTL isn’t arbitrary, it’s calculated based on worst-case processing time. I take the download timeout, add the FFmpeg timeout, add the upload timeout, then double the FFmpeg component because that thing is unpredictable, then add a 60-second margin because I don’t trust computers. For typical configurations, this results in a 5–7 minute TTL.

The value stored is an Arc, which lets threads race to claim work atomically:

let marker = self
    .inflight
    .get_with(key, async { Arc::new(AtomicBool::new(false)) })
    .await;

if marker
    .compare_exchange(false, true, Ordering::SeqCst, Ordering::SeqCst)
    .is_err()
{
    return false;
}

The get_with is crucial here. Moka guarantees that concurrent calls on the same key coalesce into one initialization. Even if fifty users spam the same link simultaneously, only one creates the initial Arc. The rest receive clones of the same atomic. Then the compare_exchange acts as a turnstile: the first caller sees false, swaps it to true, and proceeds. All others see true and back off.

This pattern, coalesced initialization followed by atomic claim is a general solution for “exactly once” execution in distributed systems. The difference here is the timeframe. Most systems deal with milliseconds. We’re dealing with minutes of processing time, which makes the window of vulnerability much larger and the correctness requirements more stringent.

If the claim succeeds, the task proceeds to download and transcode. If it fails, the task waits, but not by blocking. Instead, it polls the processed cache periodically, waiting for the in-flight marker to disappear and the result to appear. This avoids holding a task slot during long-running work.

The processed cache is simpler: straightforward Moka with LRU eviction and TTL expiration. When a tweet finishes processing, we insert its ID with an empty value. The empty value is a memory optimization — we don’t need to store the result, just the fact of completion. The real result is the uploaded Discord message, which exists outside our system.

The Persistence Mess

The persistence layer is where things get messy and where I had to embrace the “fail-open” philosophy. I wanted durability across restarts without the operational burden of a separate service. Redis would require running another container, managing connections, handling network partitions. SQLite would work, but I wanted something more modern, with better Rust integration and MVCC for concurrent reads.

I reached for redb, a pure-Rust embedded key-value store. It’s ACID-compliant, uses B-trees, has MVCC for concurrent reads, and compiles to a single static library.

But redb is synchronous. It uses standard filesystem APIs that block the calling thread. In an async Rust application, blocking the executor thread is heresy! it prevents other tasks from making progress and can cascade into latency spikes across the entire system. The solution is spawn_blocking, Tokio's mechanism for running CPU-bound or blocking I/O work on a separate thread pool.

Writes are batched and asynchronous. When a tweet finishes processing, we don’t immediately hit the disk, that would murder throughput. Instead, we queue to a VecDeque protected by an async RwLock:

let should_flush = {
    let mut pending = self.pending_writes.write().await;
    pending.push_back(PendingWrite {
        key: db_key,
        timestamp,
    });
    self.enforce_pending_cap(&mut pending);
    pending.len() >= self.batch_size
};

if should_flush {
    self.flush().await;
}

The enforce_pending_cap is a safety valve. If the queue grows beyond ten times the batch size, we start dropping the oldest writes. This is heresy in some circles—you're losing data!—but consider the alternative: unbounded memory growth until the OOM killer arrives. In a media processing pipeline, stale deduplication entries are less dangerous than the process dying. The cache degrades to in-memory-only mode, which is still correct, just not durable.

In CAP theorem sense, we’re choosing availability over consistency when under pressure. The cache doesn’t become unavailable when the disk fills up; it becomes less consistent across restarts. For a deduplication cache, this is the right trade-off. We’d rather process a duplicate after restart than stop processing entirely.

Exponential Backoff and Graceful Degradation

The flush itself runs in spawn_blocking because redb is synchronous. But what happens when the disk is full? Or when redb encounters corruption? The first instinct is to propagate the error and crash, but that's the wrong move for a cache. A cache should never be the reason your application stops working.

Instead, I implemented exponential backoff:

fn compute_backoff_secs(&self, failures: usize) -> u64 {
    let exponent = failures.saturating_sub(1).min(8);
    let backoff = FLUSH_BACKOFF_BASE_SECS.saturating_mul(1u64 << exponent);
    backoff.min(FLUSH_BACKOFF_MAX_SECS)
}

After the first failure, we wait 1 second before retrying. After the second, 2 seconds. Then 4, 8, 16, up to a maximum of 300 seconds (5 minutes). This prevents tight loops of failure that spam the logs and waste CPU.

After five consecutive failures, we permanently disable persistence and log an error. The system keeps running on memory alone. This is the “fail-open” part: when the cache’s durability mechanism breaks, the cache itself doesn’t become a liability. It degrades gracefully rather than catastrophically.

Observability Without Overhead

The metrics subsystem shares this DNA. It’s also backed by redb, tracking stage durations and error counts across the pipeline. But unlike the deduplication cache, metrics are purely best-effort. We never want metrics collection to slow down media processing.

The Tracker uses atomics for hot-path updates—Ordering::Relaxed because we don't need sequential consistency for statistics, we just need them to be roughly correct eventually:

pub fn record_stage_duration(&self, stage: Stage, duration_ms: u64) {
    if !self.inner.enabled {
        return;
    }
    let idx = stage as usize;
    if let (Some(sum), Some(count)) = (
        self.inner.stage_duration_sum_ms.get(idx),
        self.inner.stage_count.get(idx),
    ) {
        sum.fetch_add(duration_ms, Ordering::Relaxed);
        count.fetch_add(1, Ordering::Relaxed);
    }
}

Note the early return if disabled. This prevents the atomic operations entirely when metrics are turned off, which matters when you’re recording thousands of events per second.

The stages are an enum mapped to array indices: Resolve, Download, Optimize, Upload. This fixed mapping keeps the hot path allocation-free. No hash maps, no dynamic dispatch, just array indexing.

The flush resets counters after persistence, which means averages are calculated over the flush interval rather than all-time. This is a deliberate trade-off; I care more about recent performance trends than historical accuracy. If the pipeline got slower after a deployment, I want to see that in the next flush, not buried under months of historical data.

Error tracking uses a DashMap for concurrent updates without locking the entire map. We cap the number of distinct error keys at 128 to prevent unbounded growth from unique error messages. When the cap is reached, new error kinds are logged but not counted.

Resolver Caching and Negative Caching

The resolver caching follows similar patterns but with different constraints. When someone posts an X link, we need to resolve it to actual media URLs. This involves API calls to VxTwitter or spawning yt-dlp, both of which are slow and rate-limited. We cache both successes and failures: positive cache for media metadata, negative cache for “this tweet doesn’t exist or is private.”

The negative cache is dangerous. You don’t want to cache a transient 503 and permanently block a valid link. So there’s logic to detect which errors are cacheable:

fn should_negative_cache(error: &ResolveError) -> bool {
    match error {
        ResolveError::HttpStatus(code) => *code == 404,
        ResolveError::ParseFailed(_) => true,
        ResolveError::ProcessFailed { stderr, .. } => {
            let lower = stderr.to_lowercase();
            if lower.contains("timed out")
                || lower.contains("timeout")
                || lower.contains("rate limit")
                || lower.contains("429")
                || lower.contains("temporar")
                || lower.contains("server")
            {
                return false;
            }
            // ... durable error detection
        }
    }
}

Notice the string matching on “temporar” — a lazy substring check that catches “temporary” and “temporarily” because yt-dlp’s error messages aren’t standardized. “Temporary server error” and “temporarily unavailable” are different strings but the same intent.

The negative cache has a shorter TTL than the positive cache — typically 5 minutes versus 24 hours. This limits the damage from a false positive (caching a transient error as permanent) while still protecting against repeated expensive lookups of actually-deleted tweets.

The resolver chain also implements fallback logic. We try VxTwitter first because it’s fast and lightweight. If that fails with a potentially-transient error, we fall back to yt-dlp, which is slower but more robust. Only if both fail do we consider caching the negative result. This creates a hierarchy of reliability: fast path, slow path, cached rejection.

Rate Limiting with Moka

The rate limiter uses Moka differently, as a TTL-based counter rather than a key-value store. Each user ID maps to an Arc, and we increment with relaxed ordering because exact precision isn't worth the synchronization cost:

pub async fn check<Marker>(&self, id: Id<Marker>) -> bool {
    if self.max_requests == 0 {
        return true;
    }

    let user_key = id.get();
    let counter = self
        .cache
        .get_with(user_key, async { Arc::new(AtomicU32::new(0)) })
        .await;
    let current = counter.fetch_add(1, Ordering::Relaxed);
    current < self.max_requests
}

When the count exceeds the threshold, requests are rejected. The TTL provides the windowing automatically; when a user’s entry expires, they get a fresh counter. It’s approximate, Moka’s TTL isn’t millisecond-precise but for “30 requests per minute,” being off by a few seconds is acceptable.

This is a fixed-window rate limiter, which has known issues with burst traffic at window boundaries. A user could make 30 requests at 11:59:59 and another 30 at 12:00:00, effectively doubling their allowed rate. A sliding window implementation would be more accurate but would require storing timestamps for every request, not just counts. For Discord bot usage patterns, the fixed window is sufficient and much more memory-efficient.

Why Not Redis?

Could I have used Redis? Sure. Then I’d handle connection failures, cluster topology changes, serialization overhead, and the operational burden of another service. Redis is fast, but it’s another moving part. It can fail independently of your application. It requires network calls that add latency to every operation.

With this architecture, the cache is a library dependency. It compiles into the binary. Works the same in development and production. And when things go wrong as they certainly will, it degrades gracefully rather than catastrophically.

Photo by GOETZ Jean-Pierre on Unsplash

The trade-off here is scale. This architecture works for a single process on a single machine. If I needed to run multiple Azalea instances behind a load balancer, I’d need external coordination for deduplication. But for a Discord bot, vertical scaling goes surprisingly far. A single machine can handle thousands of concurrent downloads. By the time you need horizontal scaling, you’ve probably outgrown Discord’s API limits anyway.

The Philosophy of Fail-Open

This architecture embodies a specific philosophy: components should fail in the direction of reduced functionality, not total failure. When the persistence layer breaks, we don’t crash; we degrade to in-memory caching. When the cache is disabled, we don’t refuse to process media; we just process everything (and potentially duplicate work). When rate limiting is misconfigured, we default to allowing requests rather than blocking everything.

This is the opposite of “fail-safe” or “fail-closed” systems, which stop operating when they detect anomalies. Fail-closed is appropriate for security systems, if you can’t verify a cryptographic signature, you shouldn’t proceed. But for a cache, which is purely an optimization, fail-closed is inappropriate. The cache is not the product; the product is media processing. The cache exists to make it faster and cheaper. If the cache becomes a liability, discard it.

This philosophy extends to the operational design. There are no critical alerts for cache flush failures. They get logged at WARN level, and after five failures they become ERROR when persistence is disabled. But the service keeps running. On-call doesn’t get paged because a disk filled up.

The degradation is visible in metrics, cache hit rate drops to zero after restart, memory usage climbs as entries accumulate, but the user-facing functionality continues.

Precision Dissection of Git Diffs for LLM Consumption

Yehezkiel Dio Sinolungan — Sat, 31 Jan 2026 23:44:05 +0000

Lately, I’ve been deep in the weeds of building developer tooling and workflow optimizations. I’ve realized that the tools I build don’t necessarily need to be beneficial for the entire world; they just need to solve a specific friction point for me. This isn’t solipsism exactly, it’s a recognition that scratching your own itch often produces better abstractions than attempting to solve everyone’s problems simultaneously.

Photo by Annie Spratt on Unsplash

General-purpose tools dilute their opinionated effectiveness in exchange for flexibility; they become configuration engines rather than workflow accelerators. By narrowing scope to “works for my brain,” I can make aggressive assumptions about usage patterns, error recovery, and interface design that would be unacceptable in enterprise software.

I’ve been working on Christina, a Git commit message generator, because I wasn’t satisfied with the existing tools. I’d been using OpenCommit for a while, eventually forking it to implement nuanced changes. However, forking a project only takes you so far when you have a fundamentally different vision for performance.

Here’s the thing about Git diffs: they can be absolutely massive. A single dependency update can balloon your package-lock.json to 50,000 lines. A database migration might touch hundreds of files across schema definitions, generated ORM code, and migration metadata. And LLMs? LLMs have finite, expensive context windows. Past a certain point, adding more tokens doesn’t just waste money, it degrades output quality. The model can’t effectively prioritize signal over noise when both are drowning in a sea of lockfile churn.

You can’t just throw a 100KB diff at Gemini 3 Flash and hope for the best. Well, you can, but you’ll burn through your token budget faster than you can say “rate limit exceeded,” and you’ll likely get a vague summary that misses the architectural significance of the changes because the model’s attention mechanism got distracted by repetitive lockfile churn.

The challenge isn’t just about fitting things into a context window. It’s about maximizing signal while minimizing noise, which requires understanding the information entropy of different diff segments. The LLM doesn’t need to see every deleted line from that minified JavaScript bundle you’re removing. It doesn’t need the full contents of Cargo.lock, which is essentially a serialized dependency graph with high redundancy. It needs just enough context to understand what changed and why, enough to reconstruct the semantic intent without drowning in syntactic noise.

This is fundamentally a compression problem. Git diffs are already a form of delta compression, but they’re optimized for storage and patch application, not for semantic summarization by a neural network. We need a secondary compression layer that respects semantic boundaries, an understanding that package.json and package-lock.json carry different semantic weight per line, that test file changes often mirror implementation changes and can be summarized by reference rather than repetition.

The Recursive Dissection Strategy

The core idea is simple: break down a massive Git diff without losing the semantic context the LLM needs to understand the change. But “simple” in theory gets complicated fast in practice because semantic boundaries don’t align with byte boundaries. A function definition might span multiple hunks. A refactor might touch the signature in one file and the call sites in twenty others. The strategy needs to be lossy in terms of information volume but lossless in terms of semantic connectivity.

Level 1: The Greedy File Packer

The engine starts by trying to pack entire files into chunks using a First-Fit algorithm. This isn’t optimal from a theoretical computer science perspective, we could use bin-packing algorithms like Best-Fit Decreasing for better space utilization, achieving packing efficiency, but greedy is O(N) and bin-packing is O(N log N) at best. More importantly, greedy packing maintains file order, which matters semantically in ways that pure algorithmic efficiency ignores.

If you’re changing auth.ts, auth.test.ts, and auth.types.ts, you want them in the same chunk or consecutive chunks. The LLM can infer relationships between related files through proximity in context, this is the "locality of reference" principle applied to token windows. Optimal bin packing might scatter them across chunks to maximize packing density, destroying that contextual adjacency and forcing the model to reconstruct relationships without adjacency cues.

// If this single file fits in our budget
if combined_tokens <= token_limit.get() {
    // Add to current chunk (truncate if lockfile)
    if is_lockfile
        && file_diff.token_count > TokenCount::new_saturating(LOCKFILE_TOKEN_LIMIT)
    {
        let truncated = truncate_to_token_limit(
            &file_diff.content,
            TokenCount::new_saturating(LOCKFILE_TOKEN_LIMIT),
            tokenizer,
        );
        buffer.content_mut().push_str(&truncated);
        buffer
            .content_mut()
            .push_str("\n[... truncated lockfile ...]\n");
    } else {
        buffer.content_mut().push_str(&file_diff.content);
    }
    buffer.file_paths_mut().push(file_diff.path);
    current_tokens = TokenCount::new(combined_tokens);
} else {
    // Flush current chunk and start new one
    if !buffer.is_empty() {
        chunks.push(DiffChunk::new(
            Arc::from(buffer.take_content()),
            buffer.take_file_paths(),
            current_tokens.unwrap_or_else(|| TokenCount::new_saturating(1)),
        ));
        buffer.clear();
    }

    // ...
}

This is where we maintain maximum context: the LLM sees complete file headers and all changes within that file together. The file boundary acts as a natural semantic firewall, we assume files are generally cohesive units (modulo some exceptions like generated code), so keeping them intact preserves intra-file relationships like variable scoping and import dependencies.

Level 2: The Hunk-Level Splitter

When a single file’s diff exceeds the token limit, we drop down to splitting by hunks, those @@ -start,count +start,count @@ markers that Git uses to denote changed sections within a file.

This is a natural semantic boundary because hunks represent logically contiguous changes within a file, typically scoped to a function or a cohesive block of code. Git’s diff algorithm (the Myers diff or histogram diff depending on your configuration) already does the hard work of segmenting changes into minimal edit scripts. Splitting here preserves local context: the LLM still sees what function or section of code changed, even if it can’t see the entire file.

The tricky bit is handling the file header metadata. If you’re splitting a diff by hunks, each fragment needs the original file header (diff --git a/file.rs b/file.rs, index lines, mode changes) so the LLM knows what file it's looking at. Without this, you just have anonymous hunks floating in context space. But you only tokenize the header once per file during the fitting calculation, not once per chunk, because that would be wasteful and artificially deflate your capacity.

Another subtlety: hunks include context lines (the lines that haven’t changed but surround the changes). When splitting by hunks, you need to decide whether to duplicate context lines at the boundaries of chunks or truncate them. Christina duplicates up to 3 lines of context at chunk boundaries. This creates intentional overlap, redundancy that costs tokens but prevents semantic breakage when a change logically spans the boundary between two hunks. Without this overlap, renaming a variable might appear in one chunk as the removal and in another as the addition, looking like unrelated changes.

// Find the file header end (up to first hunk or end)
let header_end = content.find("\n@@").unwrap_or(content.len());
let header = &content[..header_end];
let header_tokens = tokenizer.count_tokens(header);
let header_tokens_count = header_tokens.get();

// Check if adding this hunk would exceed limit
if current_tokens + hunk_tokens_count > token_limit.get() {
    if !buffer.is_empty() {
        chunks.push(DiffChunk::new(
            Arc::from(buffer.take_content()),
            vec![file_path.clone()],
            TokenCount::new_saturating(current_tokens),
        ));
        buffer.clear();
    }
    // Start new chunk with header + hunk
    buffer.content_mut().push_str(header);
    buffer.content_mut().push('\n');
    buffer.content_mut().push_str(hunk);
    current_tokens = header_tokens_count + hunk_tokens_count + 1;
} else {
    buffer.content_mut().push('\n');
    buffer.content_mut().push_str(hunk);
    current_tokens += hunk_tokens_count + 1;
}

Level 3: Smart Line Slicing

It happens rarely, but it happens, you encounter a single hunk that’s too large. Maybe someone committed a minified JSON blob with one line containing 10,000 characters, or there’s a giant SQL migration that inserts thousands of records in a single multi-row INSERT statement, or a base64-encoded file that someone really shouldn’t have committed but did.

At this point, you fall back to line-by-line splitting. This breaks semantic units, which is unfortunate because a single line of code might contain multiple statements or a complex expression. But it’s better than failing entirely or dropping the change on the floor.

There’s a heuristic here about which lines to prioritize if you can only fit a subset. Christina prioritizes:

Lines starting with + (additions) over - (deletions) because the new state is usually more relevant than the old
Lines containing keywords like TODO, FIXME, hack, bug through simple regex matching, on the theory that these indicate high-intent changes
The first and last N lines of the hunk, under the assumption that changes usually have primacy and recency bias in importance

This is a form of semantic triage, acknowledging that when we must lose information, we should lose the least important information first.

Level 4: Binary Search for Byte-Level Precision

And then there’s the truly pathological case: a single line that exceeds your token budget.

BPE (Byte Pair Encoding) tokenizers don’t have a fixed character-to-token ratio. A line of common English prose might compress to roughly 1 token per 4 characters due to frequent subword units in the vocabulary. A line of random hex strings or minified code might approach 1 token per 1–2 characters because the tokenizer can’t find efficient subword compressions for high-entropy strings. You can’t just slice at the character midpoint and expect the token count to halve.

So we binary search for the longest UTF-8-safe slice that fits:

while low <= high {
  let mid = (low + high) / 2;

  // Ensure mid is at a UTF-8 character boundary
  let mut adjusted_mid = mid;
  while adjusted_mid > start && !line.is_char_boundary(adjusted_mid) {
      adjusted_mid -= 1;
  }

  // Guard against zero progress
  if adjusted_mid == start {
      // Take at least one character
      adjusted_mid = start + 1;
      while adjusted_mid < line.len() && !line.is_char_boundary(adjusted_mid) {
          adjusted_mid += 1;
      }
  }

  let slice = &line[start..adjusted_mid];
  let tokens = tokenizer.count_tokens(slice);

  if tokens <= token_limit {
      best = adjusted_mid;
      low = mid + 1;
  } else {
      high = mid - 1;
  }
}

This handles multi-byte UTF-8 (emoji, CJK characters, mathematical symbols) by checking is_char_boundary() at every potential split point. The guard clause ensures we always make progress, at minimum, we advance by one complete UTF-8 character, which prevents infinite loops that could occur if we naively adjusted without the decrement guarantee.

The binary search is O(log N) in the line length, with each iteration requiring a tokenizer call. For o200k_base tokenizer, this is relatively fast, but with local models using unoptimized tokenizers, this could be a bottleneck.

There’s also the question of where to split within a line. Christina prefers to split at word boundaries (spaces) when possible, searching backward from the binary-search-determined midpoint to find the nearest space. If no space exists within a 10-character window, it splits at the byte boundary and inserts a [split] marker to indicate discontinuity. This preserves readability better than mid-word truncation.

Overengineered Performance with Buffer Pooling

In Rust, high-frequency string manipulation can lead to significant allocation overhead when using the default allocator. Processing a large diff involves creating thousands of intermediate strings, temporary buffers for truncated content, concatenated hunks, file paths collections. If you allocate and drop these “scratchpads” constantly, you get memory fragmentation, allocator lock contention in multi-threaded contexts, and cache pollution from zeroing memory.

The solution is a thread-local buffer pool using an object pool pattern:

thread_local! {
    static BUFFER_POOL: RefCell<Vec<ChunkBuffer>> = const { RefCell::new(Vec::new()) };
}

pub(crate) fn acquire_buffer() -> ChunkBuffer {
    BUFFER_POOL.with(|pool| {
        let mut pool = pool.borrow_mut();
        match pool.pop() {
            Some(mut buffer) => {
                buffer.clear();
                buffer
            }
            None => ChunkBuffer::new(),
        }
    })
}

Why thread-local? Because diff chunking is CPU-bound and runs on a threadpool. Thread-local storage (TLS) avoids mutex overhead or atomic operations , there’s no cross-thread contention because each thread has its own pool, leveraging the fact that Rust’s thread model gives us true OS threads with separate stacks and TLS segments.

Each buffer is pre-allocated to 4KB, which is a typical chunk size and aligns well with memory page boundaries on most architectures (though we don’t explicitly align, the allocator typically rounds up). When you return a buffer to the pool via release_buffer(), clear() resets the length to zero but keeps the allocated capacity instead of deallocating. This prevents the allocator from returning the memory to the global heap and potentially unmapping pages.

The pool is capped at 16 buffers per thread (16 × 4KB = 64KB max overhead), which prevents unbounded memory growth in long-running processes. In pathological cases with extremely bursty traffic, we simply drop excess buffers on the floor rather than hoarding memory. This is a classic memory-time tradeoff: we trade 64KB of resident memory per thread for avoiding thousands of allocations per commit.

There’s also the consideration of allocator choice. If you’re using a modern allocator like mimalloc or jemalloc, the benefit of pooling diminishes because these allocators have thread caches and size-class binning that make small allocations cheap. However, on Windows with the default system allocator or in constrained environments, the buffer pool provides consistent performance characteristics across platforms, insulating us from allocator behavior differences.

Heuristics and Noise Reduction

Not all diff content is created equal. Information theory tells us that highly predictable content (like lockfile updates) carries low entropy and thus low information value per byte. The LLM doesn’t need to see the specific contents to understand “updated dependencies”, it just needs confirmation that this is indeed a routine lockfile update, not a malicious injection or a hand-edited dependency resolution.

Lockfiles (package-lock.json, Cargo.lock, yarn.lock, go.sum) are auto-generated noise with high internal redundancy. The LLM doesn't need to see 10,000 lines of dependency version updates to generate "update dependencies" as a commit message. Christina truncates lockfiles to 100 tokens, about 25 lines which is enough to show intent (the file header plus a sample of the changes) without wasting the context budget.

pub const LOCKFILE_TOKEN_LIMIT: u32 = 100;

// Add new file to fresh buffer
if is_lockfile
    && file_diff.token_count > TokenCount::new_saturating(LOCKFILE_TOKEN_LIMIT)
{
    let mut truncated = truncate_to_token_limit(
        &file_diff.content,
        TokenCount::new_saturating(LOCKFILE_TOKEN_LIMIT),
        tokenizer,
    );
    truncated.push_str("\n[... truncated lockfile ...]\n");
    buffer.content_mut().push_str(&truncated);
} else {
    buffer.content_mut().push_str(&file_diff.content);
}

The 100-token limit isn’t arbitrary , it’s based on the observation that most lockfile changes follow a power-law distribution: a few dependencies change significantly, most change by version bump only. 25 lines typically captures the “interesting” changes (major version bumps, new dependencies) while excluding the long tail of patch updates.

Similarly, deletion-only diffs get special treatment. If you’re deleting entire files, the LLM just needs to know what was deleted, not the full contents of the corpse. Christina detects these cases and heavily truncates, showing only the file paths and the first few lines (in case there’s a header comment explaining what the file was):

// Truncate deletion-only diffs to save tokens
// The LLM doesn't need to see all deleted content to generate "delete file" messages
if parsing::is_all_file_deletions(diff) {
    // All files are being deleted - heavily truncate
    return self.process_owned(parsing::truncate_deletion_diff(diff, 3));
} else if parsing::is_deletion_only(diff) {
    // Only deletions (no additions) - moderately truncate
    return self.process_owned(parsing::truncate_deletion_diff(diff, 10));
}

The distinction between all_file_deletions and deletion_only matters. When deleting entire files (a "remove dead code" commit), the content is irrelevant. When deleting content within a file (removing a function but keeping the file), some context helps the LLM understand what functionality was removed without seeing the full implementation.

Binary assets are replaced with [Binary file: path.png], which is still a sensible default for review because raw bytes aren’t human-readable. If you’re using a vision-capable model, though, it can interpret the actual image content meaningfully. A practical approach is to attach the relevant assets when visual changes matter, rather than trying to inline base64 blobs in the patch, but that’s complexity we don’t need yet.

Photo by Tim Gouw on Unsplash

The exact thresholds are provisional. Why 100 tokens for lockfiles? Why 3 lines for deletion previews? Not because those values are special, but because they create a measurable ceiling on noise while still exposing recognizable structure.

These constants exist to make the system observable. Once commit quality, latency, and token usage are measured in real workflows, they become tuning parameters rather than opinions. Until then, they’re deliberately conservative defaults that keep the system predictable.

Parsing as Attack Surface

One thing that’s easy to overlook in developer tooling: diff parsing is a potential attack vector. Git diffs are text with conventions, not a strict format. Treat them like structured data at your peril.

Malicious actors can craft diffs with fake headers embedded in file content to corrupt parsing or cause the tool to hallucinate changes that don’t exist. For example:

diff --git a/real.txt b/real.txt
+some code
+more code with diff --git a/fake.txt b/fake.txt embedded

If your parser uses contains("diff --git") rather than anchored matching, it might treat that embedded string as a new file boundary, causing the chunker to split incorrectly and potentially miss the malicious payload or misattribute changes to the wrong file. This is analogous to HTTP header injection or CSV injection attacks, context-sensitive parsing that fails to distinguish between metacharacters and data.

The parser needs to only treat diff --git as a header when it appears at line start:

for line in diff.lines() {
    if line.starts_with("diff --git ")
        && let Some(path) = parse_git_diff_header(line)
    {
        paths.push(path);
    }
}

But anchors aren’t enough. You also need to handle the case of newlines in filenames (yes, Git supports_files with newlines in names, though it’s pathological). diff --git line format is formally diff --git a/ b/ where the paths are escaped if they contain unusual characters. Our parser strips the a/ and b/ prefixes but needs to handle tab characters (used as delimiters in some diff formats) and the "no newline at end of file" markers that appear as \ No newline at end of file on its own line.

There’s also the question of Unicode normalization. macOS uses NFD (decomposed) form for filenames, while Linux typically uses NFC (composed). A diff generated on macOS might have decomposed UTF-8 in the header, while the filesystem expects composed. If we treat these as different strings, we might fail to correlate diff chunks with working tree files. Christina NFC-normalizes all paths internally, accepting the slight performance cost for consistency.

It’s a small detail, but security often is. The difference between contains("diff --git") and starts_with("diff --git") is the difference between a working tool and a security vulnerability that could be exploited to hide malicious code in generated commit messages or cause the tool to emit misleading metadata.

A Good Enough Engine (Maybe?)

So, does it work? Honestly? I think it might. I’ve run it against synthetic diffs, single-line typo fixes to medium refactoring across a handful of files. It hasn’t crashed on pathological inputs: null bytes, mixed encodings. The commit messages look decent: occasionally inspired, usually plausible, rarely obviously wrong.

When you’re building tools for yourself, ‘good enough’ isn’t a benchmark — it’s a feeling. Does the tool disappear into your workflow? I’m not there yet. Right now I’m optimizing for ‘does this function correctly’ rather than that flow state where you’re just thinking about code, not the commit message you’re writing.

At the time of writing, Christina is still being built. I’m documenting ideas as much as reporting results.

The chunking strategy should preserve context across architectural layers, but that’s theoretical. The buffer pooling should keep it responsive, but I haven’t measured latency under real load. Everything about scale is speculative. Production-grade? Not by my definition, reliable daily driver that doesn’t lose work or misrepresent changes. I’m nowhere near confident yet. It’s a promising prototype that solves a specific friction point in theory.

There’s a temptation to polish before publishing, to wait until everything is measured and proven. But the interesting decisions are visible mid-process, when you’re still uncertain. Once a tool disappears into your workflow, you forget why you chose these tradeoffs. I want to remember, and maybe these choices are useful to someone else wrestling with the same friction