When Drop is Your Last Line of Defense

Spawning a child process is trivial. Ensuring it actually dies along with its entire lineage, when your application panics, cancels a task, or drops a future is notoriously difficult.

This is the second article on a series of documenting Azalea’s architecture, a Discord bot that downloads, transcodes, and reuploads X media. The first covered building a fail-open cache with batched persistence; this one covers the darker art of subprocess management. In Azalea, external binaries are black boxes: FFmpeg for transcoding, yt-dlp for extraction (sometimes). They can hang, flood pipes with gigabytes of data, or spawn daemonized child trees that outlive their welcome.

The problem with async Rust subprocess management is that you’re playing three-dimensional chess against the operating system. The kernel is synchronous. Your runtime is cooperative. Your requirements are “kill everything immediately when I say so, but also don’t leak if I forget to say so.” These are incompatible desires, and resolving them requires accepting that your last line of defense is a synchronous destructor running on a potentially-dead async runtime.

A Problem of Temporal Mismatch

Here’s the fundamental issue: Rust’s Drop trait is synchronous. It cannot .await. It runs on whatever thread happens to be dropping the value, which might be the async runtime's worker thread, might be a spawn_blocking pool, might be the main thread during panic unwinding. You don't know. You can't know. And you certainly can't perform async I/O inside it.

Photo by Djim Loic on Unsplash

Meanwhile, your subprocess is an OS resource with its own lifecycle. You hold a tokio::process::Child handle, which is a thin wrapper around a file descriptor and some metadata. The handle is async-friendly, you can .await its completion, which internally calls waitpid or WaitForSingleObject depending on your platform. But if your task gets cancelled, if your future is dropped mid-await, that handle vanishes. The kernel doesn't know you lost your reference. The process keeps running.

Worse, some media tools can spawn child trees. yt-dlp, for example, may invoke helper subprocesses depending on extractor/path. FFmpeg is often thread-heavy rather than process-tree-heavy, but treating subprocesses as process groups is still a robust default when wrappers or helpers are involved. Kill only the parent, and surviving descendants can outlive your task.

The naive solution is to wrap everything in Arc> and use some kind of global registry with a background cleanup task. It works until it doesn't, well until you have a thousand entries in your registry, until the cleanup task itself gets cancelled, until you're holding locks across await points and deadlocking your runtime. Global state is a trap. We can do better.

RAII? I barely know her!

The solution is a guard struct that treats the subprocess as a loan from the kernel, collateralized by a cached PID. Here’s the core insight: Child::id() returns None after the process exits. The kernel reclaims the PID immediately upon termination, and Tokio reflects this by clearing the stored ID. If you rely on the handle to provide the PID during cleanup, you're racing the reaper.

So we cache it at spawn time, when success is guaranteed:

let child = command.spawn()?;
let pid = child.id().map(|id| id as i32); // Cache now, or forever hold your peace

The guard holds three things: the Child handle, the cached Option, and an armed boolean. The armed flag is the state machine. It starts true. If you successfully await the child's completion, we set armed = false. If the guard drops while armed is still true, we assume the worst like task cancellation, panic, early return, and perform synchronous cleanup.

This is the linear type pattern encoded in a boolean. armed = true means "I have not witnessed completion." armed = false means "the kernel has confirmed this process is done." There's no third state. The boolean makes the protocol explicit and costs exactly one byte, which the compiler probably packs into padding anyway.

The Drop implementation is where we accept our limitations. We cannot .await. We cannot gracefully shutdown. We issue SIGKILL to the entire process group and immediately reap the zombie with WNOHANG. It's violent. It's impolite. It doesn't give the subprocess time to flush buffers or clean up temp files. But it is guaranteed to run, guaranteed to complete, and guaranteed not to block the async runtime.

Or rather, it shouldn’t block. More on that later. In Azalea, this Drop path is a fallback safety net; timeout and output-limit paths proactively terminate subprocess groups asynchronously via kill_process_group(guard.child_mut()).await before Drop is reached.

Unix or Death

Unix process management has a taxonomy that seems designed to confuse. Sessions, process groups, foreground/background, controlling terminals. Most of this is historical baggage from the 1970s PDP-11 era that we’re still paying for. But process groups are genuinely useful for our purposes.

When you spawn a command with process_group(0), you're calling setpgid(0, 0) in the child. This creates a new process group with the child's PID as the group ID. Any children that child spawns inherit this group ID by default. Suddenly your "single" subprocess is an addressable collective.

The cleanup sequence in Drop exploits this:

1. killpg(pid, SIGKILL) — Terminate every process in the group simultaneously
2. kill(pid, SIGKILL) — Fallback for the leader, in case the group was already orphaned
3. waitpid(pid, WNOHANG) — Reap the zombie without blocking

The ordering matters. We try the group kill first because it’s the big hammer. We fallback to individual kill because race conditions exist, the leader might have exited between our check and our signal. We reap because un-reaped zombies accumulate and eventually exhaust the PID space.

Also, we're ignoring errors. This is deliberate. The process might already be dead. We might not have permission to signal it. The PID might have been recycled (though our short window makes this unlikely). In a destructor, you cannot propagate failure. You cannot retry. You cannot allocate. You attempt the cleanup, absorb any failure, and proceed. Panicking in Drop aborts the entire program, which is almost always worse than a leaked process.

The Windows path is simpler because Windows doesn’t have process groups in the Unix sense. Job objects would be the “correct” abstraction, but that’s complexity we don’t need for this use case. We call start_kill() and hope for the best.

Orphans of the Foreground

By moving your child into a new process group with process_group(0), it is no longer part of the terminal's foreground group. It will "ignore" a Ctrl-C sent to the main Rust app because the terminal driver only forwards SIGINT to the foreground process group. Your child is now in its own group, not the foreground group, so it doesn't get the signal.

This is “correct” behavior by Unix standards, which is to say it’s surprising and annoying. If you want ^C to propagate, you have to manually catch the signal in Rust and forward it to the negative PID (the process group ID). I don’t do this in Azalea because the guard will kill them in Drop anyway when tasks are cancelled during shutdown. The process group isolation just means they don’t receive the initial SIGINT, they get SIGKILL later when the guard drops.

But if you’re building an interactive tool where ^C should mean “stop everything immediately,” you’ll need signal handling.

I don’t include this in the guard itself because it requires global tracking of active process groups, which brings us back to the registry problem we were trying to avoid. Choose your poison: isolated children that might outlive your process, or global state that might deadlock your runtime.

No Time for Politeness

The problem is the “wait a grace period” part. In a synchronous destructor, you cannot wait. You cannot std:🧵:sleep because that blocks the async runtime. You cannot tokio::time::sleep because you can't .await. You could spawn a detached thread to do the escalation, but now you're managing thread lifetimes in a destructor, and if the main process exits before your thread finishes, you leak the escalation thread or leave the child running.

We send SIGKILL immediately. No grace period. No escalation. If FFmpeg was halfway through writing a frame, that frame is truncated. If yt-dlp was downloading a fragment, that fragment is partial. This is the "fail-fast" philosophy: partial output is detectable and retryable; leaked processes are not.

For Azalea specifically, this is fine. Discord uploads are atomic, either the entire file arrives or it doesn’t. A truncated transcode fails validation and gets retried. But if you’re building something where partial writes are dangerous (database compaction, log rotation), you’ll need a different architecture. Don’t put that in Drop. Use a proper supervisor with graceful shutdown timeouts in your main loop, and accept that panics will still bypass it.

Blocking Drop Handler

I said waitpid with WNOHANG doesn't block. This is mostly true. But "mostly" is doing a lot of work here.

WNOHANG returns immediately if the child hasn't exited yet. But if the child is in an uninterruptible sleep state, say, waiting on NFS or a FUSE filesystem, waitpid itself might block on kernel locks.

The fix is to not reap in Drop at all. Just send the signals, set a flag, and let a background task handle reaping. But that requires global state. Or you accept that WNOHANG is "non-blocking enough" for your workload and you monitor for edge cases.

The Leaderless Group

For this case, child process exits extremely quickly, faster than the kernel can set up the process group, it might die before setpgid completes in the child. The parent sees a successful spawn, caches the PID, but the child is already a zombie. When we call killpg, we're targeting a PGID that was never established, or the child's PID which is now a zombie not associated with any group.

The child also might have called setpgid successfully, then immediately exited, leaving the process group empty but technically existing. killpg on an empty group returns success (no processes to signal), but waitpid on the leader might fail if we've already reaped it elsewhere.

There’s a deeper issue: if the group leader dies but children survive, they become “orphaned” and reparent to init (PID 1). They’re still in the process group, but the group has no leader. killpg still works on them. the kernel doesn't require a living leader to signal a group, but some Unix variants handle this differently. (Solaris anyone?)

Defensive? Defensive!

Subprocess output is untrusted input. This seems obvious when you say it out loud, but production code could stream subprocess stdout into an unbounded Vec because "it's just a few kilobytes of JSON." Then someone points it at a malicious or merely broken program that outputs infinite newlines, and suddenly your "few kilobytes" is consuming gigabytes and the OOM killer is selecting victims.

We allocate an 8KB stack buffer and loop:

let remaining = limit.saturating_sub(data.len());
if read <= remaining {
    if let Some(slice) = buffer.get(..read) {
        data.extend_from_slice(slice);
    }
} else {
    if remaining > 0
        && let Some(slice) = buffer.get(..remaining)
    {
        data.extend_from_slice(slice);
    }
    exceeded = true;
    if let Some(tx) = notify.take() {
        // Best-effort signal to cancel the owning process.
        let _ = tx.try_send(());
    }
}

The saturating_sub is defensive against integer underflow, though data.len() should never exceed limit by construction. The slice bounds are checked. The memory footprint is provably bounded: limit + 8KB maximum, where the 8KB is the temporary read buffer.

But here’s the subtle part: once exceeded becomes true, we keep reading. We don't break the loop. We discard the data, but we keep issuing read() calls until we get EOF. Why? Because if we stop reading, the subprocess blocks on write(). It might be stuck in a write() syscall waiting for us to drain the pipe, and if we're not draining, we can't kill it cleanly. The pipe buffer fills, and the subprocess can block in write(). We still send SIGKILL immediately, which usually terminates the process promptly; however, in pathological kernel-level I/O states (for example uninterruptible sleep), teardown visibility can lag. Keeping the pipe drained is still a practical defensive measure to reduce backpressure and improve shutdown reliability.

A process could be “killed” but still in ps. The guard's Drop ran but the PID is still occupied. The solution is to keep the pipe empty so the process remains killable. We pay CPU cycles to read data we'll throw away, in order to guarantee that the process can actually die.

The notification channel is a one-shot circuit breaker. When we first exceed the limit, we take() the Optionmpsc::Sender<()> and try_send(()). The take() ensures we signal exactly once. The try_send() is non-blocking we're already in truncation mode, so we can't afford to wait for channel capacity. The parent can use this signal to tear down the entire pipeline, not just this one read.

Why Not Higher-Level Abstractions?

Could I have used tokio::process::Command directly? I did, initially. Just like with any cases, it's fine until it isn't, until you need process groups, until you need guaranteed cleanup, until you're debugging why your app has ten thousand PIDs and no apparent processes.

There’s a process pool or supervisor. Sure. Then I’d handle pool exhaustion, queue management, and the complexity of returning results from pooled workers. For Azalea’s workload, bursty, short and long-running, heterogeneous commands, a pool is overkill. We want per-task isolation and cleanup, not resource sharing.

The guard is the minimal abstraction that provides the guarantees we need. It doesn’t require global state. It doesn’t need configuration. It composes with any async code that can hold a struct until completion.

But minimal doesn’t mean simple. The guard encodes assumptions about Unix process semantics that are not portable, not guaranteed, and not even consistently documented, at least from what I can gather.

Hopes and Prayers

Defensive programming (and by extension systems programming) is the art of assuming failure. Not handling it gracefully, sometimes that’s impossible but preventing it from spreading.

I like elegant code, though I’ll grant that elegance is in the eye of the beholder. This, however, is not it. Elegant code would have async destructors and structured concurrency and graceful shutdown protocols. We don’t have those things. We have synchronous cleanup running on dying threads, sending violent signals to process groups we hope still exist, reaping zombies we hope are ours to reap.

The guard pattern encodes a protocol in the type system. The armed boolean is a promise: either we witnessed completion, or we will clean up. The PID cache is a hedge against kernel timing. The bounded read is a denial-of-service prevention mechanism. Each piece addresses a specific failure mode discovered through production pain.

But the guard is also a collection of trade-offs I’m not fully comfortable with. The terminal control issue means ^C doesn’t work intuitively. The lack of graceful shutdown means truncated files. The blocking Drop handler means potential runtime stalls. These are not bugs to be fixed. They're inherent tensions in the problem space. You can move them around, but you can't eliminate them.

It is the guarantees that hold when grace is impossible. The guard doesn’t trust you to clean up. It doesn’t trust Tokio to unwind cleanly. It doesn’t trust the subprocess to exit politely. It trusts only the kernel’s promise that SIGKILL is unblockable and waitpid will eventually reap.

Just remember: when you move a process into its own group, you become its parent, its supervisor, and its executioner. The terminal won’t help you. The runtime won’t help you. It’s just you, a cached PID, and hopes and prayers.