Reading the orchestrator's Rust source to fix my own outage

This is part two of three. In part one I split my personal services onto their own cluster. The cutover went fine. Then two things broke that had nothing to do with the migration itself and everything to do with the orchestrator's code, and because the orchestrator is a small open-source tool I run myself, I could go read it.

That turned out to be the whole point.

Symptom one: a shared domain that 404s at random

Two demo apps share a single hostname. A storefront answers /*, an admin panel answers /admin/*. After the migration, whichever one you hit would sometimes 404, and which one "won" changed after every deploy or health-check flap.

The orchestrator, orca, keeps a route table mapping each domain to a list of upstream targets. The request path already did the right thing: it matched the longest path prefix across all targets for a host. So the read side supported several services per domain. The write side did not.

The registration function replaced the whole entry:

// on each service's route update
route_table.insert(domain.clone(), targets.clone());

insert overwrites. So when the storefront registered its targets, it wiped the admin panel's, and vice versa. Every deploy or health transition re-ran registration for one service and clobbered its sibling. Last writer wins, and the loser's paths 404 until it happens to register next.

The removal paths already used the right idea, retain by service name, so only the registration path was wrong. The fix was to merge instead of replace: drop only this service's stale targets, then extend.

let entry = route_table.entry(domain.clone()).or_default();
entry.retain(|t| t.service_name != config.name);
entry.extend(targets.iter().cloned());
if entry.is_empty() {
    route_table.remove(&domain);
}

Twelve lines. Multi-service domains became stable across deploys.

Symptom two: webhook redeploys returning 503

The second problem showed up as soon as the automated deploy pipeline fired for the first time against controller-hosted services. Push, build, push image, the webhook calls the controller, and the controller answers 503 with a message about an agent not being connected.

The clue was "controller-hosted". Services that ran on a remote agent redeployed fine. Only services pinned to the controller's own node failed.

Reading the redeploy path, the cause was clean. The controller self-registers as a node in the cluster so it shows up in the node list. A service pinned to the controller's own hostname resolved to that self-registered node id. Redeploy then treated it like any remote node and tried to send the operation over a control WebSocket. But the controller does not hold a WebSocket to itself. No channel, so the operation failed with an "agent offline" error, surfaced as a 503.

The fix was to exclude the controller's own node id from the remote-placement match, so a service pinned to the controller falls through to the local path it should have taken:

let found = found.filter(|id| *id != master_node_id());

Both fixes went into one pull request: orca#138.

Verifying in production, not in my head

I did not want to trust a reread. The route fix I checked by hammering both paths of the shared domain and forcing redeploys to confirm the routes stayed put. The redeploy fix I checked with the real pipeline: an empty commit to one service repo, which ran CI, pushed the image, called the signed webhook, and this time the controller logged a successful redeploy and returned 200 instead of 503.

Building the patched binary had its own wrinkle. One of my nodes is old enough that I had to compile inside a glibc-2.31 container to get a binary it would run. That is a story for part three, and it is where this migration stopped being tidy.

The actual lesson

Two things generalize.

First, when you self-host a small tool, its source is part of your operational surface. I have spent hours poking at closed systems from the outside, inferring behavior from logs. Here the behavior was one function call away. Reading insert versus retain was faster and more certain than any amount of black-box probing.

Second, the "it worked before" detail localizes the bug. The route bug only appeared with two services on one domain, which is a shape the old single-node layout never had. The redeploy bug only appeared for controller-hosted services once an automated pipeline started calling redeploy. In both cases the exact conditions under which it broke pointed straight at the code path responsible. When something regresses, the first useful question is not "what is wrong" but "what is different about the case that fails".

Next time you hit a wall with a tool you run yourself, check whether you can read the wall. Often you can.

Next

The fixes were real and they shipped in the next release. Installing that release is what broke the cluster three different ways in one afternoon. Part three: glibc, cgroups, and a bug I had just helped create.