`node = "ubuntu"` matched two machines: a placement split-brain

This is part of a series on building Orca, a single-binary orchestrator for the gap between Coolify and Kubernetes.

In Orca you can pin a service to a specific machine:

[service.placement]
node = "ubuntu"

Simple, obvious, and it worked fine - right up until the cluster had two nodes whose names shared a prefix: ubuntu and ubuntu-16gb-fsn1-1.

Then one push recreated every pinned service on both machines. Including a database. Two Postgres instances, on two different hosts, both believing they were the one true copy of the same service. In production. A split-brain, born from a single line of code.

The one line

The placement matcher did this:

if node.address.contains(pin) { /* schedule here */ }

contains. A substring test. "ubuntu-16gb-fsn1-1".contains("ubuntu") is true. So a pin of "ubuntu" matched both nodes, and every service pinned to ubuntu got scheduled onto both.

Worse: which node "won" a plain deploy was decided by HashMap iteration order - non-deterministic across restarts. So the behavior wasn't even consistently wrong. It was randomly wrong, which is the most expensive kind.

Substring matching on an identity is one of those bugs that looks completely reasonable in the diff and is a landmine in the field. It works in every test you write, because your test nodes have distinct names. It only detonates when someone, months later, names a second node with an overlapping prefix - which is exactly the kind of thing people do.

The fix: exact identity, and refuse to guess

Two changes, and the second matters more than the first.

One - match exact identity, never a substring. A pin now matches a node only if it equals the node's id, its full ip:port address, the host portion of that address, or its hostname label. "ubuntu" matches the node named ubuntu and only that node.

Two - when a pin is ambiguous, refuse. This is the real lesson. If a pin somehow matches more than one node, the old code picked one (arbitrarily). The new code fails the deploy loudly:

placement pin "ubuntu" matches multiple nodes: 1 (10.0.0.1), 2 (10.0.0.2)
- refusing to schedule; make the pin unique

And if a pin matches nothing - a decommissioned node, a typo - it also refuses, instead of silently falling through to "just run it on the master." That silent fallback was its own latent bug: a mistyped pin would quietly deploy your service to the wrong machine and everything would look fine until it very much wasn't.

The principle is refuse, don't guess. An orchestrator that guesses when it's uncertain will eventually guess wrong at 3am, silently, in a way you find out about from a customer. An orchestrator that refuses hands you a clear error at deploy time, when you're right there watching. A loud failure you see beats a quiet success you don't.

The part I got wrong on the way

Here's the honest bit. When I shipped the exact-match fix, I audited the production config to make sure every existing pin still resolved. I checked the pins against the node addresses I had in memory - 217.154.26.121:9443 and friends - and concluded they were fine.

They weren't. Agents register with a hostname:port address, not ip:port. So the ten pins that used a bare IP suddenly matched nothing, and - thanks to the new "refuse, don't guess" behavior - those deploys started failing loudly.

Which was, ironically, the system working exactly as designed. The fix caught a real config problem (pins pointing at an identity the node doesn't actually register under) instead of silently papering over it. The remedy was to add peer-IP matching so IP pins work regardless of what the agent self-reports - but the meta-lesson stuck: audit against what the system actually stores, not against what you assume it stores. My mental model of the node addresses was wrong, and only the exact-match refusal surfaced it.

Two rules I'd tattoo on a junior engineer

1. Never substring-match an identity. Names, ids, addresses - compare them whole. The prefix collision is always coming; it's just a question of when.
2. Refuse when you're not sure. Ambiguity and "no match" are not edge cases to paper over with a default. They're exactly the moments to stop and shout, because a wrong guess in placement means your database is running in two places.

The whole thing was maybe forty lines of code to fix. The split-brain it prevents is unbounded. That ratio - tiny fix, huge blast radius avoided - is most of what infrastructure work actually is.