One exit for every error of every transport: a mapped code instead of Unknown, a fixed message instead of internal text, and a panic that becomes Internal instead of taking the process with it.
👋 Hi, I'm Anton - a software engineer. These days I work on a platform: contracts, generated code, data paths and the tooling around them. Running notes live on my GitHub: github.com/brilliant-almazov.
The exit guard is built into the shared library and is on by default in its gRPC server chain; its tests, benchmarks and leak checks were run as targeted runs with the race detector. How often the old leak happened in production I did not measure, and I do not claim a number.
This is part 3 of the series on what a shared platform does for every service, told as a zoo. Maybe you already do it better, maybe you'd do it another way - this is how it is going, and where it stops.
In real life
Every animal that leaves an enclosure in our zoo wears a collar. The collar says whose animal it is and where it belongs. Keepers put it on at the gate, every time, by hand.
One morning Nora finds Biscuit, our dog, trotting along the main path with no collar at all. "Whose dog is this?!" Biscuit, very pleased with himself, answers the only thing he can: "Unknown! Woof!"
It gets worse. A dog without a collar does not know what not to say. Biscuit happily tells every visitor what he saw behind the scenes: "Table seven, row forty-two!" Miso hisses "Shh! Nobody asked you" - but by then the visitors have heard it.
The zoo's fix is not "keepers, please remember the collar". The fix is a keeper at the one gate everybody leaves through. Nobody leaves without a collar - even Biscuit. And when somebody faints at the gate, the gate keeper does not panic either: the animal is carried out with a plain tag that says "internal", and the zoo stays open.
The mapper you had to remember
In our services, a handler turns a domain error into an answer for the caller. For gRPC that is a status code with a reason; for HTTP a status and a body; for a consumer a decision to drop the message or try again. The library has a builder for exactly that mapping, and for a long time each handler called it itself.
That is the keeper who has to remember the collar. The library's changelog describes what happened when one did not: the forgotten call handed the caller Unknown with the internal text of the error, and a panic in a consumer flew into the goroutine of the resource that ran it. Two failures, one cause: the last step of every error was optional.
One gate for three transports
The library now has one exit for errors, with three doors on it. In the shape of the code - names shortened, the meaning is what matters:
// one exit for every transport - a form
guard.Answer(ctx, err) // gRPC / HTTP: a mapped status, never raw text
guard.Recover(ctx, p) // a panic -> Internal, a fixed message, the span marked
guard.Settle(ctx, msg, err) // a consumer: drop the message, or retry it
Answer is the collar. Whatever error a handler returns, it goes through the mapping on the way out, so a forgotten mapper is no longer a leak - the gate maps it.
Recover is the keeper at the gate when someone faints. A panic in a handler becomes Internal with a fixed message, the trace span is marked, and the process keeps serving everyone else.
Settle is the consumer's door. A message whose error code is in the drop list - by default InvalidArgument and its neighbours, errors that will not get better by trying again - is dropped; anything else is retried. Before, that decision was another thing each consumer had to remember.
The guard is on by default in the chain of the library's gRPC server. A service does not import it to get it - which is why the count of services that import it directly is zero, and the count that run it is every service on that server.
| Door | Before | After |
|---|---|---|
| gRPC | the handler maps, or the caller gets Unknown with internal text |
the gate maps every error |
| HTTP | the same, by hand per handler | the same gate, the same mapping |
| consumer | a panic flies into the resource goroutine |
Internal, span marked, process alive |
| consumer decision | each consumer decides drop or retry | one rule: drop codes are dropped, the rest retried |
What the caller sees
The difference is easiest to see from the outside. The same failure, as a caller received it before and after the gate - shapes only, with the internal text replaced by a placeholder:
before: code = Unknown
message = "<internal error text: query, table, row>"
after: code = <the mapped code, e.g. NotFound / FailedPrecondition>
reason = <a reason code the caller can branch on>
panic: code = Internal
message = "<a fixed panic message>" # the span is marked, the process lives
The caller of the first answer can do nothing but log it and give up, and the text in it belongs to the service, not to the caller. The caller of the second can branch on the reason. The third answer is the important quiet one: a single broken request no longer takes the process down with it.
Tested at the gate
A gate is only worth something if it holds under load and under races. The tests, benchmarks and goroutine-leak checks for the guard were written in a pass over several prompt sets and run as targeted runs with the race detector.
What went wrong
Two real leaks, both from the same design: the last step of an error was the handler's job. A forgotten mapper sent Unknown with internal text to a caller - the dog without a collar, telling visitors about table seven. A panic in a consumer escaped into the goroutine of the resource that ran it - the animal fainting in the middle of the path instead of at the gate.
The lesson is the one this whole series keeps repeating. A rule that every handler must remember is not a rule; it is a hope. The fix was not a review checklist or a linter warning but a gate in the one place every error passes, on by default.
What I do not claim
- Not how often the leak hit callers before the gate. I did not count it.
- Not that every service already runs the newest guard. It is on by default for services on the library's server.
- Not that the drop list fits every consumer. It is the default.
- Not a full load test. The guard has benchmarks and race runs, not a production measurement.
What of this is method
- The last step of every error lives at one gate. Not in each handler.
- A panic is an answer, not an exit. Internal, a fixed message, the span marked, the process alive.
- Drop or retry is decided once. By the error's code, in one place.
Zoo keeping, built in - Part 3. Next: One Ticket, Punched Once.
If your handlers map their own errors, how would you find the one that forgot? If you recover panics, where - per handler, or at one gate? If your consumers decide drop or retry, is that decision written once or many times? What did your last error without a collar say to a caller?








Top comments (0)