Here is a question worth being uncomfortable about: if one of your API keys leaked right now, how would you find out?
Not "how would you rotate it" — that part is easy. How would you notice?
A stolen key does not look stolen. It authenticates correctly, because it is a valid key. It hits the same endpoints, because whoever took it wants the same functionality you built. It bills to the same account. Every row it writes to your usage log looks exactly like a row you wrote.
The only thing that differs is where it comes from. Someone else's server is not the one you deploy from.
Which means if you are not recording the caller's address alongside the usage, there is no evidence a theft ever happened. You find out from the bill, weeks later, and even then you cannot tell which calls were yours.
What makes this a nasty class of problem is that the fix is trivial and almost nobody does it until after the incident. One column. The hard part is the alerting: most legitimate integrations call from several addresses — a few app instances, a laptop, CI — so "more than one address" fires constantly and gets ignored, and then it is not there on the day it matters.
The heuristic I settled on: never flag the busiest source, and only flag a minority share once there is enough traffic for a share to mean anything. Quiet enough to still be believed.
What do you do — per-key IP allowlists, anomaly alerting, or nothing until something looks weird on the invoice?
Top comments (4)
The address itself carries signal before the volume does. Watch the ASN, not just the IP. A key that always called from your deploy provider suddenly hitting from M247 or Datacamp is anomalous on request one. Caveat: most proxy-detection feeds mislabel entire AWS and Hetzner ranges as proxies, so a naive check trips on your own CI.
That tension is real and I didn't resolve it in the post. Share-based alerting needs volume before a share means anything, so on request one there is no share to compute and you're pushed straight back onto the lists you're warning about.
What survives it, I think, is that a per-key baseline doesn't need the feed to be right. A global list has to be correct about the whole internet. A baseline only has to be correct about one customer — this key has never once called from that ASN — and that holds whether or not anyone got round to labelling it.
On the AWS and Hetzner mislabelling: the part that bites is that most checks can't tell "this is a datacenter proxy" from "we have no idea what this is", and both end up adding risk. We had a line doing exactly that, scoring a missing signal the same as a bad one. Unknown and absent are different states, and collapsing them is where the CI false positive comes from.
A missing signal isn't a vote. Datacenter IPs need two sources agreeing, residential one.
Great reminder to monitor API callers.