The ID Went Negative. The Sharding Router Didn't Notice For Six Days.

The problem

A payments-adjacent service I worked on used a plain AtomicInteger-backed auto-increment ID for a high-write event table. It had been running fine for two years. Then a backfill migration kicked write throughput from ~300/sec to ~4,200/sec for three weeks straight, and the counter did something nobody had modeled: it hit 2,147,483,647 and wrapped to -2,147,483,648.

No crash. No exception. No log line. The service kept issuing IDs — they just went negative.

Six days later, a nightly reconciliation job flagged that roughly 0.006% of events existed in the source stream but not in the sharded store. Not zero — worse than zero. A small, silent, growing hole.

Why it happens

The routing layer picked a shard with shard = id % NUM_SHARDS. That line had been correct for two years because id had always been positive. Once IDs went negative, the formula didn't stop working — it started returning different wrong answers depending on which service evaluated it, because languages don't agree on what negative modulo means.

Python uses floored division: the result always takes the sign of the divisor. -7 % 16 in Python is 9 — still a valid, in-range shard index.

Java, C, C++, C#, JavaScript, and Go (and Rust, by default) use truncated division: the result takes the sign of the dividend. -7 % 16 in Java is -7. Not in range. Not a valid shard.

Our ID generator was a Java service. Our routing layer was Go. Both computed id % 16 on the exact same ID and both were "correct" by their own language's rules — and both disagreed with the one thing that mattered, which was "which of our 16 real shards owns this record."

The Go router held shard clients in a map[int]*ShardClient. A negative key it had never seen didn't panic — Go just handed back the zero value for a missing map key, which in this codebase was a *ShardClient with a no-op Write() stubbed in for testing and never removed. Every write to a negative shard index silently succeeded from the caller's point of view and went nowhere.

That's the actual failure: not the overflow, and not even the sign mismatch — the fact that an impossible routing key was handled by quietly doing nothing instead of failing loudly.

What to do about it

Stop using 32-bit signed IDs for anything unbounded. At 4,200 inserts/sec, int32 gives you about six days of headroom before wraparound. int64 gives you roughly 68 years at the same rate. This is a five-minute schema change that eliminates the entire bug class.

Normalize modulo sign explicitly, or avoid modulo entirely. If you must support mixed languages or can't rule out negative inputs, use the floor-mod pattern: ((id % n) + n) % n. Better: if your shard count is a power of two, use id & (n - 1) instead of id % n. Bitwise AND operates on the two's-complement bit pattern directly, so it's immune to the sign question altogether — and it's faster than a division-based modulo to boot.

Make "unknown routing key" a hard failure. A map lookup that silently returns a zero-value client is a landmine. Write() on an unrecognized shard should panic, alert, or reject — never no-op. The cost of a loud failure is a page. The cost of a quiet one is a six-day-old, slowly growing gap in your data that a human has to notice on their own.

Instrument the counter, not just the outage. We added an alert at 80% of int32 max on every unbounded counter in the system, well before any of them are near the edge. Overflow should be a scheduled maintenance ticket, not an incident.

Key takeaways

• Signed integer overflow doesn't crash — it wraps and keeps producing plausible-looking, wrong answers.
• The identical id % n expression can be correct in one language and silently wrong in another, because languages disagree on the sign of negative modulo.
• For power-of-two shard counts, id & (n - 1) is both faster and sign-safe — prefer it over modulo.
• An unrecognized routing key should fail loudly. A silent no-op is how a bug becomes a six-day data gap instead of a two-minute page.