Hey. I want to share a case that came up while supporting a Sentry instance, and how I debugged it.
The Problem
Morning, as usual. Developers post in Slack: "Hey, something's wrong with Sentry. It's running but issues aren't coming through." I check the status, the instance is up, but it's true: no logs, no issues being created.
First instinct is to restart it. I do compose down && compose up. A minute later the instance is back, issues start flowing, developers are happy. Things are normal for an hour or two.
Then it happens again. No issues. Need another restart.
After a few days of this pattern, I realize this isn't a random crash. Something systematic is happening. Just restarting every day isn't a solution.
First Attempt: Basic Metrics
I look at the monitoring dashboard. The server only has basic metrics: disk, memory, CPU, network. Everything looks fine, nothing jumps out. This isn't telling me anything useful.
I wait for the problem to happen again. When it does, I check the basic metrics during the outage. Still nothing obvious. The graphs don't show what's broken.
Adding More Detail: Disk and Redis Metrics
I realize I need to see more granular information. I add:
- Disk I/O breakdown by device (dm-0, dm-1, dm-2, sda, sr0)
- Load Average
- Redis: RAM, evicted keys, expired keys, connected clients
I enable these and wait for the next morning.
First Real Clue
Next morning around the same time, I see it on the graphs.
Redis memory spikes up to its limit. Then keys start getting evicted. This is a real clue, if events are being processed normally, memory should free up. If it's filling up and keys are being evicted, events are getting stuck somewhere.
They're piling up in Redis because they're not being processed.
Adding Kafka Metrics
I dig through logs and metrics and find the answer. Snuba consumers are down. They're not processing events from Kafka. That explains it all, events aren't being processed, aren't being deleted from Redis, Redis fills up, keys get evicted.
I restart the Snuba consumers, they come back up, events start clearing out, things normalize. Developers see issues again.
But I know this will happen again tomorrow. Something is consistently killing these consumers. I need to find out what.
Now I need to control consumers. I add more detailed metrics on Kafka:
- Stuck consumers
- Consumer group members
- Consumer lag
I also set up alerts on consumer lag so I can catch issues immediately.
Following the Trail: Disk I/O
I wait for the problem to show up again. This time I'm watching closely.
On the next occurrence I look more carefully at the disk I/O graph I added earlier. That's when I see it.
Every morning around the same time, disk I/O is maxed out for almost 30 minutes straight. Disk I/O = nearly 100%, the disk is completely saturated.
This isn't random. It happens at exactly the same time each day.
That's when it clicks. Kafka works with disk. When the disk is completely loaded, Kafka can't read or write properly. Consumers start timing out, connections drop, they disconnect.
But why is the disk so loaded? There are no active processes writing large amounts of data. No jobs, no backup services. I check, nothing suspicious.
I ask the admins what's happening with the disk every morning.
"Oh yeah," they say, "we run a full server backup or snapshot every morning. It ties up the disk for about half an hour."
There it is.
The Full Picture
Now I see the complete chain of events:
- Morning: backup/snapshot starts - disk becomes completely saturated
- Kafka can't work properly - disk is busy, I/O is blocked
- Snuba consumers get timeouts - connections drop, they disconnect
- Events stay in Redis - they only get deleted after successful processing
- Redis starts filling up with unprocessed events
- Redis evicts old keys to free space
- After 30 minutes: disk is freed - consumers start working again, events clear out
- But some consumers stayed disconnected from the timeout, so processing is slower
- Redis keeps growing, slower than before but still growing
- Until I restart Snuba consumers
A backup is an inevitable part of production infrastructure. You can't just turn it off, the data is critical. You have to adapt the system to survive during backup windows.
The Solution
I implemented multiple layers of protection.
First: relay on all services. Relay is a buffer. If Redis is unavailable or overloaded, events accumulate locally on relay, then get forwarded when Redis recovers. Protection against data loss:
Official Sentry Relay
The Sentry Relay is a service that pushes some functionality from the Sentry SDKs as well as the Sentry server into a proxy process.
Documentation
- Product documentation can be found at: https://docs.sentry.io/product/relay/.
- Code and development documentation can be found at: https://getsentry.github.io/relay/.
License
Like Sentry, Relay is licensed under the FSL. See the LICENSE.md file and this
blog post
for more information.
Development
Note
Relay project has strict rules for AI usage. Please see the HOWTOAI.md.
To build Relay, we require the latest stable Rust (install via rustup). The crate is split into a
workspace with multiple features, so when running building or running tests
always make sure to pass the --all and --all-features flags.
The processing feature additionally requires a C compiler and CMake.
To install cmake run brew install cmake.
To install the development environment, run direnv allow then devenv sync…
Second: increased Redis memory limit. Instead of evicting keys when memory is full, just have more memory:
...
redis:
...
command: redis-server --maxmemory 6gb --maxmemory-policy allkeys-lru
...
I didn't change the evicted keys policy so that in a worst case scenario Redis doesn't get completely stuck. Better to lose a few keys than block the entire service.
Third: healthcheck and auto-restart for Snuba consumers. If a consumer crashes, let it restart itself automatically. Quickly:
snuba-consumer:
image: getsentry/snuba:latest
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:1218/health"]
interval: 30s
timeout: 10s
retries: 3
start_period: 40s
restart: on-failure:5
environment:
SNUBA_SETTINGS: docker
KAFKA_BROKERS: kafka:9092
REDIS_HOST: redis
...
Result
After these changes, things stabilized. Redis doesn't overflow thanks to relay and the increased memory limit. Snuba consumers, if they do crash, restart themselves. Developers see issues like they should.
Top comments (0)