This tutorial was written by Darshan Jayarama.
As DBAs, waking up at 2 AM because production is down or slow is a nightmare, and none of us like it. However, we all know that it is not instant karma; it is like a pressure cooker where the database has reached its breaking point. Database whistleblowers constantly give signals that we ignore before it explodes into a Production Issue.
Whistleblower 1: Performance Informant
High CPU utilization?
CPU utilization is not just an alert, it is a future prediction with regard to the database. When you get constant High CPU utilization alerts, think of it as a Whistleblower reviewing the database logs, metrics, and processes that are consuming CPU. If possible, shift those processes to a different server and let your Database breathe without pressure. Ignoring high CPU utilization can lead to high latency.
Query <5 seconds?
A query that was blazing fast is now causing a slow query alert since the last data load. It is a clear indication of a Missing or Unoptimized index. If an index is missing, create one. If unoptimized, then redesign a better index. Still facing the same? It needs a deeper and wider investigation.
Cache hit ratio <80%?
A lower cache hit ratio means higher disk reads, which tends to cause high IO Wait, and an Increased disk queue depth. It can happen if your query is unoptimized (we had addressed this earlier), or if lower memory/cache is assigned. Increase the memory if there is nothing pending in Database tuning.
Whistleblower 2: Cost Informant
Storage >80%:
Set your whistleblower when storage utilization hits >80% its your duty now to analyze storage to archive any old logs, old data, or remove unwanted processes that consume your space.
IOPS > Budget?
We usually get an alert for a high document-scanned-to-returned ratio, which indicates that queries are performing more IO than required. Optimize them. Reduce IOPS consumption by rewriting queries, redesigning the schema, and implementing a better Index strategy.
Unexpected auto scale?
MongoDB Atlas has a feature that monitors system load and automatically scales up or down as needed. Implement the solution. In case of a scale down, it does require 24-hour monitoring, but if you are sure there are no more surprises, you can scale down yourself.
Whistleblower 3: Reliability Informant
Replication lag >10 Seconds?
Investigate where it is lagging. Is it in primary? Must be loaded from writes, which lowers the read request priority. It usually triggers the built-in flow-control logic. If secondary is the issue, then is it at writing or reading? A detailed understanding of flow can help resolve this data durability. Ask developers to set writeConcern to match the application's requirements.
Oplog <24Hours?
A lower oplog size means lower retention of replication events. Set at least 24 hours, resize as per the application write load minOplogRetention.
Connection >80%?
Each connection can consume 1 MB of memory; the more connections, the more memory is consumed. Maintain connection hygiene, use a connection pool, and close connections when no longer needed.
Conclusion: Your production incident is one missed alert away; hire your whistleblower. Choose: Pressure cooker explosion → OR → Proactive DBA legend?
Top comments (0)