Design a metrics monitoring system

How do you design a metrics monitoring system for 100,000 servers?

I’ve shared my approach here: https://engineering.by-tech.workers.dev/articles/metrics-monitoring-system-design/

Here’s the high-level flow:

1. Agents collect metrics; a durable message queue buffers them.
2. A time-series database stores metrics for dashboard queries.
3. A stream processor evaluates alert rules independently of dashboard storage.
4. Notification workers deliver alerts and retry failed sends.
5. Older measurements become summaries to keep storage manageable.

I also cover capacity planning, push vs. pull collection, data retention, and recovery from failures - along with alert rules for sustained spikes and missing data, since a brief CPU spike needs a different response from prolonged overload.