The Problem: When Monitoring Becomes a Second Job
A while ago, my colleague and I found ourselves in a classic infrastructure trap. We manage a few production servers running Docker containers for customer-facing services (payments, app backends, worker nodes). Downtime means immediate business losses.
We used to have a full Prometheus + Grafana + Alertmanager stack. It worked perfectly—until our dedicated DevOps engineer left the team.
Suddenly, maintaining the monitoring stack started taking more time than developing the actual product. Updating exporters, tuning silent alerts, and fixing broken dashboards turned into a second job. For a small infrastructure, the total cost of ownership (TCO) for Grafana was simply disproportionate to the value it provided.
We didn’t need complex charts. We needed an operational reaction channel that could:
- Catch container deaths instantly, not via periodic polling.
- Attach the last few lines of the Docker logs to the alert for instant diagnostics.
- Allow restarting the service from a smartphone without SSHing.
Since we couldn't find a lightweight tool with this exact balance, we wrote HostPulse.
The Client Side: A 20MB Stateless Agent
To keep things dead simple for the end-user, we built the client side as a lightweight Go agent.
It deploys with a single command: docker run. The image size is around 20MB, memory footprint sits at 15–20MB, and CPU usage is close to zero. The source code is open, so anyone can audit what data is collected.
Currently, we have:
- Metrics Agent: Sends CPU, RAM, and disk utilization every few seconds.
- Docker Agent: Subscribes to the Docker daemon events, captures failures, grabs trailing logs, and sends them over.
- Windows Agent: A dedicated Windows Service to monitor non-Linux machines.
All alerts end up in a convenient chat interface (like Telegram, with Discord/Slack integrations planned next), complete with a secure "Restart Container" action button.
Moving to the Next Level: The High-Availability Backend Architecture
The initial version worked great in our staging and early production environments. But as we began onboarding more servers, a critical question arose: If HostPulse is a mission-critical tool for our users, how do we guarantee its backend never drops an alert?
Even if our own data center experiences a total blackout, your server’s critical alerts must be delivered.
Right now, we are in the process of building and actively testing a fully redundant, high-availability architecture for the HostPulse backend. We wanted to share the blueprint of what we are deploying:
(Insert your final architecture diagram here)
How it works under the hood:
- Active-Active Traffic Routing: We deploy two parallel ingress points (Entry Points). Agent requests (heavy metrics and event logs) go into independent entry points to distribute the load.
- Kafka Buffering & Mirroring: To prevent data loss during unexpected traffic spikes or backend restarts, agent requests are pushed into Apache Kafka clusters. We are implementing cross-cluster Mirroring so that messages in Kafka 1 are safely duplicated to Kafka 2. If one backend node fails, the other can pull the mirrored messages.
- Stateless Backends: Backend 1 and Backend 2 are completely stateless. They pull jobs from their local Kafka queues and can dynamically back each other up via cross-consumption lines if one of them goes down.
- Master-Slave Storage with Failover: Both backends actively write to a single DB Master. This completely eliminates split-brain data conflicts. Data is continuously replicated to a DB Slave. In case of a catastrophic failure of the Master database, our failover automation will promote the Slave to Master, and Backend 2 will instantly route traffic to it via a fallback connection.
Current Status: Testing and Tweaking
This backend architecture is currently in the active testing phase. We are simulating split-brain scenarios, dropping Kafka brokers mid-transit, and measuring replication lag between our databases.
By separating a lightweight, dead-simple agent from a robust, heavy-duty enterprise backend, we get the best of both worlds: extreme ease of use for the customer and guaranteed reliability under the hood.
What about you?
How do you handle monitoring on small-to-medium infrastructures once there is no dedicated DevOps engineer to babysit a massive Prometheus/Grafana stack? Do you stick with it, use custom cron jobs, or look for hyper-focused niche tools?
Check out our project website at hostpulse.link and let us know what you think about our architectural approach in the comments!
Top comments (0)