The People versus a web server that was “working five minutes ago.”
Prosecution: The website is unavailable.
Defense: The process exists.
Judge: A process existing is not an alibi. My printer exists. It has been obstructing justice since 2014.
Meet MatrixSwarm’s apache_watchdog: a small, opinionated witness that checks Apache’s local service state and configured listening ports, raises the alarm when health changes, and can request a service restart. It also brings diagnostic evidence, because “it broke” is a mood, not an incident report.
Exhibit A: what it actually checks
The watchdog runs on the Linux/systemd host it watches. Its basic healthy verdict requires both:
- The configured systemd service is active.
- Every configured port appears in the host’s TCP listener output from
ss -ltn.
That is a useful service-level check. It does not make an HTTP request to every virtual host, validate your TLS chain, or place a test order for three emergency pizzas. A listener also does not prove which application owns it. Keep application and external availability probes for those questions.
When a healthy service becomes unhealthy, the agent gathers diagnostics, routes an alert, and starts its restart routine. When health returns, it can announce recovery. Optional forensic reports can go to a separate role.
One detail deserves its own witness protection program: the first probe establishes a baseline and returns. This implementation reacts to later health transitions. Starting the watchdog while Apache is already down does not immediately trigger its transition-based restart path. Establish a healthy baseline during commissioning, and test a controlled failure on a test host.
An unchanged down state does not launch a fresh restart routine every poll. This is a watchdog, not a toddler discovering the elevator button.
Exhibit B: give the witness the correct address
In Phoenix’s Swarm Workspace, add apache_watchdog from the Agent Palette. Configure the service name and ports for the machine you are actually deploying to.
An illustrative config fragment for a host using the apache2 unit and both HTTP and HTTPS:
{
"service_name": "apache2",
"ports": [80, 443],
"check_interval_sec": 10,
"restart_limit": 3,
"always_alert": 0,
"alert_cooldown": 300,
"alert_to_role": "hive.alert"
}
Use httpd instead if that is your installed service. Remove port 443 if this host does not serve HTTPS locally. A config copied from somebody else’s architecture is just a future incident with excellent formatting.
The agent needs permission to inspect service state and listeners. Automatic restarts use noninteractive sudo for the configured systemd unit; make sure the service account has the narrowly scoped permission that deployment requires.
The restart routine has a bounded attempt loop controlled by restart_limit. Exhausting those attempts disables further restart calls in that agent instance. Investigate the underlying failure instead of expecting a restart to negotiate with a syntax error.
The evidence bag
Diagnostic context can include service status and recent Apache error-log lines. If you configure mod_status_url, the agent also tries to retrieve it as diagnostic context; it is not a separate health threshold in this implementation. Apache’s mod_status documentation explains the endpoint and access controls. Keep that evidence desk accessible only where intended.
For structured incident data, configure report_to_role and a matching consumer. That is optional for the human alarm route below.
Summon the operator: any watchdog, any alarm relay
Apache is not married to email. Nginx does not get exclusive custody of Slack. All four watchdogs—Apache, MySQL, Nginx, and Redis—can use any of the swarm-alarm relays:
| Destination | Relay agent | Optional alert-payload encryption |
|---|---|---|
| Slack | slack_relay |
No equivalent toggle in this implementation |
| Discord | discord_relay |
Yes |
| Telegram | telegram_relay |
Yes |
email_send |
Yes |
The watchdog’s alert_to_role is hive.alert. The relays advertise hive.alert@cmd_send_alert_msg. Configure the relay credentials, resolve required Registry assignments, and make sure service scope and routing let the watchdog discover it.
Choose one reachable relay or several. The watchdog can fan the alert out to matching endpoints. Your escalation policy may be sophisticated; it may also be “put it everywhere Dave might look.” Both require working routes.
For Discord, Telegram, or email, explicitly enable that relay’s encrypt_alerts option and configure its assigned packet-signing/encryption keys. Having packet signing configured does not automatically enable encryption of the outgoing platform message.
The editor labels are Encrypt Discord alert message, Encrypt Telegram alert message, and Encrypt alert email subject and body. Secure wrapping fails closed: it does not silently send the protected alert as plaintext. Authorized operators can use the matching relay’s Decrypt Message panel in Phoenix with the proper keys.
Email protects the alert’s subject and body inside the envelope; routing metadata is still visible. Slack uses its normal HTTPS delivery but has no corresponding MatrixSwarm payload-encryption switch here. Encrypting the Discord copy does not encrypt a simultaneous Slack copy. The court rejects security by proximity.
Before the verdict
Deploy on a test host, confirm a healthy baseline, exercise a controlled failure, and verify the restart behavior, alarm delivery, recovery notice, and decryption where enabled. Test every selected destination. “The webhook looked convincing” is inadmissible.
Apache gets a witness. You get evidence. The phrase “nobody noticed” can finally retire from active duty.
Victory Always. Court adjourned.
🌐 Links & Resources:
Try MatrixSwarm: https://matrixswarm.com
Join the Community / Discord: https://discord.gg/2USbWVBVV
Download Server: https://github.com/matrixswarm/matrixswarm
Youtube: https://www.youtube.com/channel/UCMjiY4_-W2KP5fHXO0eC2ug
Top comments (0)