DEV Community

weiwuji
weiwuji

Posted on

From 996 to 007: The Self-Healing Ops Stack for a One-Person Company

#ai #agents #devops #automation

The Pain: You automated 100 tasks, but the system itself still needs a babysitter — and the babysitter is you. 3 AM alerts, weekend log-watching, disk-full panic on your busiest day. The most ironic moment of a one-person company: everything is automated except operations.
What You'll Learn:

  • Why "the process is alive" is not the same as "the service is healthy" — and how to check what actually matters
  • The 4-layer unattended ops stack: health checks → auto-restart → alerting → log rotation
  • A /healthz endpoint that really connects to the database, plus a 30-second probe script
  • systemd Restart=on-failure so small failures fix themselves in 5 seconds — without infinite restart loops
  • An alert script that only notifies after 3 consecutive failures — no false alarms, and a recovery notice when it's back
  • Why unattended ops is really about drawing a trust boundary, not max automation

1. Opening: 3:07 AM — Your Phone Lights Up

At 3:07 AM, the WeCom alert shakes you awake. It's not a customer question, not a new order — the content factory pipeline stopped. Database unreachable, articles not published, the A/B experiment convergence check never ran. You get up, open the laptop, read the logs, restart the service, confirm recovery, lie back down. The moment your eyes close, the second alert arrives.

This is OPC's most ironic moment: you automated every task — except operations, which can't automate itself. The system does 100 things for you, but the system itself needs a caretaker — and the only caretaker is you.

In the previous article we built a 3-layer automated A/B testing system (A/B Testing for One Person — an AI-Driven Automated Optimization System). The system tries, converges, and switches by itself. But all of it rests on a fragile assumption: the server must stay alive. Keeping it alive was still human on-call — 996 would be a luxury; it was 007.

Today's cure is a 4-layer unattended ops stack: health checks, auto-restart, alerting, log rotation. After this, the system takes care of itself: small illnesses it gets up from on its own, big ones wake you — and it won't wake you at 3 AM just to say "I'm better now."

2. First, Get This Straight: OPC Ops Is Not Enterprise Ops

Enterprise ops targets: 99.99% availability, multi-datacenter redundancy, on-call rosters, ticket systems, change approvals. OPC has none of these resources and needs none of these goals. The OPC reality is: one server, one deploy directory, no roster, no dedicated ops engineer.

So OPC's ops goal is rewritten into three sentences:

  1. If it dies, it gets back up by itself (self-healing)
  2. If it can't get up, it wakes me (alerting)
  3. Don't let small problems become big ones (log rotation and disk watermarks)

The core insight is one sentence: unattended does not mean "nobody watches" — it means "the machine handles it first, and only calls a human when it can't." Every design below follows from this sentence.

3. The Architecture: A 4-Layer Unattended Ops Stack

4-layer unattended ops architecture: health checks, auto-restart, alerting, log rotation — each layer labeled with its core component and parameters, with the failure-data flow from detection to handling marked on the right
The architecture diagram is the roadmap for this article.

┌────────────────────────────────────────────────────────┐
│ Layer 1: Health Checks                                 │
│ → service exposes /healthz, probe every 30s            │
├────────────────────────────────────────────────────────┤
│ Layer 2: Auto-Restart                                  │
│ → systemd Restart=on-failure, up in 5s                 │
├────────────────────────────────────────────────────────┤
│ Layer 3: Alerting                                      │
│ → only after 3 consecutive fails, WeCom push           │
├────────────────────────────────────────────────────────┤
│ Layer 3 (cont.)                                        │
│ → + recovery notice when self-healed                   │
├────────────────────────────────────────────────────────┤
│ Layer 4: Log Rotation                                  │
│ → logrotate daily+size, compressed, keep 7             │
└────────────────────────────────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

Each layer solves one class of problem, and a lower layer is the fallback for the one above. Let's take them apart one by one.

4. Layer 1: Health Checks — Let the System Report "I'm Alive"

First, an unintuitive pitfall: the process being alive is not the same as the service being healthy.

The first time I wrote a liveness script, I only checked "is the process there": pgrep -f content_factory. If the process was there, everything was fine. Then one day the process was there, but the service was already dead — the database connection pool was exhausted, every request hung until timeout. The process wasn't dead; the business was down.

So health checks need three levels:

  1. Process level: is the process there (weakest)
  2. Interface level: does the HTTP endpoint respond (medium)
  3. Business level: is the business state correct (strongest — and what OPC actually needs)

Step one, add a health-check endpoint to the service (FastAPI example):

# app/main.py - health check endpoint
import fastapi
import psycopg2
import time

DB_DSN = "postgresql://user:***@127.0.0.1:5432/analytics"
app = fastapi.FastAPI()

@app.get("/healthz")
def healthz():
    status = "ok"
    try:
        conn = psycopg2.connect(DB_DSN, connect_timeout=3)
        conn.close()
    except Exception:
        status = "degraded"
    return {"status": status, "service": "content-factory", "ts": int(time.time())}
Enter fullscreen mode Exit fullscreen mode

This endpoint does one thing beyond "process is alive" — it really connects to the database once. An interface being reachable doesn't mean the dependency is reachable; the probe must probe the weakest dependency. The database is the most fragile dependency in an OPC system, so /healthz must really connect once, with a 3-second connect timeout, and the probe script waits at most 5 seconds.

Step two, create the healthcheck.sh probe script (cron every 30 seconds):

#!/bin/bash
# healthcheck.sh - probe script
HEALTH_URL="http://127.0.0.1:8000/healthz"

code=$(curl -s -o /dev/null -w "%{http_code}" --max-time 5 "$HEALTH_URL")
if [ "$code" != "200" ]; then
  echo "health check failed: http=$code at $(date)" >> /var/log/healthcheck.log
  exit 1
fi
body=$(curl -s --max-time 5 "$HEALTH_URL")
if echo "$body" | grep -q '"status": "degraded"'; then
  echo "health check degraded: db unreachable at $(date)" >> /var/log/healthcheck.log
  exit 1
fi
exit 0
Enter fullscreen mode Exit fullscreen mode

Both curl's exit code and the HTTP status code must be checked, and the degraded state must be logged too. This layer doesn't handle anything — it only detects. Detection is handed to the next layer.

5. Layer 2: Auto-Restart — Small Illnesses Get Up by Themselves

When the health check finds a problem, who handles it? If it's the middle of the night, the answer is: not a human — let systemd handle it.

systemd's Restart policy was designed for exactly this. Step one, create content-factory.service under /etc/systemd/system/:

# /etc/systemd/system/content-factory.service
[Unit]
Description=OPC Content Factory
After=network.target postgresql.service

[Service]
User=opc
WorkingDirectory=/home/opc/content-factory
ExecStart=/home/opc/.venv/bin/python3 -m uvicorn app.main:app --host 127.0.0.1 --port 8000
Restart=on-failure
RestartSec=5
StartLimitIntervalSec=300
StartLimitBurst=5

[Install]
WantedBy=multi-user.target
Enter fullscreen mode Exit fullscreen mode

Two parameters matter most:

  • Restart=on-failure: auto-restart when the process exits abnormally. Do not use Restart=always — if the process keeps crashing because of a config error, always will restart forever and flood the logs; while on-failure combined with StartLimitBurst=5 means at most 5 restarts in 5 minutes, then it stops trying and enters the failed state — that's when alerting gets its turn.
  • RestartSec=5: wait 5 seconds before restarting, giving the database a little recovery time.

Run these commands to enable the service and check its status:

systemctl daemon-reload
systemctl enable content-factory
systemctl start content-factory
systemctl status content-factory --no-pager
Enter fullscreen mode Exit fullscreen mode

The complete self-healing decision flow:

self-healing closed-loop: probe fails -> systemd auto-restart -> re-probe -> success records 'self-healed'; consecutive failures accumulate and trigger the alert layer

Probe fails → systemd auto-restarts → re-probe → success records "self-healed"; consecutive failures accumulate, and past the threshold it enters the alert layer. Auto-restart is the "action"; health checks are the "eyes" — they must come in pairs. A restart without a probe is a blind person driving.

6. Layer 3: Alerting — If It Can't Get Up, Call a Human

If systemd restarts a few times and still can't get up, it's not a random crash — it's a problem that needs a human. That's when alerting gets its turn — and alerting itself must be restrained.

Alerting's worst enemy is false positives. The cost of one false alert is not a single message; it's that you stop trusting alerts, mute notifications, and then the real big thing gets drowned out. So alerting must do two things: consecutive confirmation + recovery notice.

Create monitor.py, independent of systemd (cron every minute):

# monitor.py - alert script, push only after 3 consecutive failures
import urllib.request

HEALTH_URL = "http://127.0.0.1:8000/healthz"
WEBHOOK_URL = "https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=YOUR_KEY"
STATE_FILE = "/var/run/opc-monitor.state"
MAX_FAILS = 3

def check():
    try:
        req = urllib.request.urlopen(HEALTH_URL, timeout=5)
        return req.status == 200 and b'"status": "ok"' in req.read()
    except Exception:
        return False

def send(text):
    payload = '{"msgtype": "text", "text": {"content": "%s"}}' % text
    data = payload.encode("utf-8")
    req = urllib.request.Request(WEBHOOK_URL, data=data, headers={"Content-Type": "application/json"})
    urllib.request.urlopen(req, timeout=10)

def main():
    try:
        with open(STATE_FILE) as f:
            fails = int(f.read().strip())
    except Exception:
        fails = 0

    if check():
        if fails >= MAX_FAILS:
            send("Service recovered, self-healed: content-factory")
        fails = 0
    else:
        fails += 1
        if fails == MAX_FAILS:
            send("ALERT: content-factory failed 3 health checks in a row; systemd restart ineffective, manual intervention needed")

    with open(STATE_FILE, "w") as f:
        f.write(str(fails))

if __name__ == "__main__":
    main()
Enter fullscreen mode Exit fullscreen mode

This uses only the standard library urllib — no requests to install. State lives in a file under /var/run, so a machine reboot clears it and the count restarts. Behavior breakdown:

  • Fail 1 or 2 times: count only, no push — jitter doesn't bother you
  • Fail 3 times: push the WeCom alert, and only once
  • Recovery: if it was in alert state, push "self-healed" and reset the count to zero

The WeCom bot webhook is a one-line config, and you receive alerts on your phone with WeCom installed. The cron config:

# crontab -e add the following line
* * * * * cd /home/opc && /usr/bin/python3 monitor.py >> /var/log/opc-monitor.log 2>&1
Enter fullscreen mode Exit fullscreen mode

The full alert chain:

alert channel: service anomaly -> monitor script consecutive confirmation -> threshold hit -> WeCom bot webhook -> phone push, while logs land on disk and are rotated by logrotate

At this point, "the system takes care of itself" holds: small illnesses self-heal, big ones call a human, and when it's better it tells you.

7. Layer 4: Log Rotation — Don't Let Logs Fill the Disk

The last layer is the most unremarkable, but it's the most common accident root cause on an OPC server: logs fill the disk, and every service dies.

As soon as the service runs, logs grow at a visible pace. nginx access logs, application logs, probe logs, monitor logs — several files stack on top of each other. Once the disk is full, the database can't write, services can't start, even SSH can hang. And it usually happens on your busiest day.

The fix is logrotate — Linux's built-in log rotation tool:

# /etc/logrotate.d/opc
/var/log/content-factory/*.log {
    daily
    rotate 7
    compress
    delaycompress
    missingok
    notifempty
    size 50M
    copytruncate
}
Enter fullscreen mode Exit fullscreen mode

What this config means: rotate daily, or immediately when a single file exceeds 50M; keep 7 copies; compress old logs to .gz; copytruncate is friendly to actively-writing processes — copy first, then truncate, so the app doesn't need a restart.

Rotation alone isn't enough — the disk watermark itself needs monitoring too. Create disk_watch.sh:

# disk_watch.sh - disk watermark check, cron hourly
usage=$(df / | awk 'NR==2 {print $5}' | tr -d '%')
if [ "$usage" -gt 85 ]; then
  curl -s -H "Content-Type: application/json" \
    -d '{"msgtype":"text","text":{"content":"ALERT: disk usage over 85% - clean logs or expand storage"}}' \
    "https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=YOUR_KEY"
fi
Enter fullscreen mode Exit fullscreen mode

Push an alert when disk usage exceeds 85%. Log rotation is the "treatment"; the disk watermark is the "prevention". Together they close the most common pitfall.

8. Before vs After: What You Save

Area Human on-call Unattended stack
Probing "should be fine" by gut feel /healthz really connects to the database every 30s
Restart crawl out of bed at 3 AM, systemctl restart on-failure auto-up in 5s, at most 5 times in 5 minutes
Alerting users find out first and come question you push after 3 consecutive fails; "self-healed" notice on recovery
Logs forget about them until the disk explodes logrotate daily + 50M, compressed, keep 7
Your sleep fragmented, phone vibrating all night a whole night; the phone only rings when it's real

9. Deeper: The Core of Unattended Is Not Automation — It's a Trust Boundary

Many people read "unattended" as "fully automatic," so they cram automation into every step, and end up with a system nobody can understand — when it fails, everything fails. The direction is backwards.

The essence of unattended ops is drawing a trust boundary around failure recovery: which things the machine handles itself, and which things must call a human. If the boundary is drawn too wide, the machine restarts in the wrong direction again and again, making things worse. If it's drawn too narrow, the machine dares to do nothing, and you're still the on-call.

In this stack, the boundary is drawn on three lines:

  1. Retryable actions go to the machine: restart, retry, rotation — if it fails, try again; machines have more patience than humans
  2. Actions that need judgment go to alerting: consecutive failures, disk watermark, database unreachable — these can't be solved by retrying; a human must be called
  3. Recovery actions must have a receipt: self-healed must be announced — to avoid "woken at 3 AM, only to find it's already fine." Trust is built one receipt at a time

One level deeper, this stack is the same pattern as the previous articles: detect → decide → act → detect again. A/B testing is an experiment loop; unattended ops is an operations loop. The essence of Loop Engineering is turning every repetitive chore into this kind of closed loop. Operations stops being a burden — it becomes part of your product. The availability of the system is itself the user's experience of the product.

10. Summary: Two Things You Can Start Today

Reviewing the 4-layer stack:

  1. Health checks: /healthz really connects to the database, probe every 30 seconds, curl probe script
  2. Auto-restart: systemd Restart=on-failure + RestartSec=5, at most 5 times in 5 minutes
  3. Alerting: monitor.py pushes after 3 consecutive failures; "self-healed" on recovery
  4. Log rotation: logrotate daily + 50M, compressed, keep 7, disk 85% watermark alert

Don't wait for the next incident. Do two things today: add a /healthz endpoint to your service, and create a systemd unit with Restart=on-failure. After these two, you've deleted "3 AM restarts" from your life.

Next article: Selling the System to More People — From a One-Person Company to a Replicable Business System

Your own OPC system can write articles, serve customers, deliver products, analyze data, and take care of itself. But it all serves only one person. Next, we'll talk about turning this system into a replicable product — letting other people's businesses run on your system, upgrading you from a "one-person company" to a "system company".

About the author: Wu Ji (无记) — AI & digitalization practitioner focused on Agent engineering, Loop Engineering, and digital transformation. Practical, hands-on tutorials — follow along and it just works.

Top comments (0)