The Case of the 3 a.m. Outage Nobody Saw

A small server ran three small services. A status page, an RSS digest fed by cron, and an API that one mobile app depended on. Nothing critical, until a database filled its disk at 3 a.m. and all three went down together. Nobody noticed for nine hours. The postmortem was short and uncomfortable: no alerting, no history, no way to know when the silence started.

This is the case study of the fix. The constraint was real money — the services were cheap, so the monitoring had to be cheaper. The developer set a budget of zero dollars and one evening. The goal was a minimal uptime monitor: check three endpoints every sixty seconds, record every result in SQLite, and fire a webhook after two consecutive failures. No dashboard. No pager duty. No metrics pipeline.

The build used MonkeyCode, an open-source AI coding assistant. Disclosure: This article was prepared as part of MonkeyCode's product outreach. At the time of writing, the project's free tier includes a 10-million-token model allowance and a free server option, and both played a role in the workflow. The model access wrote the scaffold; the server option gave the monitor a home. Token numbers and server terms can change, so the repository README is the only source of truth.

The workflow had three passes. First, a plain-language spec was written before any code: inputs, outputs, failure rule, storage schema. Second, that spec went into a single prompt, and the assistant returned a working Python scaffold. Third, the scaffold was reviewed line by line and tightened. The spec did the heavy lifting; the model did the typing.

The core script stayed under a hundred lines.

# monitor.py — minimal uptime checker with SQLite history
import json
import sqlite3
import sys
import time
import urllib.request
from datetime import datetime, timezone

FAIL_THRESHOLD = 2
TIMEOUT = 5


def probe(url: str) -> tuple[bool, float]:
    started = time.monotonic()
    try:
        with urllib.request.urlopen(url, timeout=TIMEOUT) as resp:
            ok = 200 <= resp.status < 400
    except Exception:
        ok = False
    return ok, time.monotonic() - started


def main() -> int:
    with open("endpoints.json") as fh:
        endpoints = json.load(fh)["endpoints"]

    con = sqlite3.connect("status.db")
    con.execute(
        "CREATE TABLE IF NOT EXISTS checks ("
        "url TEXT, ts TEXT, ok INTEGER, latency REAL)"
    )
    con.execute(
        "CREATE TABLE IF NOT EXISTS alerts (url TEXT, ts TEXT)"
    )

    for ep in endpoints:
        ok, latency = probe(ep["url"])
        ts = datetime.now(timezone.utc).isoformat()
        con.execute(
            "INSERT INTO checks VALUES (?, ?, ?, ?)",
            (ep["url"], ts, int(ok), round(latency, 3)),
        )

        recent = con.execute(
            "SELECT ok FROM checks WHERE url=? ORDER BY ts DESC LIMIT ?",
            (ep["url"], FAIL_THRESHOLD),
        ).fetchall()
        if len(recent) == FAIL_THRESHOLD and all(row[0] == 0 for row in recent):
            con.execute("INSERT INTO alerts VALUES (?, ?)", (ep["url"], ts))
            webhook = ep.get("webhook")
            if webhook:
                payload = json.dumps(
                    {"text": f"DOWN: {ep['url']} at {ts}"}
                ).encode()
                req = urllib.request.Request(
                    webhook,
                    data=payload,
                    headers={"Content-Type": "application/json"},
                )
                urllib.request.urlopen(req, timeout=TIMEOUT)

    con.commit()
    con.close()
    return 0


if __name__ == "__main__":
    sys.exit(main())

The config file is the contract between the script and reality.

{
  "endpoints": [
    {
      "url": "https://status.example.com/health",
      "webhook": "https://hooks.example.com/notify"
    },
    {
      "url": "https://api.example.org/ping"
    },
    {
      "url": "https://blog.example.net/"
    }
  ]
}

Two tables hold the whole history. checks stores one row per probe with a timestamp, a boolean, and latency. alerts records every time the threshold trips. That is enough to answer the two questions that matter: is it down now, and when did it start. A log file would have been simpler, but SQLite made the failure window queryable, and that difference mattered later.

Deployment on the free server took one cron line.

* * * * * cd /srv/uptime && /usr/bin/python3 monitor.py >> monitor.log 2>&1

The developer did not trust the first green run. Verification came first. A local mock server stood in for the real endpoints, and the monitor pointed at it.

python3 -m http.server 8080 --bind 127.0.0.1 &
python3 monitor.py
sqlite3 status.db "SELECT url, ts, ok FROM checks ORDER BY ts DESC LIMIT 3;"

The first probe recorded ok=1. Then the mock was killed and the monitor ran twice more.

kill %1
python3 monitor.py
python3 monitor.py
sqlite3 status.db "SELECT * FROM alerts;"

The alert row appeared exactly as specified. Two consecutive failures, one webhook, zero surprises. The threshold logic was the part most worth testing, because a single failed probe is normal and two in a row is news.

The monitor ran for ten days. On day six, the RSS digest service failed during a routine deploy. The webhook fired at 02:14, the failure window showed up in the log, and the issue was fixed before breakfast. Nine hours of silence became nine minutes of noise. The latency column turned out to be a quiet early-warning signal: response time had crept from 120 ms to 900 ms over three days before the crash. The alert caught the hard failure; the history showed the slow climb that led to it.

The first lesson from the case: the model was not the bottleneck, the spec was. Every vague prompt produced vague code. Every prompt that named a schema, a threshold, and a webhook produced code that matched the prompt. The free model access handled a well-scoped project without drama, and the 10-million-token allowance covered the whole build with room to spare.

The second lesson: the free server is a fine home for a cron job and a bad place for a promise. The monitor is a single process with a single point of failure. If the box dies, the monitor dies with it. For a side project that is acceptable. For a customer-facing SLA it is not even close. The script also does not deduplicate alerts; a service that stays down fires every minute. That is acceptable noise for a hobby tool, and a good reason to add a cooldown before this pattern touches anything serious.

The approach is not for every team. Teams that need multi-region checks, on-call rotation, or real paging should look elsewhere. Anyone monitoring production revenue should look elsewhere too. The approach also assumes the monitored services live somewhere other than the monitoring box; a monitor that shares a disk with its targets misses the exact failure that started this story.

The whole project — script, config, cron line — fits in one directory and costs nothing to run. The quiet lesson is that constraint is not the enemy of quality. A tiny tool, a clear spec, and a free tier caught an outage that nine hours of silence had hidden. The same recipe is easy to repeat, and the free tier makes the first run cost nothing.