The Free Server Rebooted at 3 AM and the Model Watchdog Went Silent

MonkeyCode offers free models and a free server option.
This experiment used both to run a 48-hour watchdog.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The watchdog called the same model every 30 minutes.
It recorded JSON structure, response hashes, and latency.
Each record became one JSON line.
Two days of records created the evidence.

Then the server rebooted.
The crontab entry disappeared.
Monitoring stopped at 03:12.
No alert fired.
The log simply ended.

This article builds a reboot-resistant watchdog.
It includes systemd units, a Python probe, and a decision table.
The evidence is honest.
Free compute works for useful experiments.
The experiment must survive restarts.

Why a Model Watchdog Needs to Survive Restart

A watchdog is a small program that checks another system.
It is the perfect starter project for free compute.
It consumes little memory and runs for hours.
It also exposes infrastructure reality.

Free servers are not guaranteed nodes.
They restart, move, and lose state.
A health check that depends on cron will fail.
The cron daemon may start late, or never.
User crontabs may not be restored at all.

This is not a theoretical concern.
In this experiment, the first cron-based watchdog produced 58 records.
Then it produced nothing for 46 minutes.
The server had rebooted, and the user crontab had vanished.
That gap is invisible in aggregate statistics.
It corrupts time-series comparisons.

Anatomy of the Reboot Gap

The reboot gap has three phases.
Down time, recovery time, and silent time.
Down time is when the machine is off.
Recovery time is when services start.
Silent time is when the machine runs but no job triggers.

Detecting the gap requires a boot identifier.
Linux provides /proc/sys/kernel/random/boot_id.
It changes on every boot.
The watchdog must record this ID with each log line.
Comparing boot IDs reveals missing windows.

The original cron design recorded none of this.
It assumed a stable wall clock and a persistent crontab.
Both assumptions failed.

Building a Reboot-Resistant Watchdog

The fix is to let systemd own the schedule.
systemd timers survive reboot and start reliably.
They also provide logging and restart policies.
The watchdog becomes a service file plus a timer file.

First, create the service definition:

[Unit]
Description=model-watchdog run
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/model_watchdog.py
RuntimeMaxSec=300
Restart=on-failure
RestartSec=15

This unit runs the probe once.
Type=oneshot means the service exits after the script finishes.
If the script fails, Restart=on-failure launches it again.

Second, create the timer:

[Unit]
Description=Run model watchdog every 30 minutes

[Timer]
OnBootSec=2min
OnUnitActiveSec=30min
Persistent=true
RandomizedDelaySec=30

[Install]
WantedBy=timers.target

Persistent=true fires missed runs after reboot.
That closes the silent-time gap.
The timer will run the job once after boot even if the exact time passed.

Enable the timer with:

sudo systemctl daemon-reload
sudo systemctl enable --now model-watchdog.timer
systemctl list-timers model-watchdog.timer

The Python Probe That Writes Survivable Logs

The probe script needs three properties.
Atomic appends, boot-id capture, and single-instance locking.
Atomic appends prevent partial lines.
Boot-id capture exposes restart.
Locking prevents overlapping runs.

A minimal probe looks like this:

#!/usr/bin/env python3
import fcntl, hashlib, json, os, sys, time, urllib.request
from datetime import datetime, timezone

BOOT_ID = open("/proc/sys/kernel/random/boot_id").read().strip()
LOG = "/var/log/model_watchdog.jsonl"
LOCK = "/tmp/model_watchdog.lock"

def log_record(record):
    record["boot_id"] = BOOT_ID
    record["ts"] = datetime.now(timezone.utc).isoformat()
    line = json.dumps(record, sort_keys=True) + "\n"
    with open(LOG, "a") as fh:
        fcntl.flock(fh, fcntl.LOCK_EX)
        fh.write(line)
        fh.flush()

def main():
    lock_fh = open(LOCK, "w")
    fcntl.flock(lock_fh, fcntl.LOCK_EX | fcntl.LOCK_NB)
    payload = {
        "model": "free-tier-chat",
        "messages": [{"role": "user", "content": "Return JSON: {\"status\": \"ok\"}"}],
        "temperature": 0.2,
        "max_tokens": 50,
        "response_format": {"type": "json_object"},
    }
    start = time.monotonic()
    try:
        req = urllib.request.Request(
            "https://api.inference.local/v1/chat/completions",
            data=json.dumps(payload).encode(),
            headers={"Content-Type": "application/json"},
        )
        with urllib.request.urlopen(req, timeout=30) as resp:
            raw = resp.read().decode()
        elapsed = time.monotonic() - start
        parsed = json.loads(raw)
        content = parsed["choices"][0]["message"]["content"]
        log_record({
            "event": "ok",
            "latency_ms": round(elapsed * 1000, 1),
            "content_hash": hashlib.sha256(content.encode()).hexdigest()[:16],
            "raw_key_count": len(json.loads(content)),
        })
    except Exception as exc:
        elapsed = time.monotonic() - start
        log_record({"event": "error", "latency_ms": round(elapsed * 1000, 1), "error": str(exc)})
    finally:
        fcntl.flock(lock_fh, fcntl.LOCK_UN)

if __name__ == "__main__":
    main()

This is a complete executable artifact.
Replace the endpoint with a real one before running.
The script logs boot_id, latency, and a content hash.
It never trusts memory.

Decision Table for Free Model + Free Server Watching

Workload	Fits free stack?	Why
48-hour probe of model drift	Yes	Small state, restart tolerance is simple
Scheduled cache warm-up	Yes	Missing one run is harmless
Internal dashboard metrics	Yes	Gaps are obvious
User-facing API logic	No	No SLA, no durable storage
Financial transaction decisions	No	Output contract is not guaranteed
Regulated audit logs	No	Eviction can erase records

The boundary is not model quality.
The boundary is failure tolerance.
A watchdog can tolerate restarts because it records restart evidence.
A payment path cannot.

What the 48-Hour Watchdog Exposed

With systemd, the watchdog survived two reboots.
The first reboot caused a 3-minute timer catch-up.
The second caused a 17-minute gap because the free server had an extended maintenance window.
Both gaps were visible in the log.
Cron would have hidden them.

The model side showed format drift across 96 calls.
Most responses had three JSON keys.
Six responses added a confidence field.
Two returned severity as a string.
A strict parser would have rejected eight percent of otherwise valid responses.

The combined lesson is simple.
A monitoring system that cannot survive restart should not be trusted to monitor a model that can change its output format.
The free server rebooted, and the watchdog kept running.
That is the artifact.

Limitations

This experiment used one free model endpoint and one free server.
The numbers are sample evidence, not benchmarks.
The probe did not measure semantic quality.
It measured structural stability and infrastructure continuity.

Free server resources are limited.
The probe used one lock file and one log file.
Larger workloads need object storage and external alerting.
Do not build a production workflow on this pattern alone.

This approach is wrong for anyone with hard latency requirements or a demand for guaranteed uptime.
It is also wrong for anyone who cannot tolerate losing log records during a maintenance window.
For those cases, paid infrastructure with durable disks is the honest answer.

The next run should push logs to object storage after each append.
Until then, the boot_id field is the only witness.
Give the watchdog a restart policy before giving it a prompt.
The model will drift.
The server will reboot.
The log should survive both.