48-Hour Field Notes: I Made My Batch Job Ask Permission Before Touching a Free Model Endpoint

Two days ago I moved a batch job onto a free server and pointed it at a free model endpoint, and for the first eight hours I was sure I had found capacity with no strings attached. Then the job that "worked yesterday" spent most of an hour producing nothing, exited zero, and wrote a summary file that looked perfectly fine at a glance. Sound familiar? The endpoint never went down, the box never ran out of memory, and no single log line used the word failure — which is precisely why it took me so long to see it.

What I got wrong was not a library choice, a prompt, or a retry policy. I had shipped a job that assumed its environment would keep behaving like the environment I measured it in, and I had never written that assumption down anywhere a program could check. These are my notes from the 48 hours it took to correct that, including the preflight script I now run before the first item, the test plan that proves it works offline, and the parts that are still fragile.

Hour 0–8: the failure was a missing envelope, not a broken endpoint

An envelope is the set of environmental facts a job needs to be true before its output means anything: how much wall clock it has, how much CPU it actually gets, how much room is left on disk, and whether the endpoint answers at all. My job checked none of those, so every one of them was an implicit assumption with no error message attached. When one assumption quietly stopped holding, the job did the only thing it knew how to do — keep going.

Here is the uncomfortable part. A free endpoint gives you no service-level agreement, and a free server gives you no promise about the machine underneath it, so both of them push the entire correctness burden onto your code. That is not a complaint about free capacity; it is a statement about where the deadline has to live. If the deadline lives in the endpoint's response time, you do not own it. If it lives in your process, you do.

So ask yourself the question I should have asked on day one: if the endpoint answered in three times its usual latency, would my job still be correct, or would it just be slower and wrong? Those are very different outcomes, and only one of them shows up in a dashboard.

The setup, and where the free parts actually sit

My run used MonkeyCode's free server option for compute and its free model access for the calls, which is what made the whole experiment cheap enough to be worth doing on a whim. And that is exactly the trap, right? When the compute is free, you skip the capacity conversation, and when the endpoint is free, you skip the latency conversation, and then you are surprised when either one shows up unannounced.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I want to be precise about what I am claiming here, because it is easy to over-claim. I am not claiming anything about throughput, quota, machine size, or how long any of it stays free — those are operator-supplied availability claims and they can change without my code knowing. What I am claiming is that the script below makes those changes visible at runtime instead of visible in a half-written output file at hour thirty.

The artifact, part 1: write the envelope to disk

This is stdlib only, because a preflight check that needs a dependency tree before it can tell you the box is broken is not a preflight check. It reads cgroup v2 limits, the disk headroom on /tmp, the Python version, and the endpoint's reachability, then prints a single JSON object.

# envelope.py - stdlib only. Prints one JSON object describing the box and the endpoint.
import json, os, shutil, sys, time, urllib.request


def cgroup_limits():
    out = {}
    for key, path in (("cpu_max", "/sys/fs/cgroup/cpu.max"),
                      ("mem_max", "/sys/fs/cgroup/memory.max")):
        try:
            with open(path) as fh:
                out[key] = fh.read().strip()
        except OSError:
            out[key] = None  # cgroup v1, container without cgroup fs, or not Linux
    return out


def cpu_quota_cores(cpu_max):
    # "200000 100000" -> 2.0 cores; "max 100000" -> None (no quota)
    if not cpu_max or cpu_max.startswith("max"):
        return None
    quota, period = cpu_max.split()
    return round(int(quota) / int(period), 2)


def probe_endpoint(url, timeout_s=2.0):
    started = time.monotonic()
    try:
        with urllib.request.urlopen(urllib.request.Request(url, method="GET"),
                                    timeout=timeout_s) as resp:
            resp.read(64)
            status = resp.status
    except Exception as exc:  # timeout, DNS, TLS, refused -- all the same to a deadline
        return {"ok": False, "error": type(exc).__name__,
                "elapsed_ms": int((time.monotonic() - started) * 1000)}
    return {"ok": 200 <= status < 300, "status": status,
            "elapsed_ms": int((time.monotonic() - started) * 1000)}


def build_envelope(budget_s, endpoint_url):
    limits = cgroup_limits()
    return {
        "generated_at": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
        "python": sys.version.split()[0],
        "budget_s": budget_s,
        "cpu_count": os.cpu_count(),
        "cpu_quota_cores": cpu_quota_cores(limits["cpu_max"]),
        "cgroup": limits,
        "tmp_free_mb": shutil.disk_usage("/tmp").free // (1024 * 1024),
        "endpoint": probe_endpoint(endpoint_url) if endpoint_url else None,
    }


if __name__ == "__main__":
    print(json.dumps(build_envelope(float(os.environ.get("BUDGET_S", 900)),
                                    os.environ.get("ENDPOINT_URL")), indent=2))

Two details matter more than the code. First, I probe reachability, not correctness: a 404 or 405 still proves that DNS, TLS, routing and the process on the other end are alive, and that is all I need before item zero. Second, cpu_count and cpu_quota_cores are reported separately on purpose, because a container can see sixteen CPUs while being allowed two, and worker pools that trust os.cpu_count() find that out the hard way.

BUDGET_S=900 ENDPOINT_URL="$MY_ENDPOINT" python envelope.py > envelope.json

The shape of the output is what you should look at here; the specific values are illustrative and will differ on your machine.

{
  "budget_s": 900.0,
  "cpu_count": 16,
  "cpu_quota_cores": 2.0,
  "tmp_free_mb": 4096,
  "endpoint": { "ok": true, "status": 200, "elapsed_ms": 180 }
}

The artifact, part 2: spend the budget instead of hoping

Once the envelope exists, the job needs a deadline object and a graceful degradation path that does not pretend to be success. The rule I settled on is simple: each item gets a fair share of the remaining budget, and if the remaining budget is smaller than that share, the job stops and reports how many items it skipped.

# guard.py
import time


class Deadline:
    def __init__(self, budget_s):
        self.end = time.monotonic() + budget_s

    def remaining(self):
        return max(0.0, self.end - time.monotonic())

    def slice_for(self, items_left, floor_s=0.5):
        return max(floor_s, self.remaining() / max(1, items_left))


def run_items(items, deadline, call_model, fallback):
    report = {"ok": 0, "degraded": 0, "skipped": 0}
    results = []
    for i, item in enumerate(items):
        left = len(items) - i
        slice_s = deadline.slice_for(left)
        if deadline.remaining() < slice_s:
            report["skipped"] += left
            break
        try:
            results.append(call_model(item, timeout_s=slice_s))
            report["ok"] += 1
        except TimeoutError:
            results.append(fallback(item))
            report["degraded"] += 1
    return results, report

Notice there is no retry loop in there, and that is deliberate rather than lazy. A single retry inside a fixed slice is fine, but an unbounded retry inside a shared deadline means item three can consume the budget of items four through forty. The fallback is the pressure valve: a deterministic heuristic, a cached answer, or a clearly marked placeholder that the report counts.

And the exit contract, which I now treat as part of the interface:

• 0 — every item came from the endpoint.
• 3 — finished, but at least one item is degraded; the run summary lists them.
• 75 — nothing useful happened: no budget, no disk, or no reachable endpoint.

The decision table I actually use

A probe is only worth running if a specific reading maps to a specific action, so here is the mapping I wrote down before I trusted any of it.

Envelope reading	What it means	Action
cpu_quota_cores below worker count	bursty parallelism, throttling	shrink the pool, keep concurrency at or under quota
tmp_free_mb under ~2x expected output	disk will fill mid-write	exit 75 before item zero
endpoint.elapsed_ms over half the per-item slice	slow path, not dead path	degrade early, do not wait for the first timeout
endpoint.ok false	routing, auth, or endpoint down	run in fallback-only mode, exit 3
budget_s below items * 0.5	arithmetic says you cannot finish	refuse to start, exit 75

That last row is the one that saved me the most time, and it took a single line of division to discover I had been starting runs that were mathematically doomed. Why does nobody check the arithmetic first? Because the arithmetic is boring, and boring checks are the ones that get skipped.

The test plan: three tests, no network

The trap with preflight code is that you test it against a healthy endpoint and call it done. I run a fake endpoint instead, so all three states are reproducible on a laptop with the Wi-Fi off.

# test_envelope.py  ->  python -m pytest -q
import threading, time
from http.server import BaseHTTPRequestHandler, HTTPServer

import envelope


class Fake(BaseHTTPRequestHandler):
    delay = 0.0

    def do_GET(self):
        time.sleep(self.delay)
        self.send_response(200)
        self.end_headers()
        self.wfile.write(b"ok")

    def log_message(self, *args):
        pass


def serve(delay):
    Fake.delay = delay
    srv = HTTPServer(("127.0.0.1", 0), Fake)
    threading.Thread(target=srv.serve_forever, daemon=True).start()
    return srv, f"http://127.0.0.1:{srv.server_port}/"


def test_healthy_endpoint():
    srv, url = serve(0.0)
    assert envelope.probe_endpoint(url, timeout_s=1.0)["ok"] is True
    srv.shutdown()


def test_slow_endpoint_reports_elapsed():
    srv, url = serve(0.4)
    probe = envelope.probe_endpoint(url, timeout_s=1.0)
    assert probe["ok"] is True and probe["elapsed_ms"] >= 400
    srv.shutdown()


def test_dead_endpoint_fails_fast():
    srv, url = serve(2.0)
    probe = envelope.probe_endpoint(url, timeout_s=0.2)
    assert probe["ok"] is False
    srv.shutdown()

The third test is the important one, because it asserts behavior rather than speed: a timeout must produce ok: false and a recorded error name, not an exception that escapes the preflight and kills the run with a traceback. Then I add one more assertion one level up — that run_items returns degraded > 0 when call_model raises TimeoutError — because the degradation path is the part that will silently rot.

What still broke, and who should not do this

The honest limitations, in the order they bit me:

1. Cgroup paths are not universal. On cgroup v1 hosts or unusual container runtimes, cpu.max simply is not there, and my probe reports null. Treat null as unknown, not as unlimited.
2. One probe cannot see the future. A two-second check proves the endpoint is reachable now; it says nothing about minute forty. The deadline and the fallback are what handle minute forty.
3. Silent degradation is worse than a crash. If a degraded result can be mistaken for a real one, do not write it into the same column — the exit code and the counted report exist so that a partial run cannot masquerade as a complete one.
4. The probe costs time. It adds its own latency to every run, which is a bad trade for a job that takes ten seconds and a good trade for one that takes twenty minutes.

If your job must never produce partial output, skip the fallback and keep only the preflight plus the 75 exit — a refusal to start is a perfectly good outcome. If you are on paid capacity with a real agreement behind it, most of this is ceremony and your provider's client library already gives you the deadlines. And if you cannot state a wall-clock budget for your job in one sentence, this whole envelope is just extra code around a decision you have not made yet.

What I would repeat from the last twelve hours

Run envelope.py on every start and keep the JSON next to the output, so that two runs can be compared with diff instead of with memory. Make the exit contract explicit and treat 3 as a real state that a caller has to handle, not as a soft success. Keep the probe under two seconds and never put a model call inside it. And write the decision table before the code, because the table is where you discover that half your failure modes needed arithmetic, not engineering.

If you want to try this against the same kind of setup that started it, MonkeyCode's free server option and free model access are cheap enough to run the three tests and the probe in an afternoon; the scripts above import nothing from them, so the envelope works the same on whatever you point it at tomorrow.