48-Hour Field Notes: My Job Had 30 Seconds. I Gave the Model 8 and Kept the Rest.

My batch job had a 30-second deadline, and the model call happily ate all 30 of them. The scheduler killed the worker mid-flight, the log ended on a half-written line, and nobody downstream could tell whether the request had even reached the endpoint. Here is the part that stung: the request was not slow because the provider was slow. It was slow because I never told anyone how much time the call was allowed to take.

So I spent 48 hours on a single narrow question. If a job has a wall-clock budget, how do I hand a slice of it to a model call, keep the rest for my own work, and still return something useful when the slice runs out? These are the field notes: what I tried, what broke, and what I would repeat.

What I measured before I changed anything

I added phase timings using time.monotonic() and printed one structured line per phase, which took about twenty minutes and told me more than a week of guessing. The pattern was obvious immediately: the planning call finished in roughly two seconds on a good run, and the extraction call absorbed whatever was left. Wall-clock timeouts on the HTTP client did nothing, because my client library quietly retried twice inside the timeout I thought I had set.

That is the first lesson, and it has nothing to do with AI. A socket timeout bounds one attempt, not the operation. If you want a real bound, you have to own the clock above the client.

phase=plan    budget_ms=9000 used_ms=7312 ok
phase=extract budget_ms=4000 used_ms=4000 status=timeout fallback=regex
job total_ms=29871 within_budget=true

The artifact: a deadline object, not a pile of timeouts

I wanted one object that owns the total budget and hands out slices, so every call site asks permission instead of computing its own numbers. The version below is the one I kept; it is short on purpose, because anything longer stops being used.

# deadline.py
import time
from dataclasses import dataclass, field

class DeadlineExceeded(RuntimeError):
    pass

@dataclass
class Deadline:
    total_s: float
    _start: float = field(default_factory=time.monotonic)

    def elapsed(self) -> float:
        return time.monotonic() - self._start

    def remaining(self) -> float:
        return max(0.0, self.total_s - self.elapsed())

    def slice(self, fraction: float, floor: float = 0.5) -> float:
        """Reserve a fraction of what is left, but never less than floor seconds."""
        return max(floor, self.remaining() * fraction)

    def check(self, phase: str) -> None:
        if self.remaining() <= 0:
            raise DeadlineExceeded(phase)

Call sites then read like a budget conversation rather than a magic number. The plan phase gets about a third of the window, extraction gets a quarter of what survives, and a guard runs between them.

# pipeline.py
from deadline import Deadline, DeadlineExceeded

TOTAL_BUDGET_S = 30.0

def run(prompt: str, call_model) -> dict:
    dl = Deadline(total_s=TOTAL_BUDGET_S)

    plan_budget = dl.slice(0.30)
    plan_text = call_model(prompt, timeout_s=plan_budget)   # single attempt, no internal retry
    dl.check("plan")

    extract_budget = dl.slice(0.25)
    try:
        fields = call_model(plan_text, timeout_s=extract_budget)
    except TimeoutError:
        fields = regex_fallback(plan_text)                  # degraded, still returns
    dl.check("extract")

    return {"fields": fields, "elapsed_s": round(dl.elapsed(), 3),
            "budget_s": TOTAL_BUDGET_S}

Keeping the test offline and deterministic

A budget harness you cannot test without paying for calls is a harness you will stop testing. I replaced the endpoint with a fake that sleeps for whatever timeout it is given, so a timeout path can be exercised in under a second of real time.

# test_deadline.py
import time
import pytest
from deadline import Deadline, DeadlineExceeded
from pipeline import run

def slow_model(prompt: str, timeout_s: float) -> str:
    time.sleep(min(timeout_s, 60.0))
    return '{"ok": true}'

def test_total_runtime_stays_inside_budget():
    t0 = time.monotonic()
    result = run("extract fields", slow_model)
    wall = time.monotonic() - t0
    assert wall <= 30.5                       # budget plus a small scheduling slack
    assert result["elapsed_s"] <= 30.0

def test_empty_deadline_raises_instead_of_calling():
    dl = Deadline(total_s=0.0)
    with pytest.raises(DeadlineExceeded):
        dl.check("plan")

Run it with pytest -q test_deadline.py -k budget and it finishes in a couple of seconds. That matters more than it sounds: the moment a test suite needs a live endpoint, people skip it locally and the guarantee rots.

Day two: the four things that broke

1. The client retried inside my timeout. I set timeout=4 and saw a phase take twelve seconds. Fix: disable library-level retries, keep one attempt per phase, and let the wrapper decide whether a retry is affordable.
2. Cancellation did not stop the work. Abandoning a thread does not kill an in-flight socket read, so the phase "ended" while the request kept burning time in the background. Fix: pass the remaining budget as the client timeout so the transport gives up on its own.
3. My guard ran after the expensive step. Checking the deadline after the call is theater; the check has to sit between phases, which is why dl.check("plan") exists.
4. Degraded output looked identical to good output. The fallback returned a dict with the same shape and worse values, and a downstream consumer accepted it silently. Fix: an explicit degraded: true flag plus a metric, so the fallback is visible instead of plausible.

That last one is the failure I would have shipped without the notes. Valid-looking data is more dangerous than a clean exception, and no schema validator will save you from a field that is technically a string.

When to use this, and when to use something else

| Situation | Right tool | Why |
| --- | --- |
| Hard wall-clock deadline (scheduler, HTTP request, queue worker) | Deadline budget from this article | Only a total budget bounds the whole operation |
| Transient 5xx on an otherwise healthy endpoint | Bounded retries with jitter | The problem is flakiness, not time |
| Interactive UI where the user is waiting | Stream partial output, budget on first token | Users tolerate slow better than silent |
| Offline batch job with hours to spare | Retries plus alerting, no tight budget | Throughput matters more than latency |
| Anything where a wrong answer is worse than no answer | Skip the fallback entirely | Degraded output is a product decision, not a code one |

Where free model access and a free server fit

I ran these experiments on MonkeyCode's free model access and its free server option, which the project provides for evaluation. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am not going to quote a token allowance or machine size here, because free tiers change and a number copied today becomes wrong next month; check the project's own page for the current limits before you size a budget around them. What I can say is that a deadline harness is exactly the kind of thing a free tier rewards, since a job that finishes inside its window is cheaper to run than one killed at the boundary.

There is a pleasant side effect worth mentioning. Once the budget object exists, swapping the model behind it becomes a one-line change, and comparing two endpoints on the same workload stops being a research project.

Limitations, and who should not bother

This approach does not make a slow endpoint fast, and it will not tell you whether your prompts are any good. It adds roughly forty lines and one concept, and it only pays off if you actually have a deadline to defend. If your job can run for an hour and nobody notices, skip the harness and spend the time on output quality instead. If you are in Python with thread-based I/O and no way to cancel in-flight requests, budget the transport timeout carefully, because the guard alone will not stop the work.

• Not suitable for genuinely unbounded batch pipelines.
• Not a substitute for rate-limit handling; those are different failures.
• Not safe for correctness-critical paths where a fallback must never appear.

What I would repeat on Monday

I would write the phase-timing log first, before any refactor, because two hours of measured data beat two days of intuition. I would build the offline fake before the real integration, so the timeout path is tested once and never again by hand. And I would keep the degraded flag loud in logs and metrics, because the quietest failure I hit was a fallback that looked exactly like success. What is your job's real budget, and who owns the clock right now?

If you want to try the same harness against a free tier, the project is on GitHub, and I would start by measuring your own slowest phase rather than trusting my numbers.