The Deadline Used time.time(). Then the Laptop Slept.

I spent forty-eight hours chasing a worker that looked idle while its deadline quietly expired overnight. Have you ever watched a timeout fire too early and immediately blamed the network instead of your clock? I did that, then I blamed NTP, then I blamed Docker, and only later I blamed my own helper. This is the field notebook from that hunt, including the runnable code I would actually keep.

Hour 0–8: The symptom that looked like a hang

The worker polled a queue, processed a job, and used a remaining-seconds budget so a stuck HTTP call could not run forever. That budget started as deadline = time.time() + 30, which looks boring and correct until a wall clock steps. Why would anyone distrust time.time() when almost every timeout tutorial still uses it without a single warning? Because wall time is a social agreement, not a stopwatch, and my process treated it like a stopwatch.

Symptoms I wrote down before I understood them still look embarrassing, and that is why I am keeping the list. I kept treating the negative remaining value as a formatter bug instead of a physics bug. Would you have stopped at the first -1842.7 in a log, or would you have kept scrolling too?

• A job that should have run for twenty seconds returned immediately after my laptop woke from sleep.
• A second job sat inside an HTTP call far longer than the budget I thought I had configured.
• Unit tests on my machine passed because they never slept, never warped the clock, and never crossed midnight.
• Logs showed remaining=-1842.7 once, which is not a value any thirty-second budget should ever honestly print.

That negative remaining time should have been the clue, yet I filed it under logging and kept going. I even padded the log format with extra decimals, as if precision could make a jumped clock look civilized. Have you noticed how a pretty log line can hide a physically impossible number in plain sight?

Hour 8–20: What I tried before I touched the clock

I did the usual noisy debugging during this stretch, and almost none of it actually touched the clock. The more tools I opened, the more convinced I became that sockets, not timestamps, were lying to me. That is a bad bargain, because extra instrumentation feels like progress while the theory stays wrong. Would another capture file have helped, or would it have just postponed the embarrassing clock question?

What I actually tried:

1. I added prints around the HTTP client, then swapped libraries, then wrapped the call in a retry that still used wall time.
2. I stared at packet traces long enough to prove the socket was quiet, which only deepened the haunted-network story.
3. I checked container CPU limits after every lid-open freeze, because Docker is an easy villain when a laptop sleeps.
4. I grepped for sleep( because an older note in this series had been an event-loop wait, and I wanted no sequel.

None of those changes moved the remaining-time field in the log, which should have been a louder alarm. Commands I ran while still defending the wrong theory are below, and the first one already contained the answer. I just did not read it as an answer yet.

python -c "import time; print('wall', time.time()); print('mono', time.monotonic()); print('perf', time.perf_counter())"
pmset -g log | grep -i 'wake\|sleep' | tail -n 20
journalctl -u systemd-timesyncd --no-pager | tail -n 40

time.time() returns seconds since the Unix epoch, and that value can jump forward or backward. time.monotonic() returns an arbitrary baseline that only moves forward, which is what a duration budget actually wants. time.perf_counter() is also monotonic and better for short intervals, but a thirty-second HTTP budget does not need nanosecond theater. Three clocks, three jobs, and I had been using the calendar as a stopwatch.

Hour 20–32: Moving the repro off a sleeping laptop

My laptop kept sleeping, so the wall clock jumped forward by hours, and every deadline looked expired at once. I wanted a machine that stayed awake while I iterated on the helper, without turning my notebook into an NTP lab. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode's free server option as the always-on box during this overnight clock-sensitive repro.

I used the free model access only to draft extra failing tests, which I then edited by hand. Did the model save the design of the helper, or did it just repeat the same wall-clock pattern? The first helper it drafted still called time.time(), because that pattern sits in nearly every snippet dump. I almost merged it, which would have reproduced the laptop bug on a machine that never sleeps.

I will not invent model names, quotas, or hardware claims, because I do not have those figures. The spare machine mattered because it did not sleep when I closed a lid. The draft tests mattered only after I rewrote their clock. Everything else in this notebook should still work if you never touch that product at all.

The smallest reproducer I kept

Here is the lab I wish I had written at hour one, with an injectable clock and no warped system time. It is ordinary pytest, and it does not require stepping NTP on a shared host. If you cannot inject now, you will start mocking time.time globally, and that bleed is a second outage.

# deadline_lab.py
from __future__ import annotations

from dataclasses import dataclass
from typing import Callable

Now = Callable[[], float]


@dataclass
class WallDeadline:
    """Broken on purpose: wall clocks can step forward or backward."""

    expires_at: float

    @classmethod
    def after(cls, seconds: float, now: Now) -> "WallDeadline":
        return cls(expires_at=now() + seconds)

    def remaining(self, now: Now) -> float:
        return self.expires_at - now()


@dataclass
class MonoDeadline:
    """Duration budget. Do not use this for 'run at 15:00 local'."""

    expires_at: float

    @classmethod
    def after(cls, seconds: float, now: Now) -> "MonoDeadline":
        return cls(expires_at=now() + seconds)

    def remaining(self, now: Now) -> float:
        return self.expires_at - now()

# test_deadline_lab.py
from deadline_lab import MonoDeadline, WallDeadline


class FakeClock:
    def __init__(self, start: float) -> None:
        self.value = start

    def __call__(self) -> float:
        return self.value

    def step(self, delta: float) -> None:
        self.value += delta


def test_wall_deadline_expires_when_the_clock_steps_forward():
    clock = FakeClock(1_700_000_000.0)
    deadline = WallDeadline.after(30, now=clock)
    clock.step(3600)  # laptop wake, or a hard NTP step
    left = deadline.remaining(now=clock)
    assert left < 0
    assert left < -3500


def test_wall_deadline_grows_when_the_clock_steps_backward():
    clock = FakeClock(1_700_000_000.0)
    deadline = WallDeadline.after(30, now=clock)
    clock.step(-120)  # NTP stepped backward
    assert deadline.remaining(now=clock) == 150


def test_mono_deadline_ignores_a_wall_step():
    mono = FakeClock(100.0)
    wall = FakeClock(1_700_000_000.0)
    deadline = MonoDeadline.after(30, now=mono)
    wall.step(3600)  # wall jumps; monotonic does not
    assert abs(deadline.remaining(now=mono) - 30) < 1e-9
    mono.step(30)
    assert deadline.remaining(now=mono) <= 0

Run the lab like this, and keep the live snippet for a sanity check after the tests pass:

pytest -q test_deadline_lab.py
python - <<'PY'
import time
from deadline_lab import MonoDeadline

mono = time.monotonic
d = MonoDeadline.after(5, now=mono)
time.sleep(1)
print('remaining', round(d.remaining(now=mono), 2))
PY

The first two tests document the failure I lived with, including the backward step that inflated a hang. The third test is the behavior I wanted from the beginning, and it never converts monotonic ticks into a human timestamp. That conversion is a lie, because the monotonic origin is undefined on purpose.

What actually broke

Three separate mechanisms stepped time.time() under me, and I had conflated them into one vague clock story. Splitting them was the first useful hour in the second day. If your notes still say "time is weird," you are probably mixing at least two of these.

1. Laptop sleep and wake

Closing the lid suspends the machine, and the wall clock usually jumps to real-world time at wake. A time.time() + 30 deadline created before sleep is already expired after a forty-minute nap. On many Linux hosts, time.monotonic() follows CLOCK_MONOTONIC and does not count suspend time, which is closer to process runtime. That mismatch alone explains a job that vanished immediately after the lid opened.

2. NTP stepping instead of slewing

When offset is large, a time daemon may step the clock instead of slewing it slowly. A forward step burns your remaining budget and looks like an instant timeout. A backward step inflates the budget, which is how an HTTP call outlived the timeout I believed I had. Adding more sleep() around that path made the hang worse, because I was feeding the wrong clock.

3. Mixing wall stamps with duration math

I stored expires_at as an epoch float and logged it next to datetime.now(), which made the line look authoritative. A duration budget should be logged as remaining seconds from monotonic time. If humans need a clock, log a separate timezone-aware wall stamp and never subtract the two. Different origins plus different jump rules will invent numbers that cannot exist.

Decision table I now keep above the helper:

Question you are asking	Clock to use	Why
How many seconds may this call still run?	time.monotonic()	Duration, never a calendar
When should this job appear on a dashboard?	timezone-aware wall time	Humans and audit logs
How long did this function take in a microbenchmark?	time.perf_counter()	Fine-grained elapsed time
Did the certificate expire yet?	aware UTC wall time	Certificates live on calendars
Can I subtract time.time() from a monotonic value?	never	Different origins, different jumps

A helper I would actually ship

The injectable now callable is the whole trick, because tests should not warp the real system clock. Production code should pass remaining into socket timeouts instead of spinning a wait loop. Cooperative budgets still cannot save you from a blocking C extension that never checks TimeoutError.

# budget.py
from __future__ import annotations

import time
from collections.abc import Callable
from dataclasses import dataclass

Now = Callable[[], float]


@dataclass(frozen=True)
class Budget:
    _expires_mono: float
    _now: Now

    @classmethod
    def seconds(cls, duration: float, now: Now = time.monotonic) -> "Budget":
        if duration < 0:
            raise ValueError("duration must be >= 0")
        return cls(_expires_mono=now() + duration, _now=now)

    def remaining(self) -> float:
        return max(0.0, self._expires_mono - self._now())

    def expired(self) -> bool:
        return self.remaining() <= 0

    def split(self, slice_seconds: float) -> float:
        """Cap one subcall so a loop cannot starve the outer budget."""
        return max(0.0, min(slice_seconds, self.remaining()))

import socket
from budget import Budget


def fetch_with_budget(sock: socket.socket, payload: bytes, total: float) -> bytes:
    budget = Budget.seconds(total)
    sock.settimeout(budget.split(5.0))
    sock.sendall(payload)
    chunks: list[bytes] = []
    while not budget.expired():
        sock.settimeout(budget.split(5.0))
        try:
            piece = sock.recv(4096)
        except TimeoutError:
            break
        if not piece:
            break
        chunks.append(piece)
    return b"".join(chunks)

Is this the only correct design for every worker? No, and I still want a hard process-level timeout around code that ignores cancellation. A budget is a conversation with I/O, not a kernel guarantee. If you need the latter, use a supervisor that can kill the process, then treat this helper as the polite inner layer.

Hour 32–48: What I would repeat, and what I would not

I would repeat the fake-clock tests, and I would not repeat the packet capture as my first move. I would also repeat moving the repro onto a machine that does not sleep when I close a lid. I would not repeat asking a model for a timeout helper and accepting time.time() because the snippet compiled.

Checklist I now run before I trust a deadline:

1. Is this a duration or a calendar event? If duration, monotonic. If calendar, aware UTC.
2. Can I inject now in tests without mocking the entire time module globally?
3. Do logs print remaining seconds, not a raw epoch mixed with local time?
4. After suspend, do I reset budgets for work that should not include sleep time?
5. Does every I/O call receive a finite timeout derived from remaining(), never None?

Limitations, and who should not use this

This approach is wrong for calendar work, and I want that limitation in the same notebook as the helper. Scheduling "send this email at 09:00 in Chicago" needs zoneinfo and wall time, not monotonic ticks. Measuring whether a human waited too long in real life may need both clocks, because people live on calendars while processes live on stopwatches.

Coordinating expiry across two machines still needs signed wall timestamps, because monotonic origins are process-local and not comparable. People who cannot write a fake clock should not mock time.time globally in a large suite, because the bleed is worse than the original bug. Do not use this pattern as a substitute for certificate checks, cookie expiry, or any other document that is legally a date.

On Linux, time.monotonic() often pauses across suspend, so it will not punish a worker for a closed laptop lid. If you needed "thirty real-world seconds including sleep," you wanted wall time or a boottime clock, and monotonic will look too generous after wake. I am not wrapping that edge in fake precision; measure it on the kernel you ship, then keep the injectable now so the test can name the behavior.

If you only remember one line from these notes, make it this one. Timeouts are stopwatches, and time.time() is a calendar. Would I park the next clock-sensitive repro on an always-on machine instead of a sleeping laptop? Yes, and I would start with the fake clock this time.