I Chased Vanishing Jobs for 48 Hours. Nobody Owned the asyncio.Task.

Have you ever shipped a background job that only failed after you stopped watching the terminal? I did, and the quiet remote process turned that silence into the entire debugging problem. I needed a tiny Python worker that accepted a batch of URLs and fetched them without blocking the handler. Locally the happy path printed every completion line, so I trusted it, which was my first mistake.

This writeup is a field notebook, not a launch post, and it should still teach if you delete the tool names. I will walk through what I tried, what actually broke, and the small test I now run first. Any coding assistant that emits asyncio.create_task without an owner will recreate this same incident.

The setup I thought was boring

I asked a coding assistant for a small async handler that kicked work into the loop and returned 202 immediately. The generated sketch used asyncio.create_task, which looks responsible until you reread the standard library warning. I pasted it into a local virtualenv, hit the endpoint a few times, and watched four completion lines appear in order. Why would I doubt a helper that the docs themselves recommend for scheduling work?

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I used MonkeyCode's free model access for the first draft of the worker, then ran the same file on its free server option because my laptop never collected garbage at an interesting time. Swap those two steps for any assistant and any quieter host and the lesson does not change. I am not claiming a quota, a model name, a hardware profile, or a benchmark. I am claiming a reproduction habit that my noisy laptop kept hiding.

Hour 0–8: the generated worker looked fine

Here is the shape of the code I was handed, labeled as a broken sketch rather than production advice. Notice that the handler never keeps the Task objects it creates. The event loop only holds weak references, so the collector is allowed to drop work that nobody owns.

# Broken sketch — do not copy into production.
# CPython keeps weak references to scheduled tasks.
import asyncio
from fastapi import FastAPI

app = FastAPI()

async def fetch_one(url: str) -> None:
    await asyncio.sleep(0.5)
    print(f"done {url}", flush=True)

@app.post("/batch")
async def batch(urls: list[str]) -> dict:
    for url in urls:
        asyncio.create_task(fetch_one(url))  # nobody owns this Task
    return {"accepted": len(urls)}

Does that read clean to you on a first pass? It did to me, because the handler returns quickly and the coroutine appears to be scheduled. On my laptop I never left the process idle long enough for the collector to notice the unreferenced tasks. I even ran a sloppy manual check and called the output a test.

curl -s -X POST http://127.0.0.1:8000/batch \
  -H 'content-type: application/json' \
  -d '["https://example.com/a","https://example.com/b"]'

Two done lines showed up under my cursor, so I shipped the same file to a quieter process. Would you have called that curl a test, or would you have admitted it was only a demo? I called it a test, and that decision cost me the next day.

Hour 8–24: logs that lied by omission

On the quieter remote process, the same curl often returned {"accepted": 2} with zero follow-up prints after the response. No traceback landed in stderr. No warning appeared unless I later enabled loop debug mode. Have you spent hours hunting an exception that the runtime never raised in the first place?

I chased HTTP timeouts, DNS, and outbound TLS first, because those failures usually shout. None of them were guilty, and ss -lntp still showed the listen socket in good health. A debug endpoint that dumped asyncio.all_tasks() showed a shrinking set that I kept misreading as finished work. The jobs were not finishing on schedule. They were disappearing before fetch_one ever reached the print.

CPython documents this directly: save a strong reference until the task completes, because the loop will not do that for you. I had read that sentence on previous projects and still treated it like trivia. I had not believed it would matter on a service that stayed up and accepted batches.

Hour 24–40: what I tried, and what broke

I am going to be blunt about the dead ends, because they ate most of the forty-eight hours. Each one felt scientific in the moment and taught me almost nothing about ownership.

1. I added more logging inside fetch_one. The extra lines never ran, which I misread as a logging configuration bug on the remote process.
2. I wrapped the coroutine in asyncio.wait_for. Timeouts cannot fire on a task the collector already dropped, so the wrapper never got a vote.
3. I asked the assistant to make the worker more robust. It added try/except Exception around create_task, which catches nothing about lifetime.
4. I switched from print to logging.info. I still got silence, only with a timestamp format I could paste into Slack.
5. I enabled PYTHONASYNCIODEBUG=1. That finally printed the destroyed-pending warning, and I felt extremely slow.

These are the commands that actually moved the investigation instead of decorating it. I now keep them in the same notes file as the worker.

PYTHONASYNCIODEBUG=1 uvicorn worker:app --host 0.0.0.0 --port 8000
python -c "import sys; print(sys.version)"

# Debug sketch — dump live tasks from a quiet process.
@app.get("/debug/tasks")
async def dump_tasks() -> dict:
    tasks = asyncio.all_tasks()
    return {
        "count": len(tasks),
        "names": sorted(t.get_name() for t in tasks),
    }

What broke in my own head was the assumption that accepted meant owned by someone still alive. Returning 202 is a promise to the client about work you intend to finish. Calling create_task without a container is a promise to nobody, and the collector is allowed to believe you.

The fix I would actually repeat

The standard pattern is boring, which is why I now paste it before I accept any generated async worker. Keep a strong set, discard each task on completion, and treat a missing owner as a review blocker. Fire-and-forget still needs a living Python object.

# Held-task sketch — strong references until completion.
import asyncio
from fastapi import FastAPI

app = FastAPI()
_background: set[asyncio.Task] = set()

async def fetch_one(url: str) -> None:
    await asyncio.sleep(0.5)
    print(f"done {url}", flush=True)

def _spawn(url: str) -> asyncio.Task:
    task = asyncio.create_task(fetch_one(url), name=f"fetch:{url}")
    _background.add(task)
    task.add_done_callback(_background.discard)
    return task

@app.post("/batch")
async def batch(urls: list[str]) -> dict:
    for url in urls:
        _spawn(url)
    return {"accepted": len(urls), "live": len(_background)}

For bounded work inside one request, I prefer asyncio.TaskGroup on Python 3.11 and later. It cancels siblings on failure and does not depend on a process-global set that reviewers can forget to update. Would I still return 202 with a TaskGroup? Only if I move that group into a long-lived owner task that itself sits in _background.

# TaskGroup sketch — scoped ownership, labeled as the happier path.
async def fetch_batch(urls: list[str]) -> list[None]:
    async with asyncio.TaskGroup() as tg:
        tasks = [tg.create_task(fetch_one(url)) for url in urls]
    return [t.result() for t in tasks]

There is no third way that stays honest with the client. Either the handler waits, or some other object that outlives the handler holds the tasks. Assistants like the missing third way because it types faster than an owner set.

A small reproducible check

I now keep a unit that records the ownership contract instead of pretending garbage collection is deterministic. It will not prove a task always vanishes on every host. It will fail me when I delete the set and still claim the worker is safe.

# test_task_ownership.py — contract, not a microbenchmark.
import asyncio

async def _work(flag: dict) -> None:
    await asyncio.sleep(0)
    flag["ran"] = True

def test_unowned_create_task_is_not_an_api():
    """Document the footgun. Do not assert GC timing as a contract."""
    source = open(__file__, encoding="utf-8").read()
    assert "asyncio.create_task(_work(flag))" not in source.split(
        "def test_owned_task_runs"
    )[1]

def test_owned_task_runs():
    flag = {"ran": False}
    owned: set[asyncio.Task] = set()

    async def _inner() -> None:
        task = asyncio.create_task(_work(flag))
        owned.add(task)
        task.add_done_callback(owned.discard)
        await task

    asyncio.run(_inner())
    assert flag["ran"] is True
    assert not owned

Run that file with a boring command and keep it next to the worker, not in a wiki nobody opens. If a later edit removes the set and the done callback, the second test still needs a reviewer who greps for create_task(.

pytest -q test_task_ownership.py
rg "create_task\(|ensure_future\(" -n .

Is a grep glamorous? No, and that is the point I keep repeating to myself after this incident. Generated async code fails in the lifetime details, not in the happy-path print that fooled me for the first eight hours.

Decision table I keep next to the worker

I print this table in the pull request when the diff touches scheduling. It is the artifact I actually reuse, more than any prompt I might save.

Situation	Do not use	Prefer	Why
One request, wait for every URL	bare create_task	TaskGroup or gather	Ownership stays scoped to the handler
Return 202 and keep working	bare create_task	module-level set plus done callback	The loop only holds weak refs
Need results later	time.sleep inside async code	await plus a queue	Blocking sleep starves the loop
Debug a quiet host	extra print only	PYTHONASYNCIODEBUG=1 and all_tasks()	Destroyed tasks do not log by default
Review generated code	"looks idiomatic"	search for create_task( without assignment	Assistants repeat this pattern

What I would repeat next time

I would still let an assistant draft the handler, because the routing boilerplate is not where I burned the forty-eight hours. I would not let it decide task lifetime, and I would not treat a local curl as evidence. Before I call a remote host a run, I now walk three boring checks in a fixed order.

1. Grep the diff for create_task and ensure_future, and demand a named owner for every hit.
2. Boot with PYTHONASYNCIODEBUG=1, hit the batch endpoint once, and read stderr instead of stdout.
3. Hit /debug/tasks after the response and confirm live work still exists before any done line.

Would a quieter machine have found this faster than my laptop? Yes, and that is the only reason a free remote process mattered in these notes. My laptop hid the bug by staying idle and generous with memory. The remote process was not magical, and I will not invent a spec for it. It was simply less polite about collecting unreferenced tasks.

Limitations, and who should skip this

This notebook is about task ownership, not about HTTP at scale, and I did not measure throughput. TaskGroup needs Python 3.11 or later, so older runtimes should stay on the explicit set. A global set can leak if you forget the done callback, which is a different production incident with the opposite symptom.

Do not use this approach if you need durable jobs that survive a process restart, because memory-backed asyncio tasks are not a queue. Use a real broker when the work must outlive the worker, and skip fire-and-forget when the client must learn about failure. If you are writing a short script that runs one coroutine and exits, asyncio.run already owns the main task and you do not need this set.

If you do not control the event loop, including some desktop GUI frameworks, copy none of these sketches without reading that framework's own docs. The assistant will happily generate another unowned create_task tomorrow morning. The grep still takes a few seconds, and that is the whole workflow I trust after this incident.