The Free Server Fell Asleep Before the Model Finished Thinking

Here is the border where free infrastructure meets slow intelligence: a server that wants to go to sleep and a model that wants to keep thinking. I spent 48 hours with a small summarizer deployed on a free server, backed by a free model, and the lesson was not about prompts or rate limits. The lesson was about what happens when the worker that should process your job gets recycled while your job is still somewhere on the wire.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I wrote these field notes around MonkeyCode's free model access and its free server option, because that combination makes the failure mode cheap to reproduce and surprisingly educational.

The setup I kept coming back to

The task was boring on purpose: accept a chunk of text, ask a free model to summarize it, store the summary in SQLite, and make the result available through a second endpoint. Nothing about that needs a queue, a database, or a lease table, which is exactly why I built the naive version first. The first cut was a FastAPI app with a background task and a Python list as the queue, and it looked clean enough to deploy.

# v1 — the version that looked fine at 2 p.m.
from fastapi import FastAPI, BackgroundTasks

app = FastAPI()
queue: list[str] = []

@app.post("/jobs/{job_id}")
async def enqueue(job_id: str, payload: str, tasks: BackgroundTasks):
    queue.append(job_id)
    tasks.add_task(summarize, job_id, payload)
    return {"accepted": job_id}

Locally, that code behaves itself. You post a payload, the model thinks for a few seconds, the summary shows up, and you move on. The problem only appears when you stop fiddling with the app and let the free server enforce its own sleep schedule. You see the trap already, right? The model is slow, the container is idle during the wait, and the platform recycles idle containers whenever it wants to reclaim memory.

What actually broke

The first night, I queued three summaries and went to make tea. When I came back, two of them had vanished from the list, and the third one had been processed twice because a retried request collided with the first attempt. Memory said the jobs existed, SQLite said they never arrived, and the model had happily summarised the same text into two different files.

This was not the model hallucinating and it was not a Docker misconfiguration. The free server had recycled the container while the model was still drafting its answer, so the background task died, the in-memory list died with it, and the client had no idea which stage of the pipeline had failed. The retry logic made it worse, because a second attempt re-ran the whole request without knowing whether the first one had partially succeeded.

What I learned in the next 24 hours is that you cannot fix this problem by making the model faster or by adding more retries. You fix it by assuming the server can vanish between any two instructions, and then designing the job lifecycle so that disappearance does not matter.

The fix: treat state like it will evaporate

Step one was moving the queue out of Python and into SQLite. A single table with a status column, a lease timestamp, and an attempt counter covers the whole lifecycle, and SQLite is already present in most Python runtimes, so there is no extra daemon to babysit.

CREATE TABLE IF NOT EXISTS jobs (
  job_id     TEXT PRIMARY KEY,
  payload    TEXT NOT NULL,
  status     TEXT NOT NULL DEFAULT 'pending',
  lease_until INTEGER NOT NULL DEFAULT 0,
  attempts   INTEGER NOT NULL DEFAULT 0
);

The key detail is the lease. Instead of locking a row forever while the model thinks, the worker claims a job with a timestamp that expires after a generous window, say 120 seconds. If the container dies in the middle of the model call, the job stays in the table with a stale lease, and the next worker that starts up can reclaim it.

# claim the next pending or expired job
def claim_next(now: int):
    return conn.execute("""
        UPDATE jobs
        SET status = 'running',
            lease_until = ?,
            attempts = attempts + 1
        WHERE job_id = (
            SELECT job_id FROM jobs
            WHERE status = 'pending'
               OR (status = 'running' AND lease_until < ?)
            ORDER BY attempts ASC
            LIMIT 1
        )
        RETURNING job_id, payload
    """, (now + 120, now)).fetchone()

On startup, I also sweep any jobs that are still marked as running but have an expired lease, and flip them back to pending. That tiny recovery step is what turned an unreliable free server into something that eventually delivered every summary, just not always on the first pass.

Idempotency keys came next

The retried request was just as destructive as the crash, because a client that resends the same payload gets a new job id unless you force it to derive one from the content. I changed the client to compute sha256(payload) and use that as the job id, then the server inserts with INSERT OR IGNORE so the second copy of the same request silently reuses the first one.

job_id = hashlib.sha256(payload.encode()).hexdigest()
conn.execute(
    "INSERT OR IGNORE INTO jobs (job_id, payload) VALUES (?, ?)",
    (job_id, payload),
)

That single convention removed the whole class of duplicates, and it cost almost nothing. The model still runs, the server still sleeps, but now a crash and a retry converge on the same row instead of spawning a parallel universe.

Warming the server up

The last piece was the least glamorous and probably the most important: keep the free server awake long enough for a real request to arrive. I added a /healthz endpoint that touches SQLite, and pointed an external cron job at it every five minutes.

@app.get("/healthz")
def healthz():
    conn.execute("SELECT 1")
    return {"alive": True}

# .github/workflows/ping.yml
on:
  schedule:
    - cron: "*/5 * * * *"
jobs:
  ping:
    runs-on: ubuntu-latest
    steps:
      - run: curl -s "https://your-app.free-server.dev/healthz"

Does a health check count as server activity? That depends on the platform, but in my notes it was enough to move many of the cold starts out of the request path. The heuristic I now use is simple: if the model takes several seconds to respond, the server needs a reason to stay warm for at least that long, and an external timer is the cheapest reason you can buy.

The decision table I wish I had on day one

Situation	Use a persistent queue with leases?
Free server + slow free model	Yes, always, because the container will vanish mid-think
Your job completes in under a second	No, just call the function directly
You have a long-running paid server	Maybe, but the crash window is much smaller
You need sub-second responses	No, don't put a slow model on any free server
You are prototyping and accept losing jobs	Skip the complexity until you notice the loss

That table is the only part of the experiment I would tattoo somewhere visible if I worked with free tiers every day, because every row traces back to the same question: how much time can elapse between "the server accepted this" and "the server finished this"?

Who should not use this approach

If your workload is a pure request-response with no background processing, the lease table is bureaucracy and nothing else. If you already have a container that stays up, you can keep the queue in memory and sleep peacefully. And if you expect the free server to behave like a paid one, then no table, idempotency key, or warm-up timer will save you from the disappointment.

This design also assumes that losing a few minutes of latency is acceptable. If a summarizer running out of order ruins your downstream pipeline, you need a real broker and real observability, not a SQLite table with a lease column.

What I would repeat

If I ran those 48 hours again, I would build the lease table first and skip the naive version entirely, because the first six hours of the experiment only taught me a lesson that everyone already knows. I would also keep the external ping, but I would make it conditional: ping harder when the queue is non-empty, and let the server sleep when there is plainly nothing to do. That is a healthier relationship with free infrastructure than either trusting or fighting it, and it is the one habit I actually kept after the experiment.

If you want to reproduce the failure quickly, point a background task at MonkeyCode's free model access and a free server, and keep the idempotency key from the start. You'll lose the next job, but you'll keep your queue.