My Demo Died When I Closed the Lid: A Small Deployment Case Study

Thursday, 11 pm, Halifax. I was screen-sharing my little retrieval demo to a classmate, felt clever for about four minutes, then shut the laptop to go make tea. When I came back she typed a question into the tab and got a spinner, then nothing. The server was my terminal. The terminal was my laptop. The laptop was asleep.

That is the whole problem, isn't it? A demo that only exists while you are physically looking at it is not a demo. So the learning question for this case study is narrow: what actually breaks first when a laptop-only RAG service moves to a free tier — the cold start, or the token budget? I wanted to find out without paying for the answer, and without guessing.

What I started with

A FastAPI endpoint, a chunk of lecture notes as context, and an answer function that did sentence-overlap ranking. No vector store, no framework, no GPU. Prerequisites are deliberately boring: Python 3.11, plus fastapi, uvicorn, and httpx. I am not going to quote exact patch versions, because yours will drift within a week — run pip freeze > requirements.lock.txt and keep that file next to the code. The article is reproducible only if you record what you actually installed.

The goal: three properties, not a feature list

I did not want a better demo. I wanted three properties I could test. First, the service must survive my laptop sleeping. Second, a single request must never be able to spend an unbounded amount of tokens. Third, the service must tell me what it did — latency, estimated cost, and why it refused.

That third one is the part students skip. We instrument for success and then act surprised when the bill, or the silence, is unexplained.

Step 1 — a service small enough to debug

Here is the whole thing. It is intentionally dumb: the "model" is a stdlib extractive ranker, so you can reproduce the failure modes with zero API calls. Swap extractive_answer for a real call when the harness is green.

# app.py
import os, re, time
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel

APP_START = time.time()
CEILING = int(os.environ.get("TOKEN_CEILING", "1200"))
app = FastAPI()

class Ask(BaseModel):
    question: str
    context: str

def estimate_tokens(text: str) -> int:
    # Deliberately conservative: assume ~3 chars per token, not ~4.
    # Over-estimating is the safe direction for a ceiling.
    return max(1, len(text) // 3 + 1)

@app.get("/health")
def health():
    return {"ok": True, "uptime_s": round(time.time() - APP_START, 1), "ceiling": CEILING}

@app.post("/ask")
def ask(payload: Ask):
    if not payload.context.strip():
        raise HTTPException(status_code=422, detail="empty_context")
    prompt = f"Context:\n{payload.context}\n\nQuestion: {payload.question}\nAnswer only from the context."
    cost = estimate_tokens(prompt)
    if cost > CEILING:
        raise HTTPException(status_code=413, detail={
            "reason": "budget_exceeded", "estimated_tokens": cost, "ceiling": CEILING})
    t0 = time.perf_counter()
    answer = extractive_answer(payload.question, payload.context)
    return {"answer": answer, "estimated_tokens": cost,
            "server_ms": round((time.perf_counter() - t0) * 1000, 1)}

def sentences(text):
    return [s.strip() for s in re.split(r"(?<=[.!?])\s+", text) if s.strip()]

def extractive_answer(question, context, top_k=2):
    q = {w for w in re.findall(r"[a-z0-9]+", question.lower()) if len(w) > 3}
    scored = []
    for s in sentences(context):
        words = {w for w in re.findall(r"[a-z0-9]+", s.lower()) if len(w) > 3}
        scored.append((len(q & words), s))
    scored.sort(key=lambda p: p[0], reverse=True)
    picked = [s for score, s in scored[:top_k] if score > 0]
    return " ".join(picked) or "no_support_in_context"

Note what the ceiling does. It does not truncate. It refuses, and the refusal is structured, so a caller can log estimated_tokens instead of shrugging at a 500.

Step 2 — the probe that measures the invisible part

A cold start is a latency you cannot see from inside your editor. So measure it from outside, against the deployed URL.

# probe.py
import sys, time, httpx

BASE = sys.argv[1] if len(sys.argv) > 1 else "http://127.0.0.1:8000"

CASES = [
    ("health_first",  lambda c: c.get(f"{BASE}/health")),
    ("health_warm",   lambda c: c.get(f"{BASE}/health")),
    ("ask_small",     lambda c: c.post(f"{BASE}/ask", json={"question": "What is a token budget?", "context": "A token budget caps how much text a request may consume. Exceeding it should fail loudly."})),
    ("ask_empty_ctx", lambda c: c.post(f"{BASE}/ask", json={"question": "Anything?", "context": "   "})),
    ("ask_over_ceiling", lambda c: c.post(f"{BASE}/ask", json={"question": "Summarize these notes in five sentences.", "context": "Lecture notes about retrieval. " * 150})),
]

with httpx.Client(timeout=60.0) as client:
    for name, fn in CASES:
        t0 = time.perf_counter()
        try:
            r = fn(client)
            ms = (time.perf_counter() - t0) * 1000
            print(f"{name:<17} {r.status_code}  {ms:8.1f} ms  {str(r.json())[:80]}")
        except httpx.HTTPError as e:
            ms = (time.perf_counter() - t0) * 1000
            print(f"{name:<17} ERR      {ms:8.1f} ms  {type(e).__name__}")

Point BASE at your deployed URL and run it twice: once right after the platform has idled your instance, once while it is warm. That difference is your real cold start. Do not trust a number I print here, because spin-down policy and region are not mine to promise.

Expected output

Against a warm local server you should see roughly this shape:

health_first      200      18.4 ms  {'ok': True, 'uptime_s': 41.2, 'ceiling': 1200}
health_warm       200       4.1 ms  {'ok': True, 'uptime_s': 41.5, 'ceiling': 1200}
ask_small         200       9.7 ms  {'answer': 'A token budget caps how much text...', 'estimated_tokens': 62}
ask_empty_ctx     422       3.2 ms  {'detail': 'empty_context'}
ask_over_ceiling  413       3.9 ms  {'detail': {'reason': 'budget_exceeded', 'estimated_tokens': 1709, ...}}

Your milliseconds will differ; that is the point of shipping the harness instead of a screenshot. The status codes and the ordering of latencies are the stable part. If ask_over_ceiling returns 200, your ceiling is not wired to anything and you have learned the most important thing of the day.

The input that broke my assumptions

The fixture that humbled me was not the giant context. It was ask_empty_ctx. My first version accepted an empty context, and the extractive answerer returned no_support_in_context — a string that looks like an answer and is not one. I had built a service that could confidently say nothing. Now it fails at 422, before the model call, before the budget math. Validation belongs at the boundary, not in the prose.

Where the free model access and free server fit

The deployment target for this case study is MonkeyCode, an open-source project. As the operator describes its current offer, there is free model access up to 10 million tokens and a free server option; verify the live terms yourself, because quotas and availability change and this article cannot promise them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What I actually used it for was unglamorous. The free server gave me a URL that outlives my lid, so probe.py had something honest to measure. The free model access let me swap extractive_answer for a real call without opening a billing account first — which matters when you are a student testing whether an idea is worth paying for at all. Being open source is the part I care about most here: I can read what the budget path does instead of trusting a dashboard that says "usage."

Results, stated carefully

I am not going to quote timings, because they belong to a region, a moment, and a spin-down policy, none of which are stable. What I will report is qualitative and reproducible. First, the cold path dominated the first request by an order of magnitude over the warm path — enough that I added a warm-up ping before demos. Second, the ceiling turned a vague worry into a 413 with a number attached, and that number is the only cost estimate I now trust, because I can read the code that produced it.

Limitations, and who should walk away

Situation	Free-tier deployment	Local only
Class demo, hours of idle	yes	no
Anything promising 99.9% uptime	no	no
Private student records	no	yes
You want to learn the failure modes	yes	partly

If your service must never sleep, this is the wrong tool. If your data cannot leave your machine, stop here. If you need sub-second latency on every request including the first, a free tier will disappoint you on principle, not on configuration. And the character-based estimator in estimate_tokens is a budget tripwire, not a tokenizer — do not bill anyone from its output.

What I understand now that I did not on Thursday

A demo is not a service until it fails in public in a way you can name. The lid closing was never the bug; the bug was that I had no way to see the difference between asleep and broken. Two small files fixed that: one refuses work it cannot afford, the other measures the work before and after the sleep.

Before you run the harness, predict something. A 4,000-character context with a 15-word question against a 1,200-token ceiling — pass or fail? My estimator says fail, at roughly 1,700. Yours might say pass, if you divide by four instead of three. Which of us is "right" depends on the model behind the call, and that disagreement is the interesting artifact, not my conclusion. Run it, tell me which fixture fooled you first.