Six things that break when a data microservice goes to production

I built the same data microservice three times before I wrote down what kept breaking. The pattern is always the same: someone has a notebook that cleans a CSV and computes summary statistics. It works. Someone else needs that as an HTTP endpoint. Three weeks later the endpoint exists, and the ticket queue starts.

This post is the list I wish someone had handed me before the first attempt. Each item is a decision that looks trivial and isn't. There is working code for all of it at the end, MIT licensed, no email required.

1. Lazy frames until the last moment, then collect once

The natural way to write a stats endpoint is pl.read_csv(path) at the top of the function. On a 2 GB file you've now allocated 2 GB of memory to answer a question that needs one pass over four columns.

Reading lazily and collecting a single time fixes most of it:

def load_frame(path):
    if path.endswith(".parquet"):
        return pl.scan_parquet(path)
    return pl.scan_csv(path)

The part that took longer to internalize: every helper function must accept a LazyFrame too. The first time someone adds an eager .collect() in the middle of a service layer "just to check a column exists", pushdown dies quietly and nothing errors. Nothing gets slower in a way you notice until a customer uploads ten gigabytes.

So the rule: services accept and return lazy frames. Only the API boundary collects.

2. Polars quantiles default to "nearest", and that is not what anyone means

This one cost me an honest hour. The median of [1, 2, 3, 4] is 2.5. Polars says 3.0:

s.quantile(0.5)                         # 3.0
s.quantile(0.5, interpolation="linear") # 2.5

If your users compute statistics for a living, this is the kind of bug they'll spot before you do, and they won't trust anything else you measure afterward. Every quantile in my engine now passes interpolation="linear" explicitly. Small fix, cheap lesson.

3. The two-second line between sync and a task queue

"Should this be a Celery task?" has a boring answer that works: if it can pass two seconds on your worst realistic dataset, it's a task. Otherwise run it inline and keep the API simple.

What surprised me is how much stays under two seconds. Descriptive stats on a million rows? Milliseconds with Polars. Making everything asynchronously queued "for scalability" just adds polling and retries and status endpoints to operations that finish before the HTTP connection would have closed anyway.

4. Validate data at the door, not at every read

I used to sprinkle validation calls through the codebase "for safety". That guarantees one of the copies drifts when the schema changes.

One Pandera contract at the ingestion boundary does the whole job:

TransactionSchema = pa.DataFrameSchema({
    "amount": Column(float, Check.in_range(-1_000_000, 1_000_000)),
    "currency": Column(str, Check.isin(["IDR", "USD", "EUR"])),
}, strict=False, coerce=True)

coerce=True quietly fixes "123". strict=False tolerates extra columns. Anything that fails gets a row-level error report instead of corrupting three tables downstream. One place to update, zero places to forget.

5. SIGTERM is the actual deadline

docker compose down sends SIGTERM. Celery's default prefetch multiplier is 4, which means a worker holds four queued tasks while the ten-second grace timer runs. Shutdown arrives mid-flight and Docker kills the container, and one of your tasks simply ceases to exist on the result backend.

Three config lines cover most of the wound:

celery_app.conf.update(
    task_track_started=True,
    worker_prefetch_multiplier=1,
    result_expires=3600,
)

worker_prefetch_multiplier=1 is the one that matters. Without it your graceful shutdown is graceful in name only.

6. Humans get JWTs, machines get keys, and never confuse them

I tried a single unified auth system once. It lasted until I found myself issuing a bearer token to a cron job. Two dependency-injectable layers, from that day on:

• require_jwt, issued at register/login, thirty minutes, for people.
• require_api_key, a static key from env, rotation by redeploy, for services.

Both are plain FastAPI dependencies. Per-route mixing is one import away, and neither type ever needs what the other provides.

Where the code lives

Everything above, plus a multi-stage Dockerfile, compose with healthchecks, a pre-flight script that catches most first-run failures, and a test suite covering the auth-upload-stats slice, is in one repository, MIT licensed:

github.com/FauziMatakupan/fastapi-data

Reading the code is honestly the best way to evaluate these decisions. Every module is small enough to read in one sitting, and the tests show what each endpoint actually guarantees rather than what the docs claim. If a project eventually needs auth, schema contracts, and background jobs wired together, the docs on matakupan-labs.vercel.app cover how the pieces fit.

Take the decisions even if you never clone the repo. They're the expensive part.