Serving a heavy model without blowing up the server bill

An AI model is heavy and hungry. Served carelessly, it blows up response times and the server bill. On an audio-analysis SaaS that I designed and shipped solo, it had to stay fast, secure and economically viable.

The hosting decision. I served the model on a server I run, with NGINX and automatic file cleanup, rather than on usage-billed infrastructure. It means more deployment work, but the bill no longer follows the number of analyses. It can be forecast.

On the code side, two rules matter for a service like this.

1. Load the model once, at startup. Not on every request. With FastAPI, that's what the lifespan parameter is for. The application doesn't receive requests until startup is finished.

from contextlib import asynccontextmanager
from fastapi import FastAPI

models = {}

@asynccontextmanager
async def lifespan(app: FastAPI):
    models["main"] = load_model()  # slow: done once
    yield
    models.clear()

app = FastAPI(lifespan=lifespan)

The old way, on_event, is deprecated.

2. Don't block the server during a computation. An async def function that computes for a long time blocks the whole application. A def function is run by FastAPI in a thread pool.

@app.post("/analyze")
def analyze(file: UploadFile):  # def: runs in a thread pool
    return models["main"].predict(file)

That's how the model runs in production on this SaaS.

Amine Affif, full-stack developer in Paris. amineaffif.com