Fit FastAPI AI Apps in 256MB by Streaming Remote APIs

Free-tier AI apps die with Out of Memory because a local model plus FastAPI easily exceeds 1GB of RAM. I keep the server as a thin streaming proxy to a remote model API so baseline RSS stays around 50–100MB, and a 256MB free tier stays up.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Memory, Not CPU, Is What Kills Free-Tier AI Apps

Free server tiers typically offer 256MB or 512MB of RAM, a shared CPU, and a small disk. For a low-traffic prototype, CPU and bandwidth almost never take the process down. Memory does. When RSS walks past the limit, the OS kills Python with no stack trace that points at transformers or a buffered response.

I have watched a “simple” summarizer import torch, load a checkpoint, and sit above 1GB before the first request. That is not a hosting problem; it is a design problem. Buying a bigger box only hides it. I treat 256MB as a hard budget: the web process stays stateless, and inference happens on a remote API.

Side by side, the two shapes of an AI app are not close:

Approach	Baseline memory	ML libraries loaded	Fits a 256MB free tier?
Local model inference	1GB+	Yes (torch, transformers)	No
Remote model API + streaming	~50–100MB	No	Yes

The second row is the whole strategy: do not load the model where the request lands. Extra uvicorn workers multiply RSS, so on a 256MB box I run a single worker.

A practical contrast I use when reviewing a prototype:

• Local path: from transformers import pipeline at import time — RSS jumps before app even starts.
• Remote path: import httpx only — the process is a JSON-and-bytes shuttle.
• Buffering path: collect every SSE chunk into a list, then return — peak RSS tracks output length.
• Streaming path: yield each line — RSS stays flat on a short summary and on a long chat.

Three Principles I Use to Stay Under 256MB

I apply these in order: first remove the model from the process, then stop buffering, then cut packages until the memory probe looks boring.

1. Load nothing I do not need at startup

I never import transformers or torch on the free-tier server. Those imports dominate RSS before from_pretrained even runs. HTTP calls to a model API turn the app into a proxy: validate the payload, forward it, stream the result. That one decision cuts baseline memory by about an order of magnitude versus local inference.

In practice I keep torch, tensorflow, transformers, and onnxruntime out of the web image. The model lives behind MODEL_URL; the server image stays ignorant of checkpoints.

2. Stream every token

If I buffer a full completion, I pay for peak size. A long summary or a chatty reply then becomes a liability on a 256MB box. Streaming tokens as they arrive keeps RSS flat regardless of output length. That matters for summarization and chat, where hundreds of tokens are normal.

I keep stream: True on the upstream request and yield lines as httpx delivers them. I do not join chunks into a string “for logging.” If I need logs, I log status and latency—not the full text.

3. Audit every import

Each extra package raises the floor. I measure RSS after startup, then again after a few requests. Unused packages go; standard-library tools stay. A lean requirements.txt is the cheapest way to stay under a free-tier ceiling.

My audit loop:

1. Boot the app and print RSS.
2. Hit /summarize with short and long inputs.
3. Print RSS again. If it climbed, I look for caches, global lists, or a client that was never closed.
4. Remove one dependency, rebuild, and repeat until the number is boring.

psutil is for profiling only. It does not ship in the production image.

A Minimal FastAPI Summarizer That Never Loads a Model

This app summarizes text by calling a remote chat-completions API. It loads no ML libraries, so baseline memory stays tiny, and it streams tokens back to the client.

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
import httpx
import os

app = FastAPI()
MODEL_URL = os.getenv("MODEL_URL", "https://api.example.com/v1/chat/completions")

@app.post("/summarize")
async def summarize(payload: dict):
    text = payload["text"]

    async def generate():
        async with httpx.AsyncClient(timeout=30) as client:
            try:
                async with client.stream("POST", MODEL_URL, json={
                    "model": "free-model",  # verify the model name in the docs
                    "messages": [
                        {"role": "user", "content": f"Summarize this in 3 bullets:\n{text}"}
                    ],
                    "stream": True,
                }) as response:
                    async for line in response.aiter_lines():
                        if line.startswith("data: "):
                            yield line[6:] + "\n"
            except httpx.HTTPError as exc:
                yield f"error: {exc}\n"

    return StreamingResponse(generate(), media_type="text/plain")

Notice what is missing: no transformers, no torch. Runtime dependencies are fastapi, httpx, and uvicorn. The remote model does the compute; this process only shuttles bytes. If I strip the base image far enough, I can run this shape on 128MB of RAM. On a 256MB free tier there is actual headroom.

How I wire it in practice:

• Put the endpoint in MODEL_URL so the image is not tied to one vendor.
• Keep stream: True upstream and yield SSE data: lines as they arrive.
• Use a 30-second timeout so a hung upstream cannot pin the worker forever.
• Do not accumulate the full summary in a list “just to log it”—that undoes the streaming win.
• Validate that text exists and cap its length before calling the API so a huge payload cannot balloon memory in the proxy itself.

Compared with a local pipeline("summarization") service, this file has no weights, no CUDA check, and no tokenizer. The tradeoff is network latency and a dependency on an external API. For a free-tier prototype I accept that tradeoff every time.

Measure RSS, Slim the Image, Then Deploy

I cannot optimize what I do not measure. After startup—and again after a handful of requests—I print RSS:

import psutil
import os

def memory_usage_mb():
    process = psutil.Process(os.getpid())
    return process.memory_info().rss / 1024 / 1024

print(f"Baseline RSS: {memory_usage_mb():.1f} MB")

If baseline exceeds 200MB on a 256MB tier, something is wrong: a heavy import, a global cache, or a leftover debug tool. For Python-level allocations I use tracemalloc to find the exact lines.

Deploy steps I follow:

1. Keep requirements.txt to fastapi, httpx, and uvicorn. Add psutil only while profiling, then remove it.
2. Use a minimal image such as python:3.11-slim.
3. Build and push; confirm the image stays under ~200MB.
4. Set MODEL_URL to the remote chat-completions endpoint.
5. Start with uvicorn main:app --host 0.0.0.0 --port 8000 and a single worker.

That image plus a 50–100MB process fits a typical 256MB free tier with room for the OS.

This pattern is not for every app. I skip it when I need offline inference, single-digit millisecond latency, or a policy that forbids sending data to an external API. I also skip it when traffic is bursty or high-volume and free-tier rate limits would break the app. Prototypes, internal tools, and low-traffic MVPs are the fit—not production workloads with strict SLAs.

MonkeyCode provides free model access and a free server option, which matches this architecture. The server stays small because the model runs remotely, and the free token allowance lets me experiment without watching a bill. As with any free tier, I verify current limits in the official documentation before I depend on them.

Free infrastructure teaches me to respect constraints. Designing for memory first produces apps that are easier to deploy, cheaper to scale, and simpler to reason about. The next time an AI app dies on a 256MB box, I reach for a memory profiler—not a bigger plan.

Copy the slim FastAPI proxy above, set MODEL_URL to a remote chat-completions endpoint, keep stream: True, and print RSS on boot and after a few requests. Confirm baseline stays around 50–100MB before you add features. Then run that experiment on a 256MB free tier—MonkeyCode’s free model access and free server option are a reasonable place to start, after you check the official docs for current limits. Ship the smallest version that works, and only then grow the app.