DEV Community: Niloy Nath

MinIO Was Easy, SeaweedFS Made Me Think

Niloy Nath — Fri, 10 Jul 2026 23:51:25 +0000

Why I wrote this

After I got one app working on MinIO, I thought I understood the self-hosted storage problem.

That confidence lasted exactly until the next deployment.

The new constraint looked similar: no S3, no managed object storage, media-heavy app, private/generated files, FastAPI backend, browser-facing URLs, and enough user uploads that "just put it in Postgres" would be a crime.

MinIO was the obvious first thought because it had already worked. It is S3-compatible, easy to run, and the boto3 migration story is basically one endpoint_url. But by this point I was also thinking about the enterprise side of the decision. MinIO is still open source, but serious enterprise packaging, support, and newer commercial product positioning live behind MinIO's paid AIStor/subscription world. For a client-owned deployment where the whole premise was "we want control and predictable cost", I wanted to at least look at the other serious self-hosted option.

That is how SeaweedFS entered the chat.

Note — SeaweedFS: a distributed storage system originally designed around fast file lookup and efficient small-file storage. It can expose a filesystem-like filer, volume servers, and an S3-compatible API. That last part is what made it usable for my app without rewriting every upload/download path.

MinIO felt like "S3, but on my server."

SeaweedFS felt like "a storage system that happens to speak S3."

That difference matters.

The terms, because I will forget them

If I come back to this in six months, this is the section I want near the top.

Object storage: storage for blobs addressed by keys, not rows or normal filesystem paths. Example key: users/42/gallery/image.webp. The slash is part of the name; it is not necessarily a real folder.

Bucket: a top-level container for objects. S3 calls it a bucket. S3-compatible systems usually keep the same word.

S3-compatible: not AWS S3, but speaks enough of the same API that tools like boto3, AWS CLI, and pre-signed URLs can work.

boto3: the Python AWS SDK. In this project it is not "AWS code" anymore; it is just the S3 protocol client.

Endpoint URL: the base URL boto3 talks to. For AWS it is implicit. For MinIO/SeaweedFS it is explicit, like http://seaweed:8333.

SigV4: AWS Signature Version 4, the signing algorithm used for authenticated S3 requests and pre-signed URLs.

Pre-signed URL: a temporary URL that contains a signature. Anyone with the URL can access that one object until it expires.

Path-style URL: https://media.example.com/bucket/key.

Virtual-hosted-style URL: https://bucket.media.example.com/key.

Filer: SeaweedFS's metadata/file layer. It gives a directory-like view over files whose bytes live on volume servers.

Volume server: SeaweedFS component that stores actual file content.

Master: SeaweedFS coordinator that tracks volumes and where file ids live.

S3 gateway/API: the SeaweedFS-facing door that lets normal S3 clients upload and download objects.

That is the whole mental model: my app talks S3, SeaweedFS translates that into its own filer/volume world, and the browser receives either public URLs or pre-signed URLs.

Why not just stay on MinIO?

I still like MinIO. For a single machine, simple object storage, and a codebase that already uses boto3, it is almost unfairly convenient. One binary, one console, one bucket, done.

But this app had a slightly different smell:

Lots of generated media. Images, audio, and possibly video. That means many files, many reads, and a lot of cache behavior.

A need for public and internal paths. The backend should talk to storage privately. The browser should see a clean HTTPS media domain.

A future where "single disk with object API" might stop being enough. I did not need a giant cluster on day one, but I wanted a path that did not force another migration the moment storage became more serious.

SeaweedFS looked attractive because it is not only trying to mimic S3. Its architecture has masters, volume servers, filers, and an S3 gateway. That is more moving parts, yes. But it also means the S3 API is not the whole product.

Note — filer: in SeaweedFS, the filer gives you a directory/file abstraction and metadata layer. The actual bytes live on volume servers. The S3 API sits in front of that world so S3 clients can still PUT and GET objects.

The official SeaweedFS README talks a lot about fast small-file access, volume-id lookup, and using existing metadata stores for filer metadata. That is not something I cared about in the first hour. I cared about "does boto3 upload work?" But later, those design choices started making the system feel more powerful than MinIO for the kind of messy media workload I was dealing with.

MinIO vs SeaweedFS in my head

This is not a benchmark. This is the practical decision table I wish I had while choosing.

MinIO:
  best when      I want S3-compatible object storage quickly
  mental model   server -> bucket -> object
  setup feel     simple
  operations     easier to explain
  downside       feels narrower if I later need a bigger storage architecture

SeaweedFS:
  best when      I expect lots of files and want room to grow storage topology
  mental model   master -> filer -> volume -> optional S3 API
  setup feel     more nouns, more config, more thinking
  operations     more powerful but less beginner-friendly
  downside       I need to understand the pieces before debugging production

If the app only needs "store some files and give me S3 URLs", MinIO is the clean answer.

If the app is media-heavy and might eventually need multiple storage nodes, filer behavior, public/internal endpoints, and more control over how files are organized, SeaweedFS becomes interesting.

I did not pick SeaweedFS because it was easier. I picked it because the future shape of the app looked less tiny than the first deployment.

The boto3 config was familiar

The pleasant surprise: the app code still stayed S3-shaped.

The client factory looked almost the same as MinIO. The important bits were endpoint_url, SigV4, and path-style addressing:

import boto3
from botocore.config import Config

async def get_s3_client():
    endpoint_url = settings.AWS_S3_ENDPOINT_URL

    if not endpoint_url:
        raise ValueError("AWS_S3_ENDPOINT_URL is not configured")

    config = Config(
        signature_version="s3v4",
        s3={"addressing_style": "path"},
    )

    return boto3.client(
        "s3",
        aws_access_key_id=settings.AWS_ACCESS_KEY_ID,
        aws_secret_access_key=settings.AWS_SECRET_ACCESS_KEY,
        region_name=settings.AWS_REGION,
        endpoint_url=endpoint_url,
        config=config,
    )

Note — path-style addressing: https://storage.example.com/bucket/key instead of https://bucket.storage.example.com/key. Self-hosted S3-compatible systems usually want path-style unless you have wildcard DNS and TLS set up for every bucket hostname.

Uploads stayed boring, which is exactly what I wanted:

async def upload_to_s3_file(file_obj, s3_key, content_type, bucket_name, s3_client=None):
    if s3_client is None:
        s3_client = await get_s3_client()

    def _upload():
        s3_client.upload_fileobj(
            Fileobj=file_obj,
            Bucket=bucket_name,
            Key=s3_key,
            ExtraArgs={"ContentType": content_type},
        )

    await asyncio.to_thread(_upload)
    presigned_url = await generate_presigned_url(s3_key, bucket_name=bucket_name)
    return s3_key, presigned_url

That asyncio.to_thread is doing real work. boto3 is blocking. FastAPI is async. If you call boto3 directly inside the request handler, one slow upload/download can block the event loop and make unrelated requests feel haunted. Offloading the blocking storage call into a thread keeps the async server breathing.

The storage module boundary

The smartest thing in this migration was not SeaweedFS. It was keeping storage behind one module.

The rest of the app should not know whether the backend is AWS S3, MinIO, SeaweedFS, or a directory behind nginx. Most code should only need verbs like:

await upload_to_s3_file(file_obj, key, content_type, bucket)
url = await generate_presigned_url(key)
public_url = await generate_public_s3_url(key)
await delete_s3_object(key)

That sounds boring, but boring boundaries save migrations.

The module owns:

credentials
endpoint_url
public_url
bucket name
path-style addressing
presigned URL generation
blocking boto3 calls
Redis caching
ContentType
delete behavior

The feature code owns:

what file is being uploaded
who is allowed to see it
what key/name it should have
what DB row points to it

When those boundaries blur, every storage migration becomes a treasure hunt. I prefer boring now. Boring is portable.

The part SeaweedFS forced me to understand

With MinIO, I had already learned that pre-signed URLs are fragile. The host is part of the SigV4 signature. Change the host after signing and you get SignatureDoesNotMatch.

SeaweedFS made that lesson more explicit because I wanted two endpoints:

AWS_S3_ENDPOINT_URL=http://seaweed:8333
AWS_S3_PUBLIC_URL=https://media.example.com
AWS_BUCKET_NAME=app-media

The backend container should use the internal endpoint:

http://seaweed:8333

The browser should use the public HTTPS endpoint:

https://media.example.com

My first instinct was to generate the URL internally, then rewrite the host to the public domain. That is exactly the trap from the MinIO migration. It looks like a harmless string operation, but it breaks the signature.

So I split the clients.

One client is for backend storage operations:

async def get_s3_client():
    return boto3.client(
        "s3",
        endpoint_url=settings.AWS_S3_ENDPOINT_URL,
        aws_access_key_id=settings.AWS_ACCESS_KEY_ID,
        aws_secret_access_key=settings.AWS_SECRET_ACCESS_KEY,
        region_name=settings.AWS_REGION,
        config=Config(signature_version="s3v4", s3={"addressing_style": "path"}),
    )

Another client is specifically for pre-signed URLs:

async def get_s3_presign_client():
    endpoint_url = (
        settings.AWS_S3_PUBLIC_URL
        or settings.AWS_S3_ENDPOINT_URL
    ).rstrip("/")

    return boto3.client(
        "s3",
        endpoint_url=endpoint_url,
        aws_access_key_id=settings.AWS_ACCESS_KEY_ID,
        aws_secret_access_key=settings.AWS_SECRET_ACCESS_KEY,
        region_name=settings.AWS_REGION,
        config=Config(signature_version="s3v4", s3={"addressing_style": "path"}),
    )

That is the clean version of what I learned the painful way:

Sign the URL with the hostname the browser will actually request.

If the browser requests https://media.example.com/app-media/file.webp, the pre-signing client should also use https://media.example.com. Not http://seaweed:8333. Not an IP. Not a Docker service name. The public host.

Why hostnames break signatures

This is the thing I keep needing to re-explain to myself.

A pre-signed URL is not just "a URL with a token". It is a signed representation of a specific HTTP request. The signature is calculated from pieces like:

HTTP method
path
query params
headers
host
expiry
secret key

So if boto3 signs this:

GET http://seaweed:8333/app-media/image.webp

and the browser actually requests this:

GET https://media.example.com/app-media/image.webp

SeaweedFS receives a request whose host does not match the signed host. The storage server recalculates the signature from the request it received, compares it to the signature in the URL, and says no.

This is why the fix is not "replace the hostname in the string".

The fix is:

Use internal endpoint for backend upload/download.
Use public endpoint for presigned URL generation.
Make nginx/proxy route public endpoint to the storage service without changing path/query.

That is the shape.

The public URL helper

Not every file needs a pre-signed URL. Some generated assets can be public, and for those I wanted a fast path that did not sign anything:

import urllib.parse

async def generate_public_s3_url(s3_key: str) -> str | None:
    if not s3_key:
        return None

    bucket_name = settings.AWS_BUCKET_NAME
    encoded_key = urllib.parse.quote(s3_key, safe="/")
    public_endpoint = (
        settings.AWS_S3_PUBLIC_URL
        or settings.AWS_S3_ENDPOINT_URL
    ).rstrip("/")

    return f"{public_endpoint}/{bucket_name}/{encoded_key}"

This is not a security feature. This is the opposite: it assumes the object is public or the endpoint is allowed to serve it without a signature. But it matters because not all media in an app has the same privacy level.

Private generated content gets pre-signed URLs.

Public thumbnails, static generated samples, and assets meant to be indexed can use clean public URLs.

One storage backend. Two access patterns.

Caching pre-signed URLs without lying to yourself

Pre-signing is cheap, but not free. Also, signing on every list endpoint is wasteful if you are rendering dozens of media items. So I cached pre-signed URLs in Redis, always with a TTL shorter than the actual expiry:

async def generate_presigned_url(s3_key, expires_in=None, bucket_name=None):
    if not s3_key:
        return None

    expires_in = expires_in or settings.S3_PRESIGNED_EXPIRES

    cached = await redis_cache.get_presigned(s3_key)
    if cached:
        return cached

    s3_client = await get_s3_presign_client()
    bucket = bucket_name or settings.AWS_BUCKET_NAME

    url = s3_client.generate_presigned_url(
        ClientMethod="get_object",
        Params={"Bucket": bucket, "Key": s3_key},
        ExpiresIn=expires_in,
    )

    cache_ttl = min(max(expires_in - 60, 0), 86400)
    await redis_cache.set_presigned(s3_key, url, ttl=cache_ttl)
    return url

The expires_in - 60 part is the boring bit that prevents edge-of-expiry bugs. You do not want to hand the frontend a URL that technically has three seconds left because Redis still had it. Shave a minute off and sleep better.

I also learned from the MinIO migration that cache keys should be treated as infrastructure-coupled. If you change storage hostnames, old pre-signed URLs can survive in Redis and point at the wrong world. The fix is either clear the cache during migration or store enough metadata to reject stale URLs.

The smoke tests I want before trusting storage

I do not trust storage because config "looks right" anymore. I trust storage after a script puts bytes in and gets the same bytes out.

def smoke_storage():
    client = get_s3_client()
    bucket = os.environ["AWS_BUCKET_NAME"]
    key = "smoke/test.txt"
    data = b"hello-storage"

    client.put_object(
        Bucket=bucket,
        Key=key,
        Body=data,
        ContentType="text/plain",
    )

    got = client.get_object(Bucket=bucket, Key=key)["Body"].read()
    assert got == data

    url = client.generate_presigned_url(
        ClientMethod="get_object",
        Params={"Bucket": bucket, "Key": key},
        ExpiresIn=300,
    )
    print(url)

    client.delete_object(Bucket=bucket, Key=key)

Then I test the public URL path too:

curl -I "https://media.example.com/app-media/smoke/test.txt"

And if it is pre-signed:

curl -I "$PRESIGNED_URL"

The failure messages tell you where to look:

AccessDenied              credentials or bucket policy
NoSuchBucket              bucket was never created or wrong bucket env
NoSuchKey                 object key mismatch
SignatureDoesNotMatch     host/path/query changed after signing
Connection refused        endpoint/port/proxy problem
SSL error                 public URL/TLS/proxy certificate problem

This little table alone would have saved me hours.

The production-ish environment shape

The env vars that mattered were basically these:

AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=...
AWS_BUCKET_NAME=app-media
AWS_REGION=us-east-1

# Internal service URL. Backend uses this for direct storage operations.
AWS_S3_ENDPOINT_URL=http://seaweed:8333

# Public browser URL. Presigned URLs should be generated against this host.
AWS_S3_PUBLIC_URL=https://media.example.com

S3_PRESIGNED_EXPIRES=3600
REDIS_URL=redis://redis:6379/0

The names still say AWS_, which is a little funny because AWS is exactly what I was not using. I kept them because boto3 and old code already spoke that language. Sometimes consistency beats semantic purity.

Future-me note: if public files are broken, inspect AWS_S3_PUBLIC_URL. If backend uploads are broken, inspect AWS_S3_ENDPOINT_URL. If pre-signed URLs generate but fail in browser, inspect the host in the generated URL before touching the upload code.

Why SeaweedFS felt harder

SeaweedFS was harder because it has more nouns.

MinIO gives you server, bucket, object.

SeaweedFS gives you master, volume, filer, S3 gateway, bucket, collection, replication, filer metadata, public endpoint, internal endpoint, and sometimes IAM depending on how you deploy it.

That does not make it bad. It makes it less fake-simple.

With MinIO, I mostly thought about S3 compatibility.

With SeaweedFS, I had to think about how the storage system itself was shaped. Where does metadata live? Which endpoint should the app use? Which endpoint should the browser use? What happens when I add another volume server? Am I using the S3 layer as the main interface, or do I eventually want filer-level access too?

This is where SeaweedFS started feeling more powerful. It was not trying to be only a replacement for S3. It was giving me a storage architecture I could grow into.

Note — volume server: the SeaweedFS component that stores actual file content. The master knows which volume has which file id. This is part of why SeaweedFS can make file lookup efficient: once you know the volume id, finding the bytes is direct and cacheable.

The cost of that power is operational complexity. You have to understand more before you can debug confidently. When something fails, "is the S3 API up?" is not the only question. You may need to ask whether the filer can talk to volumes, whether the public endpoint is signing correctly, whether the bucket exists, whether the credentials map to the right permissions, and whether your reverse proxy is preserving the host the signature expects.

What I would check during an incident

If media suddenly breaks, I would debug from the outside inward.

1. Is the browser URL sane?

Does it use https?
Does it contain the expected bucket?
Does the hostname match AWS_S3_PUBLIC_URL?
Does it still point to an old MinIO/S3 host?

2. Does the public endpoint answer?

curl -I https://media.example.com

3. Can the backend talk to internal storage?

curl -I http://seaweed:8333

4. Can boto3 list buckets from inside the backend environment?

client = await get_s3_client()
print(client.list_buckets())

5. Is Redis handing out stale URLs?

Clear presigned URL cache after endpoint changes.
Never trust old cached URLs after storage migration.

6. Is nginx preserving path and query string?

Pre-signed URLs need the exact path and query params. A proxy that drops query strings, normalizes paths, or redirects to a different host can break signatures.

This debugging order matters because storage failures can look like frontend bugs. The image tag fails. The video player spins. The gallery looks empty. But the real bug might be four layers down in a signature host mismatch.

Would I use it again?

Yes, but not automatically.

For a small app on one VM where the only requirement is "replace S3 quickly", I would still reach for MinIO first. It is simpler, friendlier, and easier to explain to the next developer.

For a media-heavy app with lots of generated files, a future need for scaling storage horizontally, and a client who wants self-hosted infrastructure without buying into a commercial object-store subscription, I would seriously consider SeaweedFS again.

It is lower-level in the best and worst ways.

The best way: it gives you more control over the storage architecture than "one S3-compatible box with a console".

The worst way: it makes you responsible for understanding that architecture.

The thing I liked most is that my application code did not have to become SeaweedFS-specific. It still spoke S3 through boto3:

s3_client.upload_fileobj(
    Fileobj=file_obj,
    Bucket=bucket_name,
    Key=s3_key,
    ExtraArgs={"ContentType": content_type},
)

That is the trick I keep coming back to. The app should know as little as possible about the storage backend. Put the weirdness in one module. Make the rest of the code say "upload object", "generate URL", "delete object", and move on with its life.

MinIO taught me that S3-compatible storage can save a migration.

SeaweedFS taught me that S3 compatibility is just the front door. Sometimes the building behind it is much bigger.

The Client Said On-Prem, So I Learned Linux Deployments

Niloy Nath — Fri, 10 Jul 2026 23:45:40 +0000

Why I wrote this

After the storage migration, I ran into the next very normal sentence that secretly means "your weekend is gone":

Can we deploy this on our own Linux server?

Not "can we deploy it to AWS". Not "can we use ECS". Not "please make a Terraform module and call it a day". The ask was two separate apps, two separate client-owned machines, and the same hard constraint underneath: keep it off managed AWS infrastructure.

This changed the shape of the problem immediately. ECS was out. RDS was out. ALB was out. CloudWatch was out. All the boring managed pieces I usually lean on became my problem again: process restarts, logs, ports, database connectivity, Redis, TLS, health checks, static assets, long-running requests, file permissions, and the tiny Linux details that wait until 2 AM to become interesting.

Note — on-prem: in this context, "on-prem" did not mean a polished datacenter with a platform team and runbooks. It meant Linux VMs controlled by the client, with their security rules, their access level, their firewall, their domain setup, and their appetite for giving me privileges. Which was, in one case, very little.

The funny part is that both deployments were basically the same product shape: a React/Vite frontend, a FastAPI backend, Postgres, Redis, media generation, payment/webhook endpoints, and admin routes. But the two servers allowed completely different deployment strategies.

One gave me Docker. The other gave me a shell account and no sudo.

Same app category. Completely different battle.

The mental model I wish I had before starting

Cloud deployment makes you think deployment is mainly "where do I push this image?" On-prem corrected me very quickly. Deployment is not one thing. It is a stack of small responsibilities that cloud services usually absorb for you.

This is the checklist I now keep in my head:

Runtime: how does the app process start? uvicorn, gunicorn, node, serve, something else?

Process supervision: if the process dies, who restarts it? Docker restart policy, systemd, PM2, or nobody?

Networking: which port does the app bind to, and is that port private or public?

Reverse proxy: what receives public traffic and forwards it inward? Usually nginx.

TLS: where does HTTPS terminate? nginx, a hosting panel, Cloudflare, or inside the app?

Persistence: where do Postgres, Redis, uploaded media, and generated files live?

Configuration: where do secrets and environment variables come from?

Observability: where do logs go, and how do I know the app is alive?

Recovery: after a reboot, what comes back automatically?

Note — process supervision: a process supervisor is the thing that keeps your app process alive. It starts it, restarts it after crashes, and usually gives you logs. ECS does this in AWS. Docker can do it with restart: unless-stopped. systemd does it at the OS level. PM2 can do it from a user account. A terminal tab does not do it. A terminal tab is a dare.

Once I started thinking in those boxes, the two deployments stopped feeling like chaos and started feeling like different answers to the same questions.

Docker deployment:

runtime              container CMD / entrypoint
process supervision  Docker restart policy
networking           Docker bridge network + exposed internal ports
reverse proxy        nginx container
TLS                  nginx/certbot
persistence          Docker volumes
logs                 docker compose logs
recovery             docker compose services restart on boot

No-sudo deployment:

runtime              uvicorn + static frontend server
process supervision  PM2
networking           loopback ports only
reverse proxy        whatever the host already provided
TLS                  outside my user account
persistence          existing DB/storage paths
logs                 pm2 logs
recovery             pm2 save, if startup integration exists

That second list is not pretty, but it is honest. And honest architecture is easier to debug than imaginary architecture.

The deployment I wanted: Docker, nginx, Postgres, Redis

The first machine let me use Docker, so I built the thing the way I wish every small on-prem deployment could be built: containers for the app boundary, compose for orchestration, nginx at the edge, and health checks everywhere.

The backend image was a fairly normal FastAPI production container. Build wheels in one stage, run a slim runtime in the second stage, drop privileges to a non-root user, then start Gunicorn with Uvicorn workers:

FROM python:3.12-slim AS builder

WORKDIR /build
RUN apt-get update && apt-get install -y --no-install-recommends \
    build-essential gcc g++ libpq-dev libffi-dev libssl-dev \
    && rm -rf /var/lib/apt/lists/*

COPY requirements.txt .
RUN pip install --upgrade pip setuptools wheel && \
    pip wheel --no-cache-dir --wheel-dir /wheels -r requirements.txt && \
    pip wheel --no-cache-dir --wheel-dir /wheels gunicorn uvicorn[standard]

FROM python:3.12-slim

ENV PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1 \
    PORT=8000 \
    WEB_CONCURRENCY=4 \
    GUNICORN_TIMEOUT=300

WORKDIR /app
RUN apt-get update && apt-get install -y --no-install-recommends \
    libpq5 curl ca-certificates ffmpeg \
    && rm -rf /var/lib/apt/lists/*

COPY --from=builder /wheels /wheels
RUN pip install --no-cache-dir --no-index --find-links=/wheels /wheels/*.whl \
    && rm -rf /wheels

RUN groupadd -r app && useradd -r -g app -u 1000 app
COPY --chown=app:app ./app ./app
COPY --chown=app:app docker-entrypoint.sh /usr/local/bin/docker-entrypoint.sh
RUN chmod +x /usr/local/bin/docker-entrypoint.sh

USER app
EXPOSE 8000

HEALTHCHECK --interval=15s --timeout=3s --start-period=20s --retries=3 \
  CMD curl -f http://127.0.0.1:${PORT}/health || exit 1

ENTRYPOINT ["/usr/local/bin/docker-entrypoint.sh"]
CMD ["gunicorn", "app.main:app"]

That ffmpeg line looks random until you remember media apps are never just "web apps". Someone will upload a video, the backend will need to inspect it, transcode it, or generate something from it, and suddenly your clean Python image needs system packages.

The entrypoint did the part I usually want systemd or ECS to do for me: calculate worker count, set timeouts, and start the backend in a way that can survive real traffic:

#!/usr/bin/env bash
set -euo pipefail

if [ -z "${WEB_CONCURRENCY:-}" ]; then
  CPU_COUNT=$(nproc 2>/dev/null || echo 1)
  WORKERS_PER_CORE=${WORKERS_PER_CORE:-2}
  MAX_WORKERS=${MAX_WORKERS:-12}
  CALCULATED_WORKERS=$((CPU_COUNT * WORKERS_PER_CORE + 1))

  if [ "$CALCULATED_WORKERS" -gt "$MAX_WORKERS" ]; then
    WEB_CONCURRENCY=$MAX_WORKERS
  else
    WEB_CONCURRENCY=$CALCULATED_WORKERS
  fi
fi

exec "$@" \
  -k uvicorn.workers.UvicornWorker \
  -w "${WEB_CONCURRENCY}" \
  -b "0.0.0.0:${PORT:-8000}" \
  --timeout "${GUNICORN_TIMEOUT:-300}" \
  --graceful-timeout "${GUNICORN_GRACEFUL_TIMEOUT:-30}" \
  --max-requests "${GUNICORN_MAX_REQUESTS:-10000}" \
  --max-requests-jitter "${GUNICORN_MAX_REQUESTS_JITTER:-1000}" \
  --worker-tmp-dir /dev/shm \
  --access-logfile - \
  --error-logfile -

Note — Gunicorn + Uvicorn: Uvicorn is the ASGI server that actually speaks async Python to FastAPI. Gunicorn is the process manager around it. Running gunicorn -k uvicorn.workers.UvicornWorker gives you multiple worker processes, graceful restarts, timeouts, and process supervision inside the container.

Compose then became the map of the whole app:

services:
  backend:
    build:
      context: .
      dockerfile: Dockerfile
    env_file: .env
    expose:
      - "8000"
    restart: unless-stopped
    depends_on:
      db:
        condition: service_healthy
      redis:
        condition: service_healthy
    healthcheck:
      test: ["CMD", "curl", "-fsS", "http://127.0.0.1:8000/health"]
      interval: 20s
      timeout: 10s
      retries: 3

  db:
    image: postgres:16
    restart: unless-stopped
    environment:
      POSTGRES_USER: app
      POSTGRES_PASSWORD: app
      POSTGRES_DB: app
    volumes:
      - pgdata:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U app -d app"]

  redis:
    image: redis:7-alpine
    restart: unless-stopped
    command: >
      redis-server
      --maxmemory 2gb
      --maxmemory-policy allkeys-lru
      --appendonly yes
      --appendfsync everysec

volumes:
  pgdata:

The important thing here is not that compose is glamorous. It is not. The important thing is that the runtime is explicit. If the backend dies, restart it. If Postgres is not healthy, do not start the backend yet. If Redis goes away, make it obvious. If logs grow forever, rotate them. If the server reboots, bring everything back.

Cloud platforms hide this machinery so well that you forget it exists. On a Linux box, it very much exists.

Why each Docker piece existed

This is the part I want future-me to reread before casually deleting something from a compose file.

backend existed for the API. It should not publish a public port directly. It should expose 8000 only to the Docker network and let nginx decide what the outside world can reach.

db existed because the app needed Postgres close to it. Managed RDS was not allowed, so Postgres became my responsibility. The volume mattered more than the container. Containers are replaceable; database files are not.

Note — Docker volume: a Docker-managed persistent directory mounted into a container. If you delete and recreate the Postgres container but keep the volume, the database survives. If you delete the volume, congratulations, you have invented a restore drill.

redis existed for cache, queues, rate-ish state, and pre-signed URL caching. Redis is often treated as disposable until some part of the app quietly starts depending on it. I used append-only persistence because losing cache is fine, but losing every transient app state after a restart can become annoying fast.

nginx existed because public traffic should not hit app servers directly. nginx handles TLS, upload size, compression, static assets, proxy headers, admin route guards, and long timeouts. It is boring in the exact way production needs.

certbot existed because certificates expire. It is easy to get HTTPS working once. The real question is whether it will still work 91 days later when you are not thinking about it.

Note — health check: a command that tells the platform whether a service is alive. For an API, this is usually curl /health. For Postgres, pg_isready. For Redis, redis-cli ping. Health checks are not decoration; they are how your deployment decides whether to route traffic, restart services, or wait before starting dependents.

The most useful Docker command was not up. It was this:

docker compose ps

Because it answered the question I actually had during deploy: "What does the machine think is running?"

nginx became the load balancer I did not have

Without an ALB, nginx became the front door: TLS termination, static files, API proxy, admin route protection, upload limits, long request timeouts, and health checks.

The app had image and video generation paths, so default proxy timeouts were a trap. A "normal" 60 second timeout looks fine until a user kicks off a video workflow and nginx gives up while the backend is still doing exactly what it was asked to do.

worker_processes auto;
worker_rlimit_nofile 1000000;

events {
    worker_connections 65536;
    multi_accept on;
}

http {
    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;

    keepalive_timeout 300s;
    client_header_timeout 300s;
    client_body_timeout 300s;
    send_timeout 300s;
    client_max_body_size 200M;

    upstream backend_upstream {
        server backend:8000 fail_timeout=5s;
        keepalive 16;
    }

    server {
        listen 80;
        server_name example.com www.example.com;

        location = /health {
            proxy_pass http://backend_upstream/health;
        }

        location /api/ {
            proxy_pass http://backend_upstream/api/;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_read_timeout 300s;
            proxy_send_timeout 300s;
        }
    }
}

Note — reverse proxy: nginx is not running the app. It receives the public request, decides where it should go, forwards it to an internal service, and streams the response back. In cloud terms, it is doing a slice of what an ALB, CDN, TLS manager, and routing layer might normally do together.

This deployment was not painless, but it was legible. docker compose ps showed the state. docker compose logs -f backend showed the app. docker compose restart backend did what it said. I could rebuild one service without touching the others.

That is the good version of on-prem.

My Docker deploy runbook

This is the version I wish I had written before doing it the first time.

# 1. SSH in and go to the repo
ssh app@example-server
cd /srv/app

# 2. Pull code
git pull

# 3. Check env before touching containers
test -f backend/.env
grep -E "DATABASE_URL|REDIS_URL|FRONTEND_URL|BACKEND_URL" backend/.env

# 4. Build images
docker compose build backend frontend

# 5. Start dependencies first if needed
docker compose up -d db redis
docker compose ps

# 6. Apply migrations if the app has them
docker compose run --rm backend alembic upgrade head

# 7. Start app + edge
docker compose up -d backend frontend nginx

# 8. Watch logs while hitting the site
docker compose logs -f backend nginx

The important part is not memorizing these exact commands. It is the order: get the code, confirm config, build, bring up stateful dependencies, migrate, start app services, then watch logs during real requests.

If I skip the config check, I eventually pay for it in the logs.

The preflight checks I now do

Before calling any on-prem deploy "done", I now check boring things out loud:

df -h
free -h
uptime
docker system df
docker compose ps

Note — df -h: shows disk usage. Disk-full bugs are some of the least glamorous production bugs. Postgres cannot write, Docker cannot pull layers, nginx cannot write logs, and suddenly everything looks broken in unrelated ways.

Then app checks:

curl -i http://127.0.0.1:8000/health
curl -i https://example.com/health
curl -i https://example.com/api/v1/bootstrap

Then user-flow checks:

home page loads
login works
admin route behaves correctly
one media upload/generation works
one payment/webhook route is reachable
logs are not screaming

This sounds obvious. It is only obvious after you have deployed something that looked healthy until the first real user clicked the one route you did not test.

The deployment I got next: no sudo

The second server looked similar from far away and completely different from the shell.

No Docker.

No systemd unit files.

No nginx edits.

No installing packages into /usr/bin.

No "just open this port".

I had a user account, a project directory, Node, Python, and enough permissions to run processes as myself. Which sounds bad until you realize that a web app is, at the end of the day, a few long-running processes and some files. If I could not make the machine into a platform, I could make a tiny platform inside my home directory.

That meant PM2.

Note — PM2: PM2 is a Node.js process manager, but it can run any executable. It is not a replacement for Docker or systemd, but when you only have user-level permissions, it gives you process naming, restarts, logs, environment variables, and startup resurrection if the host has been configured for it.

The frontend was a Vite app:

{
  "scripts": {
    "dev": "vite --port 5174",
    "build": "tsc -b && vite build",
    "preview": "vite preview"
  }
}

For production, I did not want to run vite dev. That is a dev server. So the pattern became:

cd ~/app/frontend
npm ci
npm run build
npx serve -s dist -l 127.0.0.1:5174

The backend was FastAPI:

cd ~/app/backend
python3 -m venv .venv
. .venv/bin/activate
pip install -r requirements.txt
uvicorn app.main:app --host 127.0.0.1 --port 8000

Running those commands manually proves the app can start. It does not make a deployment. A deployment has to survive SSH disconnects, crashes, restarts, and "wait, which terminal did I run that in?"

So PM2 became the wrapper:

module.exports = {
  apps: [
    {
      name: "app-api",
      cwd: "/home/app/app/backend",
      script: ".venv/bin/uvicorn",
      args: "app.main:app --host 127.0.0.1 --port 8000",
      interpreter: "none",
      env: {
        ENVIRONMENT: "production",
        PYTHONUNBUFFERED: "1"
      },
      max_memory_restart: "900M",
      autorestart: true,
      watch: false
    },
    {
      name: "app-web",
      cwd: "/home/app/app/frontend",
      script: "npx",
      args: "serve -s dist -l 127.0.0.1:5174",
      interpreter: "none",
      env: {
        NODE_ENV: "production"
      },
      autorestart: true,
      watch: false
    }
  ]
};

Then deploys became boring enough to repeat:

git pull

cd backend
. .venv/bin/activate
pip install -r requirements.txt

cd ../frontend
npm ci
npm run build

cd ..
pm2 startOrReload ecosystem.config.cjs --update-env
pm2 save
pm2 status

Not as clean as Docker. Not as correct as systemd. But it worked inside the permissions I actually had, which is the only deployment environment that matters.

PM2 terms I keep forgetting

I keep having to relearn PM2 because I do not use it every week, so here is the tiny glossary.

Process name: the human label, like app-api. Use names. Do not rely on process IDs.

cwd: the working directory. This matters because relative imports, .env files, and frontend dist paths often assume a current directory.

script: the executable PM2 runs. For Python, this can be .venv/bin/uvicorn. For frontend static serving, it can be npx.

args: everything after the command.

interpreter: "none": tells PM2 not to wrap the command in Node. Without this, PM2 may try to treat non-JS commands like JavaScript and you get nonsense.

--update-env: tells PM2 to reload environment variables when restarting. Without this, you can change .env or shell vars and PM2 may keep the old world alive.

pm2 save: writes the current process list so it can be resurrected later. If the host has PM2 startup configured, this is what makes your current app list survive reboot.

The commands I actually use:

pm2 status
pm2 logs app-api --lines 100
pm2 restart app-api --update-env
pm2 restart app-web --update-env
pm2 delete app-api
pm2 save
pm2 monit

Note — pm2 monit: a terminal dashboard showing CPU, memory, restarts, and logs for PM2 processes. It is not observability in the fancy sense, but when you have no CloudWatch, it is a small lantern.

The no-sudo deploy runbook

This is the version for "I have a shell account and I need to be useful."

# 1. SSH in
ssh app@example-server
cd ~/app

# 2. Pull latest code
git pull

# 3. Backend dependencies
cd backend
python3 -m venv .venv
. .venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt

# 4. Smoke start backend before PM2
uvicorn app.main:app --host 127.0.0.1 --port 8000
# Ctrl-C if it starts cleanly

# 5. Frontend build
cd ../frontend
npm ci
VITE_API_BASE_URL=https://api.example.com/api/v1 npm run build

# 6. Reload managed processes
cd ..
pm2 startOrReload ecosystem.config.cjs --update-env
pm2 save
pm2 status

# 7. Warm and verify
curl -i http://127.0.0.1:8000/health
pm2 logs app-api --lines 80

The key move is starting the backend manually once before letting PM2 supervise it. If the app cannot start in the foreground, PM2 will not fix it. PM2 will just restart the same broken command very enthusiastically.

The weird bugs were all boundaries

The code was rarely the hard part. The hard part was boundaries.

The frontend needed the right API base URL at build time. With Vite, environment variables are baked into the build. Change VITE_API_BASE_URL after npm run build, and the already-built JS does not care. You need to rebuild.

VITE_API_BASE_URL=https://api.example.com/api/v1 npm run build

The backend needed to bind privately. If nginx or the hosting panel is the public entrypoint, the app should not casually bind to 0.0.0.0 unless you mean to expose it. On the PM2 server I preferred loopback:

uvicorn app.main:app --host 127.0.0.1 --port 8000

Warmup mattered. FastAPI startup was intentionally lightweight, but the first real request still had to pay for DB connections, Redis connections, config cache, GeoIP reader load, and whatever else had been lazily initialized. So I kept a tiny post-deploy warmup script:

#!/usr/bin/env bash
set -euo pipefail

BASE_URL=${1:-http://127.0.0.1:8000}

for i in 1 2 3 4 5; do
  echo "warmup attempt $i"
  curl -fsS -w "HTTP %{http_code} - %{time_total}s\n" \
    "${BASE_URL}/bootstrap" -o /dev/null || true
  sleep 0.2
done

This is not magic. It just moves the first-request pain from an unlucky user to me, immediately after deploy, while I am still watching logs.

Logs had to be part of the workflow. In ECS I would check CloudWatch. Here it was:

pm2 logs app-api
pm2 logs app-web
pm2 monit

And for Docker:

docker compose logs -f backend
docker compose logs -f nginx
docker compose ps

I learned to never call a deploy successful until I had tailed logs through one login, one API call, one media request, and one admin route. Otherwise you have not deployed the app. You have deployed optimism.

Things I would not forget next time

I am writing this section mostly for myself, because this is exactly the kind of operational detail I forget once the incident is over.

Do not expose app ports publicly unless you mean it. Bind internal services to 127.0.0.1 or Docker-only networks. Public traffic should go through nginx or the host's reverse proxy.

Do not run Vite dev in production. vite --host feels convenient and then you remember it is a development server. Build static assets and serve dist, or run a real SSR server if the app has one.

Do not treat .env as a mystery box. Print the non-secret shape of the environment before deploy: which DB host, which Redis URL, which frontend URL, which backend URL. Never print actual secrets into logs.

Do not assume reboot recovery. Test or at least inspect it. Docker restart policies help. PM2 needs pm2 save and usually startup integration. A process currently running is not the same as a process that will come back after a reboot.

Do not ignore upload size and timeout defaults. Media apps need larger client_max_body_size and longer proxy timeouts. Otherwise the backend can be healthy while nginx kills the request.

Do not call it done before testing the boring flows. Login, admin, media upload, payment/webhook, cache-dependent endpoint, one mobile page.

I used to think this kind of checklist was overkill. Then I had to debug production on someone else's Linux box, with someone else's permissions, while trying to remember which layer owned TLS. I am very pro-checklist now. Character development, but make it curl.

Would I do it this way again?

If sudo is available, I would use Docker again. No contest.

Docker gave me a repeatable runtime, named services, network isolation, health checks, restart policies, and clean logs. The compose file became living documentation. Six months later, someone can read it and understand what the machine is supposed to be running.

If sudo is not available, I would still use PM2 again, but with a very clear mental label: this is a survival deployment, not a platform.

PM2 will keep a process alive. It will not give you container isolation. It will not own system ports. It will not configure nginx. It will not install Postgres. It will not solve backups. But if the real constraint is "you may run userland processes and nothing else", PM2 is a perfectly respectable little raft.

The lesson I took from both deployments is not "Docker good, PM2 bad". It is that deployment strategy is mostly permission strategy.

Cloud gives you APIs. Docker gives you a local platform. Sudo gives you the machine. No sudo gives you a home directory and your manners.

And sometimes that is still enough.

The Client Said No AWS, So I Learned MinIO

Niloy Nath — Fri, 10 Jul 2026 23:43:59 +0000

Why I wrote this

Until this project, "object storage" and "S3" were the same word in my head. Bucket, IAM keys, boto3, done. I had never once thought about what happens underneath, because AWS's whole pitch is that I don't have to.

Note — object storage: unlike a filesystem (folders, files, paths) or a database (rows, queries), object storage stores blobs of data ("objects") in flat containers called buckets, addressed by a string key like users/42/avatar.png. No real directories — the slashes in keys are just a naming convention. It's the standard way to store user uploads, media, backups, and anything too big or too numerous for a database.

Then a client told me the app had to run entirely on their own Linux servers. Not "prefer self-hosted", not "AWS but in a specific region" — no AWS account at all, no managed anything, their VMs, their disks. The app was a FastAPI backend generating a lot of media (images, audio, video), and every one of those files was going through S3 with pre-signed URLs.

Note — pre-signed URL: a temporary URL that grants access to a single private object without exposing your credentials. The server signs it using its secret key, embedding an expiry time and a cryptographic signature into the query string. Anyone holding the URL can fetch that one object until it expires — no login needed. This is how you serve private media directly from storage without routing every byte through your backend.

So I had a weekend to figure out what replaces S3 when S3 is not allowed.

What I looked at

I didn't do a deep bake-off, I'll be honest. I had a deadline. But I did spend an evening reading before committing, and this is roughly how each option died:

Ceph came up first because everyone says Ceph. It's the heavyweight distributed storage system — it can expose block storage, a filesystem, and an S3-compatible API (through a component called RADOS Gateway) all from the same cluster. Then I read the deployment docs and closed the tab. Ceph assumes a cluster of machines and someone whose job is to babysit it. I had two VMs and my job was shipping the app. Deploying Ceph here would have been solving a problem I didn't have while creating several I couldn't afford.

SeaweedFS looked genuinely interesting — it's designed around the Facebook Haystack paper, optimized for billions of small files, and the benchmarks are impressive. But its S3 API is a gateway bolted onto its own volume/filer architecture, and I didn't want to learn a new mental model (volumes? filers? masters?) just to serve media files for one app.

Just the filesystem + nginx was the tempting lazy option. Save uploads to a directory, let nginx serve them. But I'd lose pre-signed URLs — which the app leaned on heavily for private media — plus multipart uploads for large videos, and every line of existing storage code. What looks like the simple option was actually the biggest rewrite.

Note — multipart upload: the S3 mechanism for uploading large files in parallel chunks, with the ability to retry individual chunks on failure. For anything above ~100MB it's the difference between "upload failed at 98%, start over" and "retry chunk 47". boto3's upload_fileobj uses it automatically above a size threshold — another thing you get for free by staying S3-compatible.

MinIO won for one boring reason: it speaks the actual S3 API. Same boto3, same calls, same v4 signatures. It ships as a single Go binary (or one Docker image), stores objects on a plain disk path, and includes a web console where the client could see their files. My storage code wouldn't know the difference. When the deciding factor is "how much of my working code do I get to keep", S3-compatible wins by default.

The migration was one config value (almost)

All my storage code already went through one helper module — every S3 call in the codebase funnels through storage.py. That turned out to be the luckiest architectural accident of the project. The only real change was giving boto3 an endpoint_url:

import boto3
from botocore.config import Config
from urllib.parse import urlparse

def get_s3_client():
    endpoint = settings.AWS_S3_ENDPOINT_URL      # e.g. http://minio:9000
    access_key = settings.AWS_ACCESS_KEY_ID
    secret_key = settings.AWS_SECRET_ACCESS_KEY

    use_ssl = True
    if endpoint:
        parsed = urlparse(endpoint)
        use_ssl = parsed.scheme == "https"
        endpoint = endpoint.rstrip("/")          # boto3 dislikes trailing slashes

    botocore_cfg = Config(
        signature_version="s3v4",
        s3={"addressing_style": "path"},
        retries={"max_attempts": 3, "mode": "standard"},
    )

    return boto3.client(
        "s3",
        endpoint_url=endpoint,
        aws_access_key_id=access_key,
        aws_secret_access_key=secret_key,
        config=botocore_cfg,
        use_ssl=use_ssl,
    )

The trick is that endpoint_url is None when you're on real AWS (boto3 falls back to the standard AWS endpoints), and points at your MinIO host otherwise. Same function, both worlds.

Two things in that config I only learned by hitting errors:

signature_version="s3v4" — MinIO only supports AWS Signature Version 4. Ancient code or old SDK defaults sometimes still use v2. If your requests come back with signature errors against MinIO, check this first.

Note — Signature v4 (SigV4): the algorithm AWS uses to authenticate API requests. Instead of sending your secret key, the client computes an HMAC signature over the request (method, host, path, headers, timestamp) and sends that. The server recomputes it and compares. This is also exactly what's inside a pre-signed URL — which becomes very important later in this post.

addressing_style: "path" — AWS defaults to virtual-hosted style, where the bucket name becomes part of the hostname: my-bucket.s3.amazonaws.com/key. For that to work on a self-hosted box, you'd need wildcard DNS (*.minio.example.com) resolving to your server, which nobody sets up for a single-node deployment. Path style puts the bucket in the path instead — minio:9000/my-bucket/key — and just works with one hostname.

Everything downstream of the client factory stayed untouched, because it's all just S3 calls:

def generate_presigned_url(s3_client, bucket, key, method="get", expires_in=36000):
    client_method = "get_object" if method == "get" else "put_object"
    return s3_client.generate_presigned_url(
        ClientMethod=client_method,
        Params={"Bucket": bucket, "Key": key},
        ExpiresIn=expires_in,
    )

def upload_fileobj(s3_client, fileobj, bucket, key, content_type=None):
    extra_args = {"ContentType": content_type} if content_type else {}
    s3_client.upload_fileobj(
        Fileobj=fileobj, Bucket=bucket, Key=key,
        ExtraArgs=extra_args or None,
    )

Setting ContentType on upload is one of those details that doesn't matter until it does: it's what makes the browser render an image inline instead of downloading it as a mystery file, and what makes <video> tags work at all.

I also wrote a dumb smoke test — create client, list buckets, put an object, read it back, delete it — and I now run it against every environment before believing anything works:

def run():
    client = storage.get_s3_client()
    print("Buckets:", [b["Name"] for b in client.list_buckets()["Buckets"]])

    bucket, key, data = "app-media", "smoke_test/test.txt", b"hello-minio"
    client.put_object(Bucket=bucket, Key=key, Body=data, ContentType="text/plain")
    assert client.get_object(Bucket=bucket, Key=key)["Body"].read() == data
    client.delete_object(Bucket=bucket, Key=key)
    print("smoke test OK")

The first time I ran it, it failed — wrong bucket name in the env file. Which is exactly why it exists. Thirty seconds at the bottom of the stack instead of an hour of debugging failed uploads three layers up. Unlike S3, MinIO doesn't auto-create anything: the bucket has to exist before the first put_object, either created through the web console (port 9001) or MinIO's mc CLI.

The part that actually cost me a day

Here's what nobody warns you about. MinIO was listening on http://<server-ip>:9000. The frontend was served over HTTPS. Every pre-signed URL the backend handed out was an http:// URL pointing at a raw IP — and browsers refuse to load HTTP content from an HTTPS page. On top of that, I was leaking the storage server's IP address to every single user in every media URL.

Note — mixed content: when an HTTPS page tries to load a resource (image, video, script) over plain HTTP. Browsers block it — silently for media, loudly in the console — because it undermines the security guarantee of the padlock. There is no setting to allow it for your users; you have to serve everything over HTTPS.

My first instinct was: easy, just rewrite the URL's host to the real HTTPS domain before returning it to the frontend. I did that. Everything broke with SignatureDoesNotMatch, and it took me embarrassingly long to understand why: the signature in a pre-signed URL is computed over the request including the host and path (that's the SigV4 from earlier). Change any of it and the signature no longer matches, and the storage server correctly rejects the request. The URL is a sealed envelope — you can't edit the address on it without breaking the seal.

The fix that worked: don't modify the signed URL at all. Wrap it. The backend rewrites MinIO URLs into a proxy URL on its own HTTPS domain, with the original pre-signed URL — signature intact — tucked into a query parameter:

def rewrite_minio_url_to_https(url: str) -> str:
    # Can't touch the presigned URL itself — signature covers host+path.
    # So we encode the whole thing and let /storage/proxy fetch it for us.
    if not url or url.startswith("https://"):
        return url
    encoded = urllib.parse.quote_plus(url)
    return f"{settings.BACKEND_URL}/storage/proxy?url={encoded}"

quote_plus matters here: the pre-signed URL contains its own query string (?X-Amz-Signature=...&X-Amz-Expires=...), and if you don't URL-encode it before embedding it in another URL, everything after the first & gets interpreted as parameters of the outer URL and the signature gets truncated. Encode the whole thing, decode it on the other side.

The proxy endpoint decodes the URL, fetches it server-side, and streams the response back to the browser:

@app.api_route("/storage/proxy", methods=["GET", "HEAD"])
async def storage_proxy(request: Request, url: str):
    decoded_url = urllib.parse.unquote_plus(url)
    parsed = urllib.parse.urlparse(decoded_url)

    # Without this check you've built an open proxy that will
    # happily fetch any URL on the internet on your server's behalf.
    allowed = urllib.parse.urlparse(settings.AWS_S3_ENDPOINT_URL).netloc
    if parsed.netloc != allowed:
        raise HTTPException(status_code=403, detail="forbidden host")

    forward_headers = {"accept": "*/*"}
    if range_hdr := request.headers.get("range"):
        forward_headers["range"] = range_hdr    # video seeking = 206 responses

    async with httpx.AsyncClient(timeout=30.0) as client:
        upstream = await client.get(decoded_url, headers=forward_headers)

    passthrough = {
        k: v for k in ("content-type", "content-length", "cache-control",
                       "accept-ranges", "content-range")
        if (v := upstream.headers.get(k))
    }
    return Response(content=upstream.content,
                    status_code=upstream.status_code,
                    headers=passthrough)

Two details in there that cost real debugging time:

The host allowlist. A proxy endpoint that fetches arbitrary URLs from a query parameter is an SSRF hole — someone can point it at your internal services (?url=http://localhost:8000/admin) and your server will fetch it with its own network access. Checking the target against the one configured MinIO endpoint closes that.

Note — SSRF (Server-Side Request Forgery): an attack where you trick a server into making HTTP requests on your behalf, typically to internal systems the attacker can't reach directly. Any endpoint that accepts a URL and fetches it is a candidate. Always allowlist the destinations.

Forwarding the Range header. Without it, video worked — but seeking didn't. The player would start from zero every time you dragged the timeline. Browsers implement video seeking by requesting byte ranges (Range: bytes=1048576-), and the server answers with 206 Partial Content and just that slice. If the proxy swallows the header, the upstream sends the whole file from byte zero and the player gives up on seeking. Same story for accept-ranges and content-range on the way back — the player needs those headers to know that seeking is even possible.

Is this proxy the "right" architecture? No. The proper fix is putting MinIO itself behind nginx with a real TLS certificate on its own subdomain, so pre-signed URLs are HTTPS from birth and browsers hit storage directly with no extra hop. But the cert situation on the client's box was not getting sorted that week, and the proxy shipped that day. It also quietly fixed CORS and stopped leaking the server IP, so I've made peace with it. If traffic ever justifies it, the nginx route is documented and waiting.

One more bug from the same migration, because it's a good one: I cache pre-signed URLs in Redis so I'm not re-signing on every request (signing is cheap, but at scale, every avoided call counts). After switching endpoints, users started getting broken media — the cache was still full of perfectly valid-looking amazonaws.com URLs pointing at buckets that no longer served anything. Cached URLs that outlive the infrastructure they point to. The fix is validating cache entries against the current endpoint before trusting them:

cached = await redis_cache.get_presigned(s3_key)
if cached:
    stale = "amazonaws.com" in cached or not cached.startswith(current_endpoint)
    if stale:
        await redis_cache.delete(f"presigned:{s3_key}")   # re-sign fresh
    else:
        return cached

General lesson filed away: whenever infrastructure changes, ask what's still sitting in caches pointing at the old world.

Would I pick it again?

For this project, yes, without hesitation. The constraint was hard (data stays on their servers, full stop), the codebase was already S3-shaped, and the scale was one beefy VM — not a use case that needs a distributed storage story. The app also serves a lot of media, and not paying S3 egress on every video view is a real number, not a theoretical one.

Note — egress: cloud-speak for data leaving the provider's network, i.e. bytes served to your users. AWS charges roughly $0.09/GB for it. Uploads (ingress) are free; it's the downloads that get you. A media-heavy app's egress bill scales directly with its popularity, which is a rude thing to discover during a good month.

But I want to be honest about what I gave up, because I felt it. S3 quietly gives you eleven nines of durability — it stores every object redundantly across multiple physical facilities, and you never think about it. A single MinIO node is exactly as durable as the disk it sits on. Backups, monitoring, disk-full alerts at 2 AM — those are mine now. (MinIO does support erasure coding — spreading data plus parity blocks across multiple drives so it can survive drive failures, RAID-like but at the object level — but that needs multiple drives, which this deployment didn't have. Nightly volume backups were the honest answer at this scale.)

There's also no ecosystem. No lifecycle rules quietly moving old files to cheaper storage, no bucket events triggering functions, no CDN a checkbox away. MinIO can approximate some of it (bucket notifications, ILM policies), but you're assembling it yourself. And MinIO is AGPL-licensed — fine for running it as a service inside a deployment like this, but if you're embedding it in software you distribute, read the license before your lawyer does.

If your constraint is "not AWS" rather than "not cloud", look at Cloudflare R2 or Backblaze B2 first — both speak the S3 API, both are drastically cheaper on egress (R2's is literally free), and neither makes you the on-call storage admin. If you're drowning in billions of tiny files, SeaweedFS is built for exactly that. If you have real multi-node durability requirements and a team to match, that's what Ceph is for. And if a directory of files behind nginx genuinely covers your needs, do that and don't feel bad about it.

The thing I actually took away from this isn't about MinIO at all. It's that the thin storage.py wrapper I wrote out of habit — one module, every S3 call goes through it — turned a forced infrastructure swap into a config change plus one proxy endpoint. I didn't design it for portability. But I'll pretend I did in the retro.