I've been running FastAPI in production for a payment processing platform in Mexico for the past couple of years. We handle SPEI transfers (Mexico's interbank transfer system), OXXO Pay (cash payments at convenience stores), and crypto on-ramps through multiple PSP integrations.
I wanted to share some real lessons from production that I don't see discussed much.
The stack
FastAPI + async SQLAlchemy 2.0 + asyncpg
PostgreSQL + Redis (rate limiting + caching)
Docker → AWS ECS Fargate
GitHub Actions CI/CD
Alembic for migrations
Lesson 1: Webhook state machines save your life
When you process payments, you receive webhooks from payment providers. The naive approach is to just update the payment status directly. Don't do this.
We built a webhook state machine where each payment goes through defined states: pending → processing → completed/failed. Every state transition is logged with timestamps. When a payment provider sends duplicate webhooks (and they will), the state machine just ignores transitions that don't make sense.
This saved us during a reconciliation where we found timezone boundary issues between UTC and CST that were causing mismatches at midnight.
Lesson 2: Alembic migrations + ECS = restart loops
This one hurt. We had an Alembic migration that was taking too long, and ECS health checks were killing the container before the migration finished. Container restarts, migration starts over, health check fails again. Infinite loop.
Fix: Run migrations as a separate ECS task before deploying the new container. Never run migrations on startup in production.
Lesson 3: psycopg2 + LIKE queries will bite you
If you use % in LIKE queries with psycopg2 and parameterized queries, you need %% for literal percent signs when mixed with %s placeholders. We had a bug where a search endpoint was silently returning empty results.
Wrong
cursor.execute("SELECT * FROM users WHERE name LIKE '%s%'", (search,))
Right
cursor.execute("SELECT * FROM users WHERE name LIKE %s", (f"%{search}%",))
Lesson 4: VARCHAR timestamps are a trap
One of our PSP integrations stores created_at as VARCHAR instead of TIMESTAMP. Every time we need to do date comparisons, we have to CAST(created_at AS TIMESTAMP). If you're designing your schema, always use proper TIMESTAMP columns.
Lesson 5: Rate limiting is not optional
We got hit with a brute force attack on our auth endpoints within the first month. Redis-backed rate limiting with SlowAPI saved us. 5 attempts per minute on login, 20 requests per minute on general endpoints.
Lesson 6: Connection pooling matters more than you think
Our KYC service was falling over under load. The fix? Increasing the connection pool from 10 to 50 connections. That's it. Went from random 500 errors to zero issues.
The architecture pattern that works
After iterating through multiple services, this is the pattern I settled on:
Repository (data access)
→ Service (business logic)
→ Endpoint (HTTP layer)
Each layer only talks to the one below it. Dependencies are injected via FastAPI's Depends(). Testing becomes trivial — just mock the repository layer.
After refining this over 10+ deployments, I packaged the boilerplate into a starter kit with the full auth system, async DB, Docker, CI/CD, and tests. It's on my Gumroad if anyone wants it. But the lessons above are the real value — the code is just the implementation.
I also deploy production AWS infrastructure for teams that don't want to deal with Terraform/ECS — details here if that's useful.
Happy to answer questions about running FastAPI in production, dealing with Mexican payment providers, or anything else.
Top comments (0)