The Architecture Decision That Almost Killed My Project (And How I Fixed It)

A real story about bottlenecks, debugging, and why the obvious solution isn't always right

---

The Problem

It's 3 AM. I'm staring at Grafana dashboards. VehicleMetrics is supposed to handle 1,000 requests per second.

It's handling 50.

The thing is breaking. Not crashing. Just... slow. Every request takes 2-3 seconds. Users are complaining. The API is timing out.

And I have no idea why.

---

The Setup (What I Thought Would Work)

Let me back up. Here's what I built:

Vehicle sends data → FastAPI → PostgreSQL (direct insert)

Simple, right? Beautiful architecture:

@app.post("/api/vehicles/{vehicle_id}/sensor-data")
async def ingest_data(vehicle_id: str, data: SensorData):
    # Direct write to database
    await db.execute(
        "INSERT INTO sensor_data VALUES (...)",
        [vehicle_id, data.timestamp, data.speed, ...]
    )
    return {"status": "created"}

Problem: I didn't see it coming.

---

The Debugging Journey (The Hard Part)

Attempt 1: "It's Probably the Database"

$ curl -X POST http://localhost:8000/api/vehicles/V1/sensor-data
  -H "Content-Type: application/json"
  -d '{"timestamp": 1000, "speed": 55, ...}'

real    0m2.834s
user    0m0.012s

2.8 seconds for one request. That's death.

First thought: Database is slow.

I check PostgreSQL:

SELECT count(*) FROM sensor_data;
-- 50 million rows

EXPLAIN ANALYZE INSERT INTO sensor_data VALUES (...);
-- Cost: 0.01, Time: 0.02ms

SELECT indexrelname FROM pg_indexes WHERE tablename = 'sensor_data';
-- 15 indexes

Wait. The INSERT itself is fast (0.02ms). But my API call took 2.8 seconds.

Realization: It's not the database. It's me.

Attempt 2: "Let Me Profile This"

I add timing to my code:

import time

@app.post("/api/vehicles/{vehicle_id}/sensor-data")
async def ingest_data(vehicle_id: str, data: SensorData):
    start = time.time()

    t1 = time.time()
    # Validation
    validated_data = SensorData(**data)
    print(f"Validation: {time.time() - t1}ms")

    t2 = time.time()
    # Database insert
    await db.execute(
        "INSERT INTO sensor_data VALUES (...)",
        [validated_data.vehicle_id, ...]
    )
    print(f"Insert: {time.time() - t2}ms")

    total = time.time() - start
    print(f"Total: {total}ms")

    return {"status": "created"}

Output:

Validation: 0.2ms
Insert: 2750ms
Total: 2760ms

The insert is taking 2.75 seconds.

That's... weird. The EXPLAIN said 0.02ms.

Attempt 3: "Let Me Check the Connection Pool"

Aha! I found it.

# ❌ OLD CODE
engine = create_async_engine(
    "postgresql+asyncpg://...",
    # No connection pool settings!
)

Default connection pool size: 5 connections.

What does that mean?

• First 5 requests: use 5 connections, fast
• 6th request: wait for a connection to free up...
• Wait... wait... wait...

With 50 concurrent requests, every request is waiting in a queue for a connection.

That's the problem.

---

The Fix (What I Learned)

Solution 1: Bigger Connection Pool

# ✅ BETTER
engine = create_async_engine(
    "postgresql+asyncpg://...",
    pool_size=50,  # 50 connections
    max_overflow=20,  # 20 extra connections if needed
)

Test again:

Request time: 2750ms → 150ms  (18x faster!)

Better. But still not 50ms.

Solution 2: The Real Problem (Async I/O)

I realized: even with a big pool, I'm doing synchronous writes.

Each insert waits for the database to acknowledge. With 1,000 concurrent requests:

Request 1: INSERT → Wait for DB → Response (takes 10ms)
Request 2: INSERT → Wait for DB → Response (takes 10ms)
Request 3: INSERT → Wait for DB → Response (takes 10ms)
...
Request 1000: INSERT → Wait for DB → Response (takes 10ms + 999 queued requests)

The real solution: Fire-and-forget with Kafka.

Solution 3: The Architecture Change

from aiokafka import AIOKafkaProducer

@app.on_event("startup")
async def startup():
    app.kafka = AIOKafkaProducer(
        bootstrap_servers=['kafka:9092'],
        compression_type='gzip',
    )
    await app.kafka.start()

@app.post("/api/vehicles/{vehicle_id}/sensor-data")
async def ingest_data(vehicle_id: str, data: SensorData):
    # Fire-and-forget to Kafka
    await app.kafka.send(
        "sensor-data",
        value=json.dumps(data.dict()).encode(),
        key=vehicle_id.encode(),
    )

    # Update Redis for real-time dashboard
    await redis.hset(
        f"vehicle:{vehicle_id}:latest",
        mapping=data.dict()
    )

    # Respond immediately (< 50ms)
    return {"status": "accepted"}

# Separate process: consume from Kafka, write to database
async def kafka_consumer():
    consumer = AIOKafkaConsumer(
        'sensor-data',
        bootstrap_servers=['kafka:9092'],
        group_id='ingestion-group',
    )
    await consumer.start()

    async for message in consumer:
        data = json.loads(message.value)
        await db.execute(
            "INSERT INTO sensor_data VALUES (...)",
            [data.vehicle_id, ...]
        )

New flow:

Vehicle sends data → FastAPI (< 5ms)
                  → Kafka (send, < 10ms)
                  → Response (< 20ms total)

Meanwhile:
Lambda/Consumer → Read from Kafka
               → Write to PostgreSQL (async)

Test again:

Request time: 150ms → 18ms  (8x faster!)
Throughput: 50 rps → 1000+ rps  (20x improvement!)

---

What I Learned (The Real Lesson)

Lesson 1: Premature Optimization is Evil (But Late Optimization is Worse)

I thought: "PostgreSQL is fast, let's just write directly."

Wrong. I didn't think about concurrency. With 1,000 concurrent requests, the database becomes the bottleneck, not because it's slow, but because I'm serializing all writes.

Lesson 2: Profiling Saves Your Life

Without timing, I would have blamed the database forever.

# Add this to EVERY endpoint
import time
@app.middleware("http")
async def add_timing(request: Request, call_next):
    start = time.time()
    response = await call_next(request)
    duration = time.time() - start
    logger.info(f"{request.method} {request.url.path}: {duration*1000:.1f}ms")
    return response

Lesson 3: Connection Pools Aren't Magic

Even with async/await, if you have 5 connections and 1,000 requests, you're creating a queue.

# The math:
pool_size = 50
avg_request_time = 10ms
requests_per_second = 1000

can_handle = (pool_size / avg_request_time) * 1000
# = (50 / 0.01) * 1000 = 5,000 rps

# With pool_size = 5:
can_handle = (5 / 0.01) * 1000 = 500 rps

Lesson 4: Async All The Way Down

This is where I messed up. Even with async FastAPI, my writes were blocking:

# ❌ BLOCKING (even in async context)
result = await db.execute("INSERT ...")  # Waits for DB
return result  # Only then returns

# ✅ NON-BLOCKING
background_task.add_task(write_to_db, data)
return {"status": "queued"}  # Returns immediately

Lesson 5: Decouple Your Read/Write Paths

The biggest realization:

• Writes: Fire-and-forget, async, high throughput
• Reads: Cached, can be slightly stale, always fast

Don't make them both go to the same place at the same time.

---

The Before & After

Before (Direct Database):

Architecture:
  Vehicle → FastAPI → PostgreSQL

Performance:
  - Throughput: 50 rps
  - Latency: 2,500ms (average)
  - P99 latency: 8,000ms
  - Database CPU: 95%

Problem:
  - Connection pool exhaustion
  - All requests serialized on writes
  - Backup building up

After (Kafka + Async):

Architecture:
  Vehicle → FastAPI → Kafka → Lambda → PostgreSQL
                   → Redis (for dashboard)

Performance:
  - Throughput: 1,000+ rps
  - Latency: 20ms (average)
  - P99 latency: 100ms
  - Database CPU: 30%

Benefits:
  - Decoupled writes
  - Can replay data
  - Dashboard updates instantly
  - Database never overwhelmed

20x improvement. Just by fixing the architecture.

---

The Real Lesson (Not Just Technical)

When you're solo building, you have no one to bounce ideas off.

So here's what I do now:

1. Ask "Why?" three times

   • Slow latency? → Database slow? → No, wait times → Connection pool? → Yes!

2. Profile first, optimize second

   • Don't guess. Measure. Then fix what's actually slow.

3. Think about the future

   • Building for 1 user? Cool. Building for 1,000 concurrent users? Different architecture.

4. Async everywhere or nowhere

   • Mixing sync/async is a trap. Commit to async.

5. Decouple everything you can

   • Writes and reads
   • Real-time and batch
   • Hot data and cold data

---

The Metrics That Matter

I obsess over these now:

✓ P50 latency (average user)
✓ P99 latency (worst 1% of users)
✓ Throughput (requests/second)
✓ Connection pool utilization
✓ Database CPU
✓ Kafka lag (messages waiting to process)
✓ Redis hit rate

If any of these look bad, I can drill down instantly.

---

What This Taught Me About Solo Development

Building alone means:

❌ No code reviews to catch this
❌ No senior engineer to ask "Why are you doing it this way?"
❌ No infrastructure team to handle scaling

✅ But also: No meetings, no politics, no "that's how we've always done it"
✅ And: I get to make these discoveries myself (painful, but educational)

The mistakes hurt more when you're solo.

But the lessons stick harder.

---

The Gotcha Nobody Tells You

Here's the thing everyone skips over in tutorials:

Your code can be perfect and your architecture can still suck.

• FastAPI is fast ✅
• PostgreSQL is fast ✅
• Your connection pool is wrong ❌ = System is slow

I could have written the most beautiful async code in the world and still gotten 20ms responses per 1,000 requests.

Architecture beats code quality every time.

Design right first. Optimize code second.

---

TL;DR

• Built direct-to-database architecture, got bottlenecked at 50 rps
• Found the problem: connection pool too small (5 connections)
• Fixed pool, improved to 150ms response time
• Real solution: Kafka + async + fire-and-forget
• Achieved 1,000+ rps with 20ms latency
• Lessons: Profile first, decouple read/write, async everywhere, think about concurrency

Big lesson: One architecture decision (direct DB writes vs Kafka) meant 20x difference in throughput.

That's why design matters more than code quality.

---

GitHub: beltagyy/vehicle-metrics
Author: Mohamed ElBeltagy (@beltagyy)
Topic: Architecture | Performance | Real-World Engineering