Architecting for Zero: Building an Event-Driven, Scale-to-Zero AI Platform

#architecture #ai #serverless #cloud

Most modern AI applications follow an intuitive, straightforward pattern: the browser sends an HTTP POST, the API server invokes an LLM SDK, waits 15 seconds while streaming chunks back over the wire, and saves the result to a database.

It works smoothly in development. But in production, this simple synchronous pattern quickly reveals its cracks:

Vercel / Edge function timeouts abruptly cut off long-running multi-step agent graphs.
Traffic spikes lock server threads, ballooning costs and stalling web traffic.
Idle compute burns cash: keeping GPU-ready or high-memory container clusters running 24/7 when traffic fluctuates to zero overnight bleeds resources.
Dropped client connections kill execution: if a user on mobile loses cell signal mid-generation, the entire inference is lost, wasting tokens and compute.
When architecting Rewire—an AI platform orchestrating safety-critical cognitive reframing, Neo4j GraphRAG, and longitudinal mental health tracking—we set a clear architectural constraint:

True Zero-Scale Economics: Baseline idle cost must be near $0/month, yet the system must instantly burst to handle concurrent multi-stage AI graphs without dropping tokens, exhausting database connections, or timing out web requests.

Here is the high-level system architecture, the mechanics of our scale-to-zero pipeline, and the real-world pitfalls you encounter when taking this approach.

The High-Level Topology
Rewire is built as a decoupled monorepo (powered by Turborepo and pnpm). Instead of coupling user-facing authentication and rendering with heavy agentic inference, the architecture is split cleanly across an asynchronous message boundary:

                      ┌──────────────────────────────┐
                      │    User Browser / Client     │
                      └──────────────┬───────────────┘
                                     │ HTTP / SSE
                                     ▼
                      ┌──────────────────────────────┐
                      │  Next.js 16 BFF (apps/web)   │
                      │  Auth, tRPC, UI & SSE Stream │
                      └──────┬────────────────▲──────┘
         1. Enqueue Job      │                │ 4. Read Stream Events
         (Sub-100ms response)│                │    (XREAD with recovery)
                             ▼                │
                      ┌──────────────┐ ┌──────┴──────────────┐
                      │   AWS SQS    │ │    Redis Streams     │
                      │  (Job Queue) │ │   (ai:run:{runId})   │
                      └──────┬───────┘ └──────▲──────────────┘
         2. SQS Trigger      │                │ 3. Publish Tokens
            Event            ▼                │    & Deltas
                      ┌───────────────────────┴──────┐
                      │ NestJS AI Worker (apps/api)  │
                      │  AWS Lambda (Scale-to-Zero)  │
                      │  LangGraph + Neo4j GraphRAG  │
                      └──────────────┬───────────────┘
                                     │
                                     ▼
                      ┌──────────────────────────────┐
                      │ PostgreSQL (Prisma ORM)      │
                      │ Single Source of Truth       │
                      └──────────────────────────────┘

The system operates across three autonomous tiers:

The Edge & BFF Tier (apps/web): Built with Next.js 16 (App Router), React 19, and Better Auth. It handles authentication, renders UI components, and serves as an ultra-fast gateway.
The Elastic Queue & Stream Buffer: AWS SQS decouples ingestion from execution. Redis Streams (ai:run:{runId}) serves as an ephemeral, replayable token buffer.
The AI Worker Tier (apps/api): A modular NestJS 11 application compiled into an AWS Lambda function triggered exclusively by SQS events. It drives LangGraph cognitive state machines, verifies clinical safety gates, and retrieves therapeutic context from Neo4j GraphRAG.
Anatomy of an Asynchronous Interaction
What does a message cycle look like when client requests never wait for an LLM?

sequenceDiagram
autonumber
actor User as User Browser
participant Web as Next.js BFF (Vercel Serverless)
participant DB as PostgreSQL (Prisma)
participant SQS as AWS SQS
participant Worker as NestJS Worker (AWS Lambda)
participant Redis as Redis Streams
participant LLM as LLM Provider / Neo4j
User->>Web: Send Prompt ("I'm feeling overwhelmed...")
Web->>DB: 1. Persist User Message & AiRun(status: "queued")
Web->>SQS: 2. Enqueue Lean Payload { runId, conversationId, userId }
Web-->>User: 3. Return { runId } immediately (< 100ms)
par Ephemeral Streaming
User->>Web: GET /api/ai/runs/:runId/stream (SSE)
Web->>Redis: XREAD from ai:run:{runId}
and Serverless Worker Execution
SQS->>Worker: Trigger Lambda Event
Worker->>DB: Update AiRun -> "running"
Worker->>Redis: XADD run.started
Worker->>Worker: Run LangGraph: Load Context -> Safety Triage -> Neo4j GraphRAG
loop Token Generation
LLM-->>Worker: Stream Token Chunk
Worker->>Redis: XADD message.delta
Redis-->>Web: Event Delivered
Web-->>User: SSE Chunk Rendered in UI
end
Worker->>DB: Commit Final Message & Mark AiRun "completed"
Worker->>Redis: XADD run.completed
Web-->>User: Close SSE Connection
end

The Sub-100ms Handshake When the user submits a prompt, the Next.js API route does not invoke LangGraph or call an AI model.

Instead, it validates the request using a shared Zod schema from packages/validation, writes the message and an AiRun record (flagged as queued) to PostgreSQL, and pushes an ultra-lean pointer payload to AWS SQS:

{ "runId": "run_cuid123", "conversationId": "conv_cuid456", "messageId": "msg_cuid789", "userId": "usr_cuid000"}
The web server responds to the browser with { runId } in under 100ms. No gateway timeouts, no thread exhaustion, no risk of a 30-second multi-step graph stalling the front door.

Event-Driven Wake-Up AWS SQS triggers the NestJS worker deployed on AWS Lambda. If there is zero traffic, zero Lambda instances are running.

When a job arrives, Lambda spins up, pulls authoritative context from PostgreSQL, and hands the state over to a deterministic LangGraph state machine:

[START]
│
▼
[Load Context] ──► [Safety Triage] ──┬──► (Acute Risk) ──► [Crisis Helpline Flow] ──┐
│ │
└──► (Safe) ──► [Neo4j GraphRAG] │
│ │
▼ │
[LLM Generation] │
│ │
▼ ▼
[Response Verification] ◄──────────┘
│
▼
[Persist & END]

Decoupled Streaming via Redis Streams How does a serverless Lambda stream real-time text back to a browser without opening a direct WebSocket connection?

As tokens arrive from the LLM, the Lambda worker pushes structured events (message.delta) into a dedicated Redis Stream: ai:run:{runId}.

Meanwhile, the client connects to an SSE endpoint (/api/ai/runs/:runId/stream) served by Next.js. The SSE route reads from Redis Streams using XREAD and pipes events down to the client.

Why Redis Streams Instead of Pub/Sub or WebSockets?
Replayability & Resilient Reconnection: Redis Pub/Sub is fire-and-forget. If a user walks into an elevator and drops connection for 3 seconds, all missed tokens vanish. With Redis Streams, events are logged. When the client reconnects, it passes its Last-Event-ID, and the SSE handler resumes streaming exactly where it left off.
Serverless Compatibility: Maintaining persistent, bidirectional WebSockets on serverless runtimes requires specialized gateway infrastructure (e.g., AWS API Gateway WebSockets). Redis Streams + SSE works over standard HTTP/2.
Ephemeral Footprint: Each run’s stream has a short Time-to-Live (TTL). Once the run completes, the stream expires, keeping memory usage minimal.
Why Scale-to-Zero?
True Cost Elasticity: A startup or side project with sporadic traffic shouldn’t pay $150–$300/month for dedicated ECS/EKS clusters or container instances just to wait for user prompts. In this setup, idle compute cost is practically $0.
Infinite Headroom for Spikes: If 500 users submit prompts simultaneously, SQS buffers the traffic cleanly. AWS Lambda scales out concurrently to process the jobs without bringing down the web tier.
Protection Against Provider Outages: If an upstream LLM provider rate-limits or throttles requests, SQS handles exponential backoffs and dead-letter queueing (DLQ) without crashing the user’s browser session.
Architectural Pitfalls & Trade-Offs
While the zero-scale paradigm is cost-effective and resilient, it introduces non-trivial engineering trade-offs that every architect must evaluate.

The “Double Cold Start” Problem When both your web tier (Next.js on Vercel) and your AI worker (NestJS on Lambda) scale to zero, an idle system encounters compounding cold starts:

Next.js Serverless Function: ~300ms–800ms
SQS Delivery Latency: ~50ms–200ms
NestJS AWS Lambda Boot: 1,200ms–2,500ms (loading TypeScript metadata, Prisma engine, LangChain modules)
While the user receives an immediate UI response acknowledging their message, their Time-to-First-Token (TTFT) after an idle period can reach 3–4 seconds.

Mitigation: Optimize the Lambda bundle using esbuild/tree-shaking, minimize global module imports in NestJS, and use Lambda SnapStart or lightweight container provisioning if sub-second first-token response is mandatory.

Database Connection Flooding Serverless workers scale horizontally with zero shared memory. When 100 SQS messages arrive at once:

100 Lambda containers spin up in parallel.
Each container attempts to open a connection pool via Prisma to PostgreSQL.
PostgreSQL default max connection limit (e.g., 100–200 connections) is instantly saturated, causing Connection pool timeout errors.
Mitigation: You cannot connect directly to vanilla Postgres from serverless workers at scale. You must place a connection pooler like AWS RDS Proxy, PgBouncer, or use serverless-native drivers (Prisma Accelerate / Neon Serverless Driver) to multiplex connections.

Because i use the manage postgres database instance so there are no need to required PgBouncer on the top, pooler is associated to the layer of the database so our lambda can reuse the connection not make or create a new connection on each invocation.

The Cost Inversion Point Scale-to-zero is unbeatable at low to moderate traffic:

Monthly Active Workflows Dedicated Server (Fargate / EKS) Scale-to-Zero (Lambda + SQS + Redis)
Low (0 – 10,000 runs) ~$80 – $200 / mo ~$0 – $5 / mo
Medium (100,000 runs) ~$150 – $250 / mo ~$20 – $40 / mo
Extreme Scale (10M+ runs) ~$400 – $800 / mo ~$1,200 – $2,500+ / mo
At millions of continuous requests per day, pay-per-invocation Lambda compute, SQS API calls, and Redis read/write operations surpass the predictable monthly cost of a well-tuned auto-scaling container cluster.

The Verdict: When Should You Build This?
The Decoupled Zero-Scale Architecture is the ideal blueprint if:

✅ Your AI workloads involve multi-step reasoning, tool calls, or RAG that can exceed standard HTTP request timeouts.
✅ Your application experiences variable or unpredictable traffic patterns, making dedicated 24/7 server clusters financially wasteful.
✅ Resilience is paramount: client disconnections, mobile signal drops, or provider rate limits must never corrupt state or lose generated content.

By strictly separating your fast web gateway from your heavy agent execution, and bridging them with a durable queue and an ephemeral stream, you get the best of both worlds: a lightning-fast user experience with zero idle infrastructure overhead.

Visit HeyRewire platform