Real‑time AI voice interviews and 100 % availability even when Redis, the email provider, or the AI itself goes down.
The One‑Line Hook
Evalio runs live AI‑powered voice interviews that adapt to the candidate, remember every session, and score across six dimensions — all while the backend is designed to never return a 500, no matter which dependency fails.
Constraints That Shaped Every Tradeoff
Before I wrote a single line of business logic, I locked in four non‑negotiables:
| Constraint | Why it mattered |
|---|---|
| Real‑time audio latency < 2 s | A voice interview with lag feels broken. Candidates leave. |
| 100 % availability under dependency failure | Users trust me with their practice time. Losing a session because an email API errored is a betrayal of that trust. |
| Ship fast, alone | I'm a single developer. I can't afford Kafka clusters, service meshes, or six microservices. Every abstraction must earn its keep. |
| End‑to‑end type safety | No undefined is not a function at 2 AM because the DB schema and the frontend types drifted apart. The compiler catches everything. |
Architecture (The 30‑Second Tour)
Two servers, one monorepo.
Browser → nginx
├── HTTP :3000 (Elysia) → Auth, CRUD, Evaluation
└── WS :8080 (ws) → Audio stream, Gemini relay
→ Gemini 2.5 Flash Live API
Redis → Cache, Queue, Rate‑limiting
Postgres → Sessions, Turns, Users, Resumes
Resend → Email notifications
Why two servers instead of one? Elysia supports WebSocket on the same port. I deliberately separated them because:
- HTTP requests are short (milliseconds). WS connections live 30–60 minutes. A traffic spike on one shouldn't starve the other.
- They scale independently. When the user base grows, the WS server needs more memory per connection; the HTTP server needs more CPU. Different profiles, different scaling.
The tradeoff is operational complexity — two ports, two health checks, two services in the Kubernetes manifest. Worth it.
Key Design Decisions
This is the heart of the article. Every choice I made was a tradeoff. Here's what I chose and why.
1. Redis over PostgreSQL for the Queue
Why: Real‑time interviews need a queue. When all 4 concurrent slots are full, the next candidate waits and sees their position update live. Redis sorted sets give me O(log N) push/pop and O(1) position lookups. The position is pushed to the waiting client via WebSocket — instant feedback, no polling.
Tradeoff: Redis is in‑memory and stateful. If it restarts, the queue vanishes. PostgreSQL would survive a restart but its pub/sub model adds latency and polling complexity.
Mitigation that saved me: The queue isn't essential for correctness — it's a UX nicety. When Redis fails, dequeueNext() returns null, getPosition() returns 0, and the system treats every session as instantly ready. No candidate is blocked by a queue outage. This isn't a hack — it's a deliberate design: a queue that causes errors is worse than no queue at all.
2. Circuit Breaker for Cache (50 Lines, 100 % Availability)
The cache middleware wraps route handlers with get/set Redis. If Redis is slow or down, every request blocks and eventually 500s. That's unacceptable.
The fix is a textbook circuit breaker living in 50 lines at apps/backend/src/lib/cache.ts:
3 failures → circuit OPEN → all cache ops return null → DB fallback
15 s later → circuit HALF‑OPEN → one probe request
Probe succeeds → circuit CLOSED → caching resumes
Probe fails → back to OPEN, wait another 15 s
The mistake I almost made: The original isReady() method didn't delegate to canAttempt(). When the circuit was open, it never probed — the system stayed degraded forever. Found it during benchmark testing. A bug that would have looked like a design failure in production.
Result verified by benchmarks: During a simulated Redis outage, the API maintained 100 % availability with no error rate increase. The only cost was slightly higher DB load (our queries average 1.70 ms, so even direct DB hits are fast).
3. Graceful Degradation at Every Layer
Three independent failure modes, three different recovery patterns:
| Component | Fails How | Fallback | Recovery |
|---|---|---|---|
| Redis | Timeout / connection refused | Cache returns null → DB query | 15 s circuit breaker probe |
| Email (Resend) | 5xx / network error | Write to PendingEmail table |
Exponential backoff: 5 s → 25 s → 125 s |
| Gemini AI | API error | Mark session FAILED, save partial progress | Candidate retries manually |
The pattern is consistent: never crash, never lose data, always have a forward path.
-
Cache fallback: The
cached()middleware catches Redis errors, returnsnull, and the handler queries PostgreSQL directly. The caller never sees a failure. -
Email buffer: If Resend is down, the email is inserted into a
PendingEmailtable. A background job (flushPendingEmails) picks it up later with exponential backoff. 100 % delivery rate guaranteed, even during email provider outages. - Session preservation: If Gemini errors mid‑interview, the session is marked FAILED but all turns and scores are saved. The candidate picks up where they left off.
4. End‑to‑End Types: Prisma → Elysia → Eden → React
This is the single biggest productivity win for a solo developer.
Prisma schema → TypeScript types
↓
Elysia route → uses Prisma types in request/response
↓
Eden Treaty → auto‑generates type‑safe client
↓
React page → imports Eden client, gets full autocomplete
If I rename a field in the Prisma schema, the compiler catches every broken reference across all four packages before I even run the tests. No runtime surprises. No "why is this undefined" debugging sessions at midnight.
Tradeoff: Tight coupling between backend and frontend types. In a monorepo with exactly one frontend, this is a feature. If I ever need a public API for third‑party consumers, I'd split the contract into OpenAPI/Swagger.
5. Prompt Layering over Monolithic Prompts
The AI's system prompt is assembled dynamically from 10 independent layers:
Objective + Candidate History + Company Context
+ Role Context + Interview Style + Interaction Depth
+ Resume + GitHub + Job Description + Evaluation Dimensions
Why: A single 3000‑word system prompt is unmaintainable. You can't A/B test "Role" separately from "Style" when they're in one blob. With layers, I can tweak the roleContext.ts file and know exactly what changed.
Impact weighting (measured empirically):
- Role context: ~60 % of the interview quality
- Interview style: ~25 %
- Company context: ~10 %
- Depth level: ~5 %
Tradeoff: ~200 lines of assembly logic instead of a single string. Worth every line for debuggability.
6. Gemini 2.5 Flash over GPT‑4o for Voice
Why: Gemini's API natively supports bidirectional audio streaming. The browser mic → WebSocket → Gemini → audio response pipeline completes in ~500 ms. GPT‑4o requires a three‑stage pipeline (STT → LLM → TTS), which adds 1.5–2 s of latency.
Tradeoff: GPT‑4o gives better‑quality answers on complex technical questions. But in a voice interview, latency destroys the experience faster than imperfect answers. A 2‑second delay makes the AI feel disengaged; a 500 ms delay feels like a real person thinking.
Benchmarks That Validate the Design
Numbers are meaningless without context. Here's what each benchmark proves:
Database Performance
| Query | Total Time | Scan Type |
|---|---|---|
| Interview list (user + date) | 5.13 ms | Index Scan |
| Rate‑limit count (7‑day window) | 0.93 ms | Index Scan |
| Score trend (last 5 sessions) | 0.29 ms | Index Scan |
| Interview detail (with joins) | 1.16 ms | Index Scan |
| Refresh‑token lookup | 1.01 ms | Index Scan |
| Average | 1.70 ms | — |
What this proves: The composite indexes on InterviewSession(userId, createdAt) and (userId, status, createdAt) eliminated sequential scans. Without them, these queries would take 50–200 ms at 10k rows. Estimated improvement: 29–117×.
API Throughput
| Concurrency | Throughput | Avg Latency | P95 | Errors |
|---|---|---|---|---|
| 10× | 19,190 req/s | 0.4 ms | 1.2 ms | 0 % |
| 25× | 19,355 req/s | 0.9 ms | 1.9 ms | 0 % |
| 50× | 23,318 req/s | 1.4 ms | 5.0 ms | 0 % |
| 100× | 16,262 req/s | 3.9 ms | 6.6 ms | 0 % |
What this proves: The system saturates around 50 concurrent connections on an M3 MacBook. Beyond that, throughput drops slightly due to Bun's event loop pressure. But 23 k req/s is well above any realistic load for a bootstrapped SaaS.
Circuit Breaker & Degradation
- Circuit breaker: 3 failures → open → 15 s → half‑open → recovery. Verified end‑to‑end.
- Degradation: Redis outage → 100 % API availability. Email outage → 0 % delivery loss.
- Audio latency: ~500 ms end‑to‑end (estimate, measured locally without production network conditions).
What I'd Do Differently at Scale
The current design works for a solo‑dev bootstrapped product. If I woke up tomorrow with 10 000 concurrent users, here's what would break first and how I'd fix it:
1. Replace Redis Queue with Kafka (or PostgreSQL SKIP LOCKED)
Redis sorted sets work well until a node restarts mid‑interview and the queue state disappears. At scale:
- Kafka: Durable, replayable, partition‑aware. Overkill today, necessary when queue state becomes a correctness requirement.
-
PostgreSQL
SKIP LOCKED: Simpler, no new infrastructure, but polling adds latency for position updates.
2. Split the WebSocket Server into Its Own Autoscaling Group
Currently both servers live in one Kubernetes deployment. WS connections pin memory and CPU — an interview session consuming 200 MB for audio buffers shouldn't compete with HTTP requests for the same resources.
At scale: separate Deployment for evalio-backend-ws with its own HPA based on active connections, while the HTTP server scales on request rate.
3. CDN for Audio Response Caching
Every audio chunk from Gemini is streamed through the WS to the client. If I cached common responses (greetings, explanations) on a CDN, I could reduce Gemini API calls by an estimated 20–30 %.
The tricky part: audio responses are ephemeral and session‑specific. Only deterministic, non‑personalized chunks can be cached.
4. Shard by User ID
The current queue is a single sorted set. With 10 k concurrent users, ZRANGE operations on a set with millions of entries become noticeable. Sharding by userId % SHARD_COUNT would distribute load across N Redis instances.
5. Real Monitoring (Not Just Logs)
Today I rely on circuit breaker logs and manual health checks. At scale, I'd add:
- Prometheus metrics for cache hit ratio, queue depth, circuit breaker state
- Alerts for circuit‑breaker open events and email buffer growth
- Tracing across the audio pipeline to pinpoint latency sources
Lessons Learned
A cache that 500s is worse than no cache. The circuit breaker is 50 lines of code that made Redis a non‑critical dependency instead of a single point of failure.
Graceful degradation isn't optional. Three fallback mechanisms — cache, queue, email — cover every external dependency. None of them are complex. All of them prevent user‑visible errors.
Type‑safety is a force multiplier for solo devs. The Prisma → Elysia → Eden pipeline caught dozens of refactoring mistakes before they reached production. I'd trade a few percentage points of runtime performance for compile‑time safety any day.
Benchmarks exposed a bug that would have been catastrophic. The
isReady()→canAttempt()delegation bug was invisible during development. Only the benchmark's failure‑injection test revealed it. Always stress‑test your failure modes.
Built with Bun, Elysia, React 19, PostgreSQL, Redis, and Gemini 2.5 Flash. Deployed on Kubernetes via ArgoCD.
Check it out: Evalio
Found a bug or have an idea? Open an issue.
Top comments (0)