DEV Community

Duong Phuc Khanh (Kido)
Duong Phuc Khanh (Kido)

Posted on

How I Built a 7-Gate Multi-Agent Engineering Harness (And Survived a 6:00 AM Race Condition Storm)

#agents #architecture #performance #softwaredevelopment

At 06:00 AM in Da Nang, Vietnam, while most people are just waking up, nearly 1,000 calisthenics athletes open the Winter Arc mobile PWA simultaneously to log their morning workouts and protect their daily streaks.

Behind the scenes, there is no $5,000/month AWS Kubernetes cluster. The entire production ecosystem—serving 1,100+ weekly active users with an average session duration of 8 minutes 04 seconds and a P75 LCP of 1.12s—runs smoothly on a single 8GB RAM bare-metal Linux VPS.

I'm Duong Phuc Khanh (mostly known in the engineering community as Kido or Khánh Dương, @khanhduong22). Over the past 7+ years—from winning First Prize at Startup Runway 2018 (featured in Báo Đà Nẵng) and spending 6 years architecting financial systems at FPT Software (SBI Lab) to leading AI-Native systems at Tech Pulse—I've obsessed over one question:

How can a tiny, AI-augmented engineering team ship like a 20-person department without sacrificing production reliability?

Here are the two core architectural pillars that made it possible—and the full story behind my 114-episode engineering war journal, Kido Chronicles.

Pillar 1: Taming the 6:00 AM Check-In Storm with Valkey Distributed Locks

When hundreds of athletes tap "Complete Workout" at the exact same second—often on flaky 4G connections inside basement gyms—double-taps and network retries create a textbook TOCTOU (Time-of-Check to Time-of-Use) race condition.

Without concurrency control, concurrent requests read the same streak state in PostgreSQL, resulting in duplicate streak increments, corrupted leaderboard scores, and database lock contention.

The Fix: Sub-5ms Idempotent Mutex in Valkey 8

Instead of relying on heavy SERIALIZABLE database transactions that choke connection pools under burst load, we guard the state transition at the edge using an atomic Valkey/Redis Distributed Lock (SET NX PX) paired with an idempotency key:

// Atomic check-in guard (Valkey 8 + NestJS)
const lockKey = `lock:checkin:${userId}:${workoutDate}`;
const acquired = await this.valkey.set(lockKey, requestId, 'PX', 5000, 'NX');

if (!acquired) {
  // Concurrent retry detected — return cached idempotent state in <2ms
  return this.getCachedCheckInStatus(userId, workoutDate);
}
Enter fullscreen mode Exit fullscreen mode

Combined with a Service Worker Offline Outbox on the client and Caddy 2 native reverse-proxying, our p99 check-in latency dropped from 800ms+ to under 5ms with zero duplicate writes.

Pillar 2: The 7-Rung "Ponytail" Philosophy & agent-harness-kit

To maintain 4x delivery velocity across products like Index.vn, Linkpul (50,000+ req/sec short-link gateway), and Winter Arc, I built agent-harness-kit—a multi-agent orchestration framework for Claude Code, Cursor, Gemini, and Google Antigravity.

Every autonomous sub-agent operates in an isolated Git worktree and is governed by what I call the 7-Rung Ladder:

  1. Does this need to exist at all? (YAGNI)
  2. Is it already in the codebase?
  3. Does the standard library solve it?
  4. Does a native platform feature (HTML/CSS/PostgreSQL constraint) cover it?
  5. Does an already-installed package handle it?
  6. Can it be written in one clear line?
  7. Only then: write the minimum production-grade code.

Before any PR can merge, it must pass 7 automated verification gates: AST linting, strict TypeScript typecheck, SonarQube smell checks, unit tests, and headless Playwright E2E tests with real-time video telemetry.

Explore the Full Architecture & War Stories

If you enjoy deep-dive postmortems, distributed systems math, and real production numbers:

Top comments (0)