DEV Community: GeekyAnts

How We Engineered a Three-Path Reward Processing System for a Real-Time Sports Platform

GeekyAnts India Pvt Ltd — Fri, 13 Nov 2026 10:29:04 +0000

How one score event triggers 20+ reward evaluations across sync, async, and scheduled paths

1. The Problem

A player steps into one of our Core Soccer Tech pods, plays a 90-second game, and scores 847 points. The kiosk needs to respond instantly: show the score, trigger a celebration animation if they hit a personal high, and display any XP (experience points) earned.

But behind that single score, the system has to answer over 20 questions. Did they beat their high score? Did they unlock a new game mode? Did that score push them past a reward threshold? How many of today's seven daily missions did that game advance? What about the eight weekly missions running in parallel? And the eight seasonal missions that span the entire quarter?

All of this from one POST request.

The naive approach is to evaluate everything synchronously inside the score submission endpoint. The kiosk waits, the database holds a long transaction, and every player in the facility feels the lag. At 75+ touches per 90-second game and multiple pods running simultaneously, that is not a theoretical concern. It is a production one.

We needed to split the work. Some rewards have to be synchronous because the kiosk needs the data immediately for animations. Some can be deferred by a few seconds. Some only make sense at the end of a week or quarter.

The result is a three-path reward processing engine: synchronous rewards evaluated inside the score transaction, asynchronous processing via a BullMQ worker for missions and achievements, and cron-driven distribution for leaderboard rewards.

2. Architecture Overview

The backend is a NestJS 11 application on Node.js 24. It talks to PostgreSQL through the Knex query builder (via nestjs-knex); there is no ORM (Object-Relational Mapper), so every query in this article is either a Knex builder call or raw SQL. Schema changes, unique constraints, and database triggers all live in Knex migrations. Background jobs use BullMQ 5 on Redis, and scheduled jobs use @nestjs/schedule.

The reward engine sits between score ingestion and four output tables. Every reward, regardless of its source, writes to one or more of these: player_transaction (the universal ledger of XP and CC, the platform's credit currency), player_metadata (running totals maintained by database triggers), player_achievement (one-time milestones), and player_notification (kiosk alerts).

Three ingestion paths, including classic and weekly pattern games, OnePod batch sessions, and battle rooms, all converge into a single addScore() method. From there, processing splits into three paths based on urgency:

Sync Path (green): Runs inside the database transaction. Returns rewards in the HTTP response so the kiosk can show celebration animations immediately. Handles single game rewards, high score detection, and game mode unlocks.

Async Path (purple): Score events are enqueued to a BullMQ queue after the transaction commits. A background worker evaluates achievements, badges, daily missions, weekly missions, seasonal missions, and level-ups with a concurrency cap of 5 parallel jobs.

Scheduled Path (red): Cron jobs run on fixed schedules in the America/Denver timezone. Every Monday: the weekly pattern rotates at 12:00 AM, weekly leaderboard rewards are distributed at 12:05 AM, and the "top 10 in your age group" seasonal mission is updated at 12:10 AM. Battle season rewards run at 12:05 AM on the first day of each quarter. These jobs rank players and distribute CC, XP, and badges based on leaderboard position.

3. The Unified Entry Point

All three ingestion paths converge into addScore(). This is a deliberate architectural choice: a single method handles score creation and sync reward evaluation inside one database transaction, then dispatches the async event after that transaction commits.

Why a Single Entry Point?

Classic games, OnePod batch sessions (2–4 players ranked together), and battle rooms (deferred scoring until all players finish) have very different ingestion patterns. But once a score is created, the reward logic is identical. By funneling everything through addScore(), we guarantee that every score, regardless of how it was submitted, gets the same reward treatment.

Transaction Boundary

The transaction boundary is the most important design decision in this system. Inside the transaction, we run eight operations in a fixed order:

Fetch the previous high score for this player and game mode (weekly pattern scores excluded).
Increment player.game_count and read the new value back.
Insert the score into soccer_game_score.
Insert the game-specific record (one_pod_score or weekly_pattern_score) if the game style needs one.
Insert a NewGameUnlock achievement if the new game count crosses an unlock threshold.
Evaluate single game rewards (section 4) and, if any qualify, record the XP and write one player_transaction row.
Insert a PersonalHighScore achievement if the score beats the previous high.
Log today's play date into player_daily_play_log.

The transaction callback returns, Knex issues COMMIT, and only then is the ScoreEventPayload enqueued to BullMQ. Enqueueing is deliberately outside the transaction. If the enqueue fails, the score is saved but async rewards are lost, and because the enqueue call is not caught, the kiosk receives a 500 for a score that was in fact persisted. We accepted this tradeoff to keep the transaction short and avoid coupling score persistence to Redis availability.

const result = await this.knex.transaction(async trx => {
  const prevHighScore = await getPrevHighScore(trx, playerId, req.game_mode_id)

  // Step 1
  const [{ game_count }] = await trx('player')
    .where('id', playerId)
    .increment('game_count', 1)
    .returning('game_count')

  // Step 2
  const [createdScore] = await trx('soccer_game_score')
    .insert({
      player_id: playerId,
      game_mode_id: req.game_mode_id,
      score: req.score,
      game_style: req.game_style
    })
    .returning('*')

  // Step 3
  // Steps 4 to 8 run here, all on `trx`
  return {
    createdScore,
    prevHighScore,
    newGameCount: game_count
  }
})

// COMMIT has happened. Now, and only now, hand the event to the worker.

await this.scoreEventsQueue.addScoreEventJob({
  player_id: playerId,
  score_id: result.createdScore.id,
  score: result.createdScore.score,
  game_mode_id: result.createdScore.game_mode_id,
  game_style: result.createdScore.game_style,
  prev_high_score: result.prevHighScore,
  new_game_count: result.newGameCount,
  season: result.createdScore.season,
})

Battle rooms and OnePod sessions insert several scores in one outer transaction, so they pass their own trx into addScore(). Battle completion also passes skipEnqueue = true, collects one payload per player, and enqueues them all after the outer commit.

The ScoreEventPayload The payload sent to the async worker carries everything needed to evaluate all downstream rewards without additional database lookups at the entry point:

{ player_id, score_id, score, game_mode_id, game_style, prev_high_score, new_game_count, season }

prev_high_score is included because the worker needs it for score level achievement checks. Without it, the worker would need to query historical scores, adding latency and database load to every async job.

4. The Sync Path: Immediate Rewards

These rewards are evaluated inside the addScore() transaction and returned in the HTTP response. The kiosk uses them to display immediate feedback: XP earned animations, high score celebrations, and game unlock screens.

Single Game Rewards

Four reward types are evaluated per score, and all can stack:

Reward	Condition	Notes
`play_any_game`	Always qualifies	Base XP for playing
`score_x`	`score >= floor(third_best_score * threshold_pct / 100)`	Needs at least 3 scores in the mode
`score_y`	Same formula, higher `threshold_pct`	Harder tier
`new_high_score`	`score > prev_high_score AND games_in_mode >= min_high_score_games`	`min_high_score_games` comes from global config, default 10

Each row is configured in single_game_reward_config (threshold_pct, xp_reward, animation, and a game_style exclusion list). Weekly pattern games skip this block entirely.

The best-score formula is worth noting: instead of comparing against the player's top score, we compare against their third-best score in that game mode. This prevents a single lucky outlier from permanently raising the bar; a player has to post three strong games before the threshold moves.

// Top 3 in this mode, read after the new score was inserted (so it is included)
const topScores = await trx('soccer_game_score').select('score')
  .where({ player_id, game_mode_id })
  .whereNot('game_style', 'weekly_pattern')
  .orderBy('score', 'desc').limit(3)
const thirdBestScore = topScores.length >= 3 ? topScores[2].score : null

// score_x / score_y
if (thirdBestScore !== null && config.threshold_pct !== null) {
  const threshold = Math.floor((thirdBestScore * config.threshold_pct) / 100)
  qualifies = score >= threshold
}

// new_high_score
qualifies = gamesInMode >= minHighScoreGames && score > prevHighScore

If a player has fewer than 3 scores in a mode, thirdBestScore is null and score_x and score_y are not awarded at all. Only play_any_game fires on those first two games. There is one side effect of including the new score in the top-3 query: on exactly the third game in a mode, the new score can be the third best itself, so score_x and score_y automatically qualify for that game.

High Score Detection and Game Unlocks

If the new score beats the player's previous high score (and the game style is not weekly_pattern), a PersonalHighScore achievement is inserted. This check has no game-count gate, so a player on their third game gets the celebration animation but not the new_high_score XP. Similarly, when game_count crosses an unlock threshold, a NewGameUnlock achievement is created with metadata indicating which game mode (CoreOG, Mirror, Xpert, Clutch, or Mirror_X) was unlocked.

5. The Async Path: BullMQ Worker

After the transaction commits, a ScoreEventPayload is enqueued to a Redis-backed BullMQ queue named scoreEvents. A worker picks it up and processes the steps below sequentially.

Worker Configuration

The worker runs in the same Node.js process as the HTTP server with a concurrency cap of 5 parallel jobs. Successful jobs are removed immediately; the last 100 failed jobs are retained for debugging. This is a pragmatic choice: a separate worker process would add deployment complexity for a system that handles scoring bursts measured in dozens per minute, not thousands.

this.queue = new Queue('scoreEvents', { connection: redisConfig })

this.worker = new Worker<ScoreEventPayload>(
  'scoreEvents',
  async job => this.withDeadlockRetry(() => this.processScoreEvent(job.data)),
  { connection: redisConfig, concurrency: 5 },
)

// called by addScore() after COMMIT
await this.queue.add('processScore', payload, {
  removeOnComplete: true,
  removeOnFail: 100,
})

That is the whole configuration. We do not set attempts or backoff, so BullMQ's defaults apply: a job that throws is moved straight to the failed set with no automatic retry. The only retry in this path is application-level. withDeadlockRetry() reruns the job up to three times when PostgreSQL reports a deadlock (error code 40P01), waiting 50 * attempt + random(0..50) milliseconds between tries. Everything else fails once.

Stalled jobs follow BullMQ defaults too: a job lock lasts 30 seconds and is renewed every 15; if the process dies or blocks the event loop long enough for the lock to expire, the stalled checker (every 30 seconds) puts the job back in the wait list once (maxStalledCount: 1). A second stall causes the job to fail. After a final failure the error is logged with the player_id and score_id, the job sits in the failed set until it is pushed out by the 100-job cap, and nothing replaces it. There is no dead-letter queue and no alerting on the failed set today.

Achievement Evaluation (Steps 1–2)

Step 1 checks whether the new score crossed a score-level tier. These configurable thresholds are defined per game mode and can award XP, credits, and badges. Step 2 checks career total points milestones, using a database trigger-maintained counter on player_metadata. Both use an achievementExists(score_id, type) idempotency check before inserting, preventing duplicates if the same event is ever processed twice. A points-badge evaluation runs between step 2 and the missions.

Mission Processing (Steps 3–5)

This is where the bulk of the work happens. Steps 3 through 5 evaluate daily, weekly, and seasonal missions respectively. They run sequentially within a single job, not in parallel. This was a deliberate choice: sequential execution means a database error in daily missions prevents weekly and seasonal from running for that event, but it also means we never have two mission services competing for database connections from the same job. A level-up check runs last, after the seasonal missions.

6. The Mission System: Three Time Horizons

The mission system operates across three time horizons, each with fundamentally different state management patterns. Missions unlock on a player's fifth game overall: games 1 to 4 are skipped, game 5 creates the mission rows, and processing starts from game 6.

Daily Missions

Seven mission types reset every day. On a player's first score each day, getOrCreateDailyMissions() lazily initializes all seven missions with fresh progress counters. Goals are personalized; the TotalPoints mission, for example, sets its goal to the player's third-best Core OG score multiplied by a configurable factor (minimum 1), ensuring the target scales with skill level.

The ScoreYStreak mission is the most interesting pattern: it requires the player to score above a threshold N times in a row. One miss resets progress to zero, so each consecutive successful score increases the cost of breaking the streak. The backend tracks this with a simple conditional: if score >= threshold, increment progress; else, reset to zero.

Weekly Missions

Eight mission types reset every Monday. The week anchor is computed in the database as date_trunc('week', (current_timestamp AT TIME ZONE 'America/Denver')::date). The most complex is TopNScore, where N is locked when the week's missions are created, based on the player's total game count: experienced players must score in their top 5, newer players in their top 10. This graduated difficulty prevents both sandbagging and frustration.

Weekly missions also track consecutive play days using an append-only player_daily_play_log table. Each day a player plays, a row is inserted with INSERT ... ON CONFLICT (player_id, play_date) DO NOTHING (Knex .onConflict([...]).ignore()) for concurrent access safety. The streak count is computed from this log at evaluation time.

Seasonal Missions

Eight missions spanning an entire quarter, with 16 to 200 threshold milestones each. Unlike daily and weekly missions that track per-score progress, seasonal missions use aggregate counters stored in player_season_data. Each score event increments these counters with a single UPDATE, and the mission service compares current values against threshold tables.

The idempotency mechanism here is elegant: each player's mission row stores a last_completed_threshold_id. When processing, the service iterates through thresholds in sort order, skipping anything at or below this ID, and stopping at the first uncrossed threshold. The ID update and reward insert happen in the same database transaction, so if the transaction rolls back, neither persists.

for (const t of thresholds) { // sort_order ascending
  if (mission.last_completed_threshold_id !== null && t.id <= mission.last_completed_threshold_id) continue
  if (t.threshold_value > mission.current_value) break
  newThresholds.push(t)
}
if (newThresholds.length === 0) continue

await trx('player_seasonal_mission').where({ id: mission.id })
  .update({ last_completed_threshold_id: newThresholds.at(-1).id })
await this.levelUpService.recordXp(trx, playerId, season, totalXp, XpSource.SeasonalMission)
await trx('player_transaction').insert({ player_id: playerId, transaction_type: 'add',
  reward_name: 'seasonal_mission', xp_earned: totalXp, free_live_credits: totalCc })

Rotating Game Modes

Both daily and weekly missions include a "play the rotating game mode" mission type. Daily rotation is deterministic: day of year (Denver local date) modulo 4 maps to Mirror, Xpert, Clutch, or Mirror_X. Weekly rotation is derived from the week's start date and stored in a weekly_mission_rotation table with ON CONFLICT (week_start_date) DO NOTHING for concurrent initialization. This keeps missions fresh without requiring manual configuration.

7. The Scheduled Path: Cron Jobs

Four cron jobs handle scheduled game-pattern updates and time-bounded reward processing. All run in the America/Denver timezone to match the facilities' operational hours. The scheduler is @nestjs/schedule (which wraps the cron npm package), and every job passes the same constant:

export const CURR_APP_TIMEZONE = 'America/Denver'

@Cron('0 0 * * 1', { timeZone: CURR_APP_TIMEZONE }) // Monday 12:00 AM: rotate weekly pattern @Cron('5 0 * * 1', { timeZone: CURR_APP_TIMEZONE }) // Monday 12:05 AM: weekly leaderboard rewards @Cron('10 0 * * 1', { timeZone: CURR_APP_TIMEZONE }) // Monday 12:10 AM: top 10 age group mission @Cron('5 0 1 1,4,7,10 *', { timeZone: CURR_APP_TIMEZONE }) // Jan/Apr/Jul/Oct 1, 12:05 AM: battle season rewards

Daylight Saving Time (DST) is handled by the cron library, which evaluates the expression in the named zone, so "12:05 AM Denver" stays 12:05 AM local across the March and November changes. Daily mission dates, the weekly anchor, and the play log use the same zone inside PostgreSQL (current_timestamp AT TIME ZONE 'America/Denver'); the database session itself runs in UTC. Game-mode rotation and the battle season calendar resolve the same zone in Node.js.

Each job is wrapped in a small withRetry() helper that allows three attempts with a fixed five-minute wait between them. A job that fails all three attempts logs the error and waits for its next scheduled run.

Weekly Leaderboard Rewards – Monday 12:05 AM

Every Monday, the system ranks the previous week's weekly pattern scores. Players are split into age group leaderboards (grouped by age bracket and gender) plus an overall leaderboard. Ranks are matched against a leaderboard_reward_config table that maps rank ranges to CC, XP, and badge rewards. A unique index on (player_id, leaderboard_type, season, week_start_date) prevents duplicate awards.

Top 10 Age-Group Update – Monday 12:10 AM

Five minutes after weekly rewards, a second cron processes the seasonal mission that tracks how many weeks a player placed in their age group's top 10. It reads from the player_leaderboard_reward table that the 12:05 AM job just populated.

The five-minute gap is the only dependency mechanism. The 12:10 AM job does not check whether the 12:05 AM job finished; it simply queries last week's top-10 rows and processes whoever is there. If the first job is still running, or is in its 5-minute retry wait, the second job sees a partial or empty result and finishes early. Two properties make this acceptable rather than dangerous. First, the job recomputes each player's count from player_leaderboard_reward as an absolute value rather than an increment and skips the player when the stored value already matches. This allows the job to be rerun manually without double-awarding. Second, the first job's retry schedule (12:10 and 12:15) means the rows usually exist within minutes. Affected players are picked up the next time they finish in the top 10, when the recount includes the missed week. We accept that a player can see this mission update one week late.

Battle Season Rewards – Quarterly

On the first day of each quarter, battle season rewards process. Two leaderboards are computed: total battle points (sum of points from battle royale and OnePod ranked games) and battle efficiency (points-to-expected ratio, preventing pure volume from dominating). Both leaderboards use the same check-then-insert and unique index pattern for idempotency, with the season start date in week_start_date.

8. Idempotency and Known Risks

No distributed system is perfectly reliable. Here is an honest assessment of where our idempotency holds and where it has gaps.

Where It Holds

System	Mechanism	Status
Achievements	`achievementExists(score_id, type)` before every insert	Fully idempotent
Seasonal thresholds	`last_completed_threshold_id` bookmark, updated in the reward transaction	Safe via transaction
Leaderboard rewards	Check-then-insert inside a transaction, unique index as backstop	Fully idempotent
Level-up rewards	`player_level_reward_history` unique on `(player_id, level_config_id, season)` with `ON CONFLICT DO NOTHING`	Fully idempotent

The achievement check is a single indexed lookup:

private async achievementExists(score_id: number, achievement_type: AchievementType) {
  return !!(await this.knex('player_achievement').where({ score_id, achievement_type }).first())
}

Leaderboard rewards use the unique index from the migration plus an explicit check, because the check lets us skip the CC, XP and notification writes cleanly instead of catching a constraint error:

// migration
table.unique(['player_id', 'leaderboard_type', 'season', 'week_start_date'],
  { indexName: 'uq_player_leaderboard_reward' })
await this.knex.transaction(async trx => {
  const existing = await trx('player_leaderboard_reward')
    .where({ player_id, leaderboard_type, season, week_start_date }).first()
  if (existing) return

  await trx('player_leaderboard_reward').insert({ player_id, leaderboard_type, season, week_start_date,
    rank, cc_awarded: tier.cc_reward, xp_awarded: tier.xp_reward, badge_config_id: tier.badge_config_id ?? null })
  if (tier.cc_reward > 0) {
    await trx('player_transaction').insert({ player_id, transaction_type: 'add',
      free_live_credits: tier.cc_reward, free_virtual_credits: 0, xp_earned: 0 })
  }
  if (tier.xp_reward > 0) {
    await this.levelUpService.recordXp(trx, player_id, season, tier.xp_reward, XpSource.WeeklyPattern)
  }
})

Wherever the article says a row is inserted with conflict protection, the actual statement is PostgreSQL's INSERT ... ON CONFLICT (...) DO NOTHING, written in Knex as .onConflict([...]).ignore(). Reward history rows use .onConflict([...]).merge([...]), which becomes ON CONFLICT ... DO UPDATE SET.

Where It Has Gaps

Daily and weekly missions track progress with increment-based counters, and the seasonal player_season_data counters (games played, total points, days played) are incremented the same way. If a score event is processed twice, those counters get double-incremented. The completion rewards are protected by a separate reward history table and the threshold bookmark, but the progress inflation can cause missions to complete earlier than intended.

With no BullMQ retries configured, a second run of the same event can come from only two places: the in-process deadlock retry, which re-runs processScoreEvent() from the top after a partial write (the function is not one transaction), and BullMQ's stalled-job recovery, which re-queues a job once if its lock expired mid-run.

Known Risks We Accept

Pool exhaustion: The HTTP server and BullMQ workers share a single database connection pool. Under high scoring load, 5 concurrent workers multiplied by multiple queries per job plus HTTP traffic can compete for connections. This has not been a production issue yet, but it could become a scaling constraint if scoring volume increases significantly.

Sequential mission processing: Steps 3 to 5 in the worker run sequentially. A failure in daily missions prevents weekly and seasonal from running for that score event. We chose this over parallel execution to avoid connection pool pressure, but it means one service's bug can cascade.

Single-process cron: The cron jobs have no distributed lock. The API runs as one Node.js process today; running two instances would fire every cron job in both, and the second run would fail on the unique index rather than double-award.

Building Reward Systems That Can Handle Different Processing Needs

A reward system does not need to process every event in the same way. In this case, separating immediate, asynchronous, and scheduled work keeps player-facing responses fast while moving heavier evaluations and time-based rewards to paths better suited to them. The approach has been running in production across multiple Core Soccer Tech facilities, handling over 38 million scored points from 7,000+ players. The architecture also shows why transaction boundaries, idempotency, queue behavior, failure handling, and known system constraints need to be considered together when designing production systems. These are the same concerns that shape backend engineering for applications that need to support growing workloads without making every workflow part of one synchronous request.

Scaling Down Before Scaling Up: Why Bigger Servers Don’t Fix Bad Architecture

GeekyAnts India Pvt Ltd — Fri, 13 Nov 2026 09:27:58 +0000

This blog explores how inefficient backend architecture can cause performance issues even under low traffic, covering practical ways to reduce database load, API latency, and resource usage before scaling infrastructure.

Why Scaling Infrastructure Wasn’t the Answer

When an application becomes slow, the first instinct is often predictable: add more memory, upgrade the server, increase database capacity, or scale vertically.

Sometimes, that works until it doesn’t.

We experienced this firsthand when an application that had been running smoothly on a 64 GB server started crashing after being moved to a basic 2 GB instance.

What made the situation unusual was the workload. There were only 4–5 active users, a limited dataset, and no major traffic spikes. Yet the application experienced database connection spikes, high API latency, frequent request timeouts, and unstable performance.

This wasn’t a scaling problem. It was an architecture problem.

The Problem: When a 64 GB Server Hid the Real Issue

What we observed

The infrastructure changed, but the workload did not.

Our initial assumption was that the smaller server lacked sufficient resources. But the connection behavior and latency suggested something deeper.

Database connections were spiking despite low traffic.
API latency remained high.
Requests frequently timed out.
Connection pooling stabilized the number of connections but did not resolve the latency issue.

The Architecture Before Refactoring

The application used Hasura, Node.js services, PostgreSQL, and multiple Dockerized services. The biggest issue was in the request path for Hasura Actions.

The problematic request lifecycle

The Node.js service was calling back through Hasura instead of using a direct database path where appropriate.

This created a circular dependency between Hasura and Node.js. A single request could therefore involve multiple network hops, GraphQL execution layers, serialization and parsing, and additional database connection usage.

Hasura → Node.js → Hasura → PostgreSQL was doing more work than the request actually required.

Why Connection Pooling Wasn’t the Final Fix

Symptom vs. root cause

PgPool/PgBouncer helped control the number of active database connections, but it did not reduce the amount of work being performed per request.

That distinction mattered. We had solved connection management, not query efficiency or architectural overhead.

Connection pooling controls concurrency; it does not automatically make inefficient work efficient.

Breaking the Loop: Rethinking the Data Access Layer

The simplified architecture

For straightforward database operations, remove unnecessary internal round trips.

Where Node.js remains valuable

We reduced unnecessary Hasura Actions and removed circular Hasura → Node.js → Hasura flows where direct database access was more appropriate.

Hasura continued to handle client-facing GraphQL access, CRUD, filtering, and aggregations. Node.js remained responsible for business logic that genuinely required an application service, authentication flows, external integrations, and asynchronous workflows.

The goal was not to remove Hasura or Node.js. It was to give every layer a clear responsibility.

The Cost of Over-Abstraction

GraphQL, microservices, Hasura Actions, and Node.js each solve real problems. The issue occurs when every request is forced through every abstraction.

More network hops
More serialization/deserialization
Higher latency
Duplicate responsibilities
More database connection churn
Harder debugging and observability

Abstraction should reduce complexity—not increase the number of steps required to complete a request.

Database Optimization: Reducing Work at the Source

1. Eliminate Redundant Queries

Redundant request pattern

Optimized request pattern

We consolidated repeated reads into fewer, better-structured queries. This reduced query count, database pressure, network traffic, and response time.

2. Solve N+1 Query Patterns

N+1 pattern

Optimized pattern

Instead of repeatedly fetching related entities, we used joins, CTEs, and PostgreSQL functions where appropriate.

3. Indexing Strategy

Indexing decision flow

We added indexes based on actual access patterns rather than indexing fields indiscriminately. Foreign keys, join columns, filtering fields, sorting fields, and pagination patterns were the primary candidates.

The goal was predictable query performance. Every index also carries storage and write-maintenance costs.

4. Schema Design and Constraints

Use clear and consistent naming conventions.
Define foreign-key relationships explicitly.
Use NOT NULL, UNIQUE, and CHECK constraints where the data model requires them.
Keep relationships normalized where appropriate.
Review RBAC policies for both correctness and query efficiency.

Database constraints document the data model and enforce important data rules at the source.

PostgreSQL Functions and Triggers: Put the Right Work in the Database

PostgreSQL Functions

When is a PostgreSQL function useful

Functions are useful for complex database-oriented fetching, reusable calculations, aggregations, and returning data in a form that reduces application-layer processing.

Database Triggers

Trigger use case

Triggers are appropriate for deterministic database responsibilities such as maintaining counters, derived values, timestamps, audit records, or data-integrity behavior.

Use triggers for deterministic database responsibilities—not as a hidden replacement for the entire application layer.

Database Events vs. External Workflows

Keep external work out of the critical transaction

Email delivery, push notifications, and other external calls should be handled asynchronously when the user-facing database transaction does not need to wait for the external system.

Views and Materialized Views

Normal Views

Reusable read model

Normal views are useful when you want to encapsulate reusable joins and filtering logic while still reading the current underlying data.

Materialized Views

Precomputed analytics path\

Materialized views are useful when expensive aggregations do not need to be recalculated for every request.

For appropriate workloads, REFRESH MATERIALIZED VIEW CONCURRENTLY can reduce disruption during refreshes. A suitable unique index is required for concurrent refreshes.

Safe refresh strategy

Optimizing Analytics APIs

Avoid repeated runtime calculations

Analytics endpoints often perform expensive joins and aggregations over transactional data. We reduced this work by pre-aggregating data, using materialized views, caching suitable results, and minimizing on-the-fly calculations.

If an expensive value does not need second-by-second freshness, don’t calculate it from scratch on every request.

API Optimization Principles

Pagination and Filtering

Efficient list API

Pagination and filtering should happen as close to the database as possible. Cursor-based pagination can be useful for large or frequently changing datasets.

Reduce Over-Fetching

Minimal response path

Avoid SELECT * and fields that the client does not need. Smaller payloads reduce database work, serialization, network transfer, and client-side processing.

Computed Fields

Avoid repeated runtime computation

Expensive computed fields can introduce latency when evaluated for every record. Where appropriate, use precomputed values, summary tables, materialized views, or caching.

GraphQL: Powerful, but Give It Guardrails

Avoid unnecessarily deep nested queries.
Control expensive computed fields.
Watch for N+1 patterns.
Limit excessive Hasura Actions.
Avoid using GraphQL as an internal service-to-service communication protocol when a more direct mechanism is appropriate.
Return only the data required by the client.

Healthy boundary

Service responsibility

Redis: Cache What Doesn’t Need to Be Recomputed

Read-through cache

Redis is useful for read-heavy endpoints, configuration data, analytics results, frequently requested reference data, and suitable computed counters.

Every cache should have a clear TTL, invalidation, refresh, and consistency strategy.

Effective caching targets data that benefits from reuse and has a defined consistency strategy.

Batch Processing: Email and Notifications

Synchronous anti-pattern

Asynchronous pattern

Background processing

This keeps external service latency out of the user-facing request and allows retries, batching, and controlled failure handling.

Data Lifecycle Management

Scheduled cleanup

Expired sessions, temporary records, old notifications, logs, stale tokens, and other disposable data should have an explicit lifecycle.

Large cleanup operations should be batched and controlled so that maintenance itself does not become a resource spike.

Observability: Measure Before You Optimize

Performance investigation loop

Query logging to identify slow SQL statements
pg_stat_activity to inspect active database sessions
EXPLAIN ANALYZE to inspect query execution plans
API timing to separate application, database, external-service, and network time
GraphQL resolver timing to identify expensive fields

Without observability, you can spend weeks optimizing the wrong layer.

Reducing Backend Downtime

Reliability loop

Performance and availability are connected. Health checks, monitoring, structured logs, graceful degradation, controlled retries, and automated recovery help prevent resource pressure from becoming prolonged downtime.

Retries should recover from failures without amplifying them.

Do We Really Need That Microservice?

Microservice decision

Every service adds container memory, network communication, database connections, deployment complexity, monitoring, and operational overhead.

If multiple services are tightly coupled, communicate constantly, share the same database, and always scale together, consolidation can produce a simpler and more efficient architecture.

The Anti-Patterns We Discovered

Hasura → Node.js → Hasura circular API calls
Using GraphQL as a universal internal communication layer
Over-splitting logic across microservices
Distributing one business flow across too many layers
Treating PostgreSQL as pure storage
Using infrastructure capacity to hide inefficient request paths

Each decision looked manageable in isolation. Together, they created system-wide overhead.

When 2 GB Became an Advantage

What the smaller environment exposed

The 2 GB environment didn’t create all of these problems. It exposed them. The constrained environment forced us to question every connection, query, network hop, container, and runtime calculation.

The 2 GB environment didn’t create all of these problems. It exposed them.

What Changed After the Refactor?

Outcome

More controlled database connections
Lower API latency
Fewer unnecessary database calls
Reduced network hops
Less application-layer processing
More efficient analytics queries
Better separation of responsibilities
Reduced container overhead
More asynchronous background processing
Lower infrastructure requirements

The Bigger Lesson: Scale Down Before You Scale Up

When an application becomes slow, the instinct is to scale. But adding infrastructure to an inefficient architecture can increase its cost without addressing the underlying problem.

If every request makes unnecessary network calls, adding more servers does not eliminate those calls. If every API performs expensive queries, increasing RAM does not make those queries logically simpler. If every service opens unnecessary database connections, increasing database capacity does not fix the request flow.

The optimization mindset

In our case, the answer was to simplify the request path, remove circular Hasura–Node.js communication, reduce unnecessary Actions, move appropriate operations closer to PostgreSQL, optimize queries and indexes, address N+1 patterns, use pagination and database-level filtering, precompute analytics, cache read-heavy workloads, move external work to background processing, and reduce unnecessary services.

The 64 GB server made the system comfortable. The 2 GB server made it honest.

Before scaling infrastructure, reduce unnecessary work across the architecture, request path, and database. Then scale based on measured demand.

Build for Scale at the Architecture Level

Scaling infrastructure works best when the architecture is ready to support it. Clear service boundaries, optimized database access, efficient request paths, caching, and asynchronous processing can reduce resource pressure before additional capacity enters the equation. If your application is facing performance bottlenecks or rising infrastructure demands, explore how backend engineering services can help strengthen the system before the next stage of scale.

Building Local LLMs Using Dart FFI And llama.cpp: Beyond Wrapper Packages

GeekyAnts India Pvt Ltd — Fri, 13 Nov 2026 07:35:00 +0000

Build local LLMs in Flutter with Dart FFI and llama.cpp, and see how native bridges, GGUF models, memory management, and token streaming enable private, on-device AI.

Introduction

When I first explored running local LLMs inside Flutter, I had a lot of questions:

Why are people using C++ libraries for AI?
Why not just write everything in Dart?
What exactly is llama.cpp?
What are .so, .dll, and .dylib files?
Why does Flutter suddenly need native code? If you're coming from a Flutter-only background, all of this initially feels very unfamiliar. Most Flutter developers rarely touch:
native memory
C/C++
shared libraries
compilers
ABI architectures
or systems-level programming

But once you understand the architecture, things become surprisingly logical. This article documents my journey building a proof-of-concept meeting summarizer with local LLM inference.

A Quick Disclaimer

This article is not intended to convince anyone to replace existing Flutter local AI packages.

In fact, if your goal is simply to integrate local AI into an application, a well-maintained package may be the fastest way to ship.

My goal here was different.

I wanted to understand what happens beneath those abstractions, how Flutter communicates with native AI runtimes, and what it takes to build that bridge myself.

That curiosity ultimately led me down the Dart FFI and llama.cpp rabbit hole.

POC support scope: The working LLM path in this repository currently targets Android arm64-v8a. The platform descriptions below explain how FFI libraries generally work; the other native llama bridge integrations like iOS or desktop OS’s are not implemented in this POC.

The Problem Statement

Before diving into implementation details, let's start with a simple question.

Why go through all the complexity of Flutter FFI and llama.cpp when we could simply host a GGUF model on a Python FastAPI server and consume it like any normal API?

That's exactly what I wondered initially.

The answer is that local AI solves a completely different problem.

Server-based AI depends on internet connectivity, introduces latency, incurs infrastructure costs, and sends user data to external servers

Local AI via llama.cpp runs the model directly on the device itself, enabling:

Offline LLM inference after the model has been downloaded
Better privacy because prompts and model output do not need to leave the device
Lower latency (no network round trips)
No backend costs (computation happens on user's device)

For applications like meeting summarization, note taking, personal assistants, or enterprise environments with strict privacy requirements, these benefits can be extremely valuable.

That said, local AI isn't always the right choice.

If your application requires very large models, centralized company knowledge, or continuous model updates, a cloud-based architecture may still make more sense.

The important thing is understanding the trade-offs and choosing the right approach for your use case.

Why Are AI Libraries Usually Written in C++?

As Flutter developers, we're used to solving everything in Dart. So why do most AI runtimes seem to be written in C++, Rust, CUDA, or other low-level technologies?

The answer is performance.

Large Language Models spend most of their time performing:

Matrix multiplications
Tensor operations
Memory-intensive computations
CPU and GPU optimizations

Languages like Dart are excellent for application development, but they're not designed for squeezing every bit of performance from modern hardware.

C++ gives developers direct control over:

Memory management
CPU instructions
Threading
SIMD optimizations
GPU integrations

That's why projects such as TensorFlow, PyTorch, ONNX Runtime, and llama.cpp all rely heavily on native code under the hood. Even many Python AI frameworks are actually thin wrappers around C++ implementations.

What Is llama.cpp?

At a high level, llama.cpp is an inference engine for running Large Language Models locally. Think of it as an AI inference engine optimized for local hardware.

It takes:

Model weights (the trained parameters from the LLM, in GGUF format)
Tokens (numerical representations of text)
Prompts (user input)

And performs all the mathematical operations required to generate text.

Some reasons llama.cpp has become so popular include:

Runs directly on CPUs
No Python dependency
Optimized for mobile and embedded devices
Supports quantized models

Cross-platform support

Understanding GGUF Models

When working with llama.cpp, you'll frequently come across GGUF files.

GGUF (GGML Universal File) is a model file format designed to store model tensors and metadata for efficient loading by GGML-based inference engines.

GGUF can store a model’s weights in different levels of precision. Quantization is the process of making those weights smaller by reducing their precision. So, GGUF can store either the original high-precision weights or quantized versions like Q4.

For example, Q4 quantization can reduce a 1B model from roughly 2 GB to roughly 800 MB on disk while maintaining acceptable quality i.e., small enough to be practical on some phones.

Without formats like GGUF, running local LLMs on mobile devices would be far less practical.

One thing I learned quickly is that choosing the right model is often more important than choosing the right framework.

A smaller, well-quantized model that runs smoothly on target devices is usually more valuable than a larger model that struggles with memory and performance constraints.

Existing Flutter Packages for Local AI

Before building my own integration, I explored some of the existing Flutter packages available for running local models. Some examples include:

flutter_gemma
fllama
llama_cpp_dart

These alternatives can be a great starting point if your goal is simply to get local AI working quickly. So why didn't I just use one of them?

For this project, my goal wasn't only to run a model locally. I wanted to understand:

How models are loaded
How memory is managed
How Dart communicates with native code
How token streaming works
How worker isolates fit into the picture
What actually happens inside the FFI layer

Building the bridge myself gave me a much deeper understanding of the architecture and helped me appreciate the amount of work these packages abstract away.

If your objective is shipping a feature quickly, evaluating existing packages first is absolutely a sensible approach.

If your objective is learning how local AI actually works inside Flutter, building a small FFI bridge yourself is an incredibly valuable exercise.

Flutter + llama.cpp Architecture

Each layer has a specific responsibility.

Flutter - Handles UI, state management, and user interactions.
Provider / Services - Contains application logic and coordinates model operations.
Worker Isolate - Keeps heavy inference work away from the UI thread.
Dart FFI - Acts as the bridge between Dart and native code.
Native Bridge - Exposes a stable C API that Flutter can call.
Llama.cpp - Loads models and performs inference.
GGUF Model - Contains the trained weights that generate responses.

Understanding Native Libraries

One thing that confused me initially was seeing files like:

libllama_bridge.so

llama_bride.dll

libllama_bridge.dylib

When we used to install games on Windows, many saw files like: game.dll, audio.dll, etc.

Those .dll files are compiled native libraries. They contain optimized code for:

Rendering
Audio processing
Physics simulation
Networking

The main game executable loads them and uses their functionality.

Platform-Specific Names

Flutter integrations using llama.cpp can follow the same pattern:

Instead of physics.dll, a bridge might be packaged as:

Android/Linux: libllama_bridge.so (Shared Object)
macOS/iOS: Symbols linked directly into the process or a .dylib where platform rules allows it
Windows: llama_bridge.dll

The native library does the heavy work, and Flutter communicates with it via FFI.

Please note that in our POC we have only covered Android arm64 build configs.

How Flutter Talks to C++: Dart FFI

This is where Dart FFI comes in.

FFI stands for: Foreign Function Interface

Which means: "Allow Dart to call functions written in another language." At the Dart-to-native boundary, FFI can call native C APIs without:

Flutter Platform channels
Platform-channel message serialization or
Message passing

[Official Dart FFI Documentation](https://dart.dev/interop/c-interop)

FFI vs Platform Channels: Why Not Platform Channels?

Flutter already provides Platform Channels, so why introduce FFI?

The answer comes down to performance.

AI workloads involve:

Frequent native calls
Large amounts of memory
Continuous token generation

FFI allows Dart to call native functions directly without serialization overhead. A simple FFI binding looks like this:

DynamicLibrary.open() only loads the library. lookupFunction() resolves an exported C symbol and converts it to a callable Dart function. Only then can Dart invoke the native bridge. The repository collects these typed lookups in _BridgeBindings.

Key Concepts That Finally Made Everything Click

Opaque Pointers

One concept that initially felt strange was opaque pointers.

Instead of exposing the actual C++ model object to Dart, native code simply returns a handle.

Dart doesn't know what's inside the session.

It simply passes the handle back whenever it wants to load a model, generate tokens, or destroy the session.

This keeps the object layout and memory management on the native side. It also creates an explicit ownership contract: every successful bridgeSessionCreate() must eventually be paired with bridgeSessionDestroy(), and pointers returned by allocating native functions must be released with the matching native free function.

Streaming Callbacks

One of the coolest parts of the implementation was token streaming.

Instead of waiting for the entire response, llama.cpp generates tokens one by one.

Every generated token flows back into Flutter through this callback. Pointer.fromFunction() converts a Dart callback into something the native C code can call. The userData pointer tells the callback which token stream to send the generated tokens to, so the tokens can appear one by one in the typing effect.

One important limitation is that the callback must run on the same Dart isolate thread. In this POC, that works because generation and the callback happen on the same worker thread. If the native library calls the callback from a different thread, we should use NativeCallable.listener instead.

Worker Isolates

AI inference is computationally expensive.

Running inference on Flutter's main Isolate would quickly make the UI feel sluggish or frozen. To avoid that, I moved all native interactions into a dedicated Worker Isolate.

The UI remains responsive while the model generates tokens in the background.

The Isolate lets the main Dart thread and the worker communicate by sending messages. The main isolate sends a command to the worker, the worker runs the blocking native/FFI work, and then sends back token, completion, or error messages. FFI talks directly to the native code, avoiding platform-channel overhead, while the Isolate keeps the heavy work away from the UI thread.

The POC I Built - Meeting Summarizer

Architecture

The proof of concept I built follows a fairly simple flow.

The user records a conversation.

The app has two transcription options: recorded audio uses local Whisper, while live speech uses the device’s speech recognition. It prefers offline recognition but falls back to online recognition if offline isn’t supported.

That transcript is then passed to the local LLM through the FFI bridge.

As the model generates a summary, tokens are streamed back to Flutter and displayed in real time.

The LLM inference process happens locally on the device.

No server required for LLMs.

Preview of what I build

Challenges I Faced

While the final architecture looks clean, getting there wasn't always smooth.

Model Downloading and Distribution

One thing that surprised me initially was that llama.cpp doesn't ship models - you download GGUF files, store them, and verify checksums. Model management quickly becomes a product concern rather than just an engineering concern.

This was probably one of the first moments where I realized local AI introduces a completely different set of challenges compared to calling a hosted API.

Memory Consumption

The Llama 3.2 1B Q4_K_M model file used in this POC is roughly 800 MB. Total runtime memory is higher and varies with context size, KV cache, runtime buffers, backend allocations, and device. Bigger isn’t better on a phone. Choosing a model becomes a balancing act between:

Quality
Speed
RAM usage
Storage size

FFI Debugging Can Be Painful

A normal Dart error usually gives a helpful stack trace.

A native crash often gives something like:

Segmentation Fault

And that's it.

Most of my debugging time was spent tracking:

Invalid pointers
Memory ownership issues
Double frees
Native lifecycle bugs

FFI is powerful, but it also removes some of the safety nets Flutter developers are used to.

Device Variability

Fast on one phone, slow on another. Testing across multiple devices becomes extremely important when working with local AI.

Local AI Is Not Always Better

This was perhaps the biggest lesson.

It's easy to get excited about running models completely offline.

But local AI isn't a universal replacement for cloud AI.

Cloud-based solutions still have advantages:

Access to larger models
Easier updates
Centralized knowledge
Less device storage usage

The best solution depends entirely on the problem you're trying to solve.

When Would I Actually Use This?

One question I kept asking myself throughout the project was: "This is cool, but would I use it in a real application?"

I think the answer depends on the use case.

Great Candidates

Meeting summarization
Personal journals
Offline assistants
Enterprise privacy-focused applications
Travel apps with limited connectivity

Probably Better on the Cloud

Large-scale document analysis

Complex RAG systems

Company-wide knowledge assistants
Very large models requiring significant compute

Local AI is powerful, but it isn't a universal replacement for cloud-hosted AI.

Project Repository

The complete source code for this proof of concept is available on GitHub:

GitHub Repository:

GitHub Link to Project

The repository contains:

Flutter application code
Dart FFI bindings
Native C/C++ bridge implementation
Worker isolate architecture
Token streaming implementation
Model loading lifecycle management
Meeting summarization proof of concept

If you're interested in exploring the implementation details beyond what is covered in this article, feel free to check out the repository.

Key Learnings From This Project

Memory Management Matters

Dart uses garbage collection; native C/C++ memory must be released manually to avoid leaks and crashes.

Isolates Are Essential

Inference can take several seconds depending on the model and device. Running everything on the UI isolate is not a realistic option.

Streaming Creates Better UX

Showing tokens as they're generated makes the application feel much faster and more responsive.

Native Boundaries Should Be Simple

Keeping a small, stable API between Dart and C++ significantly reduces debugging complexity.

Don't Fight Existing Ecosystems

Building an inference engine from scratch would be an enormous undertaking.

Using llama.cpp allowed me to focus on application development rather than reinventing years of optimization work.

Conclusion

Before starting this project, Flutter FFI and local AI felt intimidating.

There were many unfamiliar concepts, from native libraries and memory management to callbacks and worker isolates.

But once I understood the architecture, it became clear that each layer has a very specific purpose.

The stack I ended up with looks like this:

Flutter for UI
Dart for application logic
FFI for interoperability
C++ for performance-critical operations
llama.cpp for inference
GGUF models for local execution

The result is a working POC on android arm64: speech -> transcript -> streamed local summary, with LLM inference that runs directly on the device without depending on cloud infrastructure.

The learning curve is definitely steeper than consuming a REST API, but the capabilities you unlock - private, offline, low-latency AI running entirely on a user's device - make the journey worthwhile.

Building this POC showed how much Flutter can do when it reaches beyond Dart into native code.

Resources

GitHub Link to Project

Dart FFI Documentation

llama.cpp Repository

GGUF Model Format

Flutter Native Integration

Provider Package

Automation Testing With Playwright Using JavaScript

GeekyAnts India Pvt Ltd — Fri, 06 Nov 2026 05:10:44 +0000

Automation testing is a critical part of software development that helps ensure the quality and reliability of your application. Playwright is a robust end-to-end testing framework that enables you to automate browser actions and interactions. This comprehensive guide will walk you through performing automation testing using Playwright in a Next.js application.

Playwright is a JavaScript library that provides a high-level API to automate browsers like Chromium, Firefox, and WebKit. It allows you to write browser automation scripts that simulate user interactions, navigation, form submissions, and more. Playwright's unique architecture provides a fast and reliable way to test web applications.

Setting Up a Javascript Project

Before we start writing tests, let's set up a Next.js project:

Create a new Javascript project:

mkdir my-js-project
cd my-js-project
npm init -y

This will create a new directory called my-js-project, navigate into it, and initialize a new Node.js project with default settings.

The -y flag in the npm init -y command is a shorthand for "yes". It automatically answers "yes" to all the prompts that npm init would normally ask you, thereby accepting all the default values for the project setup.

Install the Playwright dependencies:

npm install playwright

Writing Automation Tests with Playwright

Let's create a simple automation test for a javascript application. We want to test the navigation from the home page to a blog post page.

const { chromium } = require('playwright');

describe('Next.js Blog Automation Test', () => {
  let browser;
  let page;

  beforeAll(async () => {
    browser = await chromium.launch();
  });

  beforeEach(async () => {
    page = await browser.newPage();
  });

  afterAll(async () => {
    await browser.close();
  });

  it('should navigate to a blog post', async () => {
    await page.goto('http://localhost:3000');
    await page.click('a[href="/blog"]');
    await page.waitForSelector('h1');
    const pageTitle = await page.textContent('h1');
    expect(pageTitle).toContain('Blog Post Title');
  });
});

Let us see the components of the above code in a bit more detail.

When using testing frameworks like Jest, Mocha, Jasmine, or Playwright, the describe block is a fundamental organizational structure that helps group related test cases together. It provides a way to define a logical container for a set of tests with a common context or functionality. The describe block is primarily used to improve your test suite's organization, readability, and maintainability.

The beforeEach function is a part of testing frameworks like Jest and is used to set up a consistent and controlled environment before running each individual test case within a describe block. This is particularly helpful in maintaining a clean state and avoiding interference between test cases.

In testing, it's important to create a controlled environment for each test case to ensure that they run independently and reliably. Sometimes, tests require the same setup steps before execution. Instead of repeating the setup code in every test case, you can use beforeEach to define those common setup actions.

In automated testing, especially when using testing frameworks like Jest, you often encounter lifecycle functions that help you set up and tear down certain conditions before and after tests are run. One important lifecycle function is beforeAll.

The beforeAll function executes a block of code before all the tests within a test suite (also known as a test file or describe block) are run. This block of code typically sets up any common resources or configurations required for all the tests in that suite.

The afterAll function is a part of testing frameworks like Jest, and it is used to perform cleanup tasks or teardown operations after all the test cases within a describe block have been executed. It's the counterpart of the beforeAll function, which is used for setup operations before the test cases.

In testing, there might be scenarios where you need to perform cleanup tasks or reset the environment after all the test cases within a particular test suite have run. afterAll allows you to define these cleanup actions.

Enhancing API Testing with Playwright

Did you know that Playwright, commonly recognized for its powerful browser automation capabilities, can also be utilized for API testing? This versatile tool excels in web application testing and offers robust features to automate and validate API interactions. By leveraging Playwright for API testing, you can streamline your testing processes and ensure your APIs function as expected.

Writing Your First API Test

Let's start with a basic example. We'll test a simple REST API that returns user information.

Start with creating a new file called apiTest.spec.js.
Import Playwright:

const { test, expect } = require('@playwright/test');

Write a Basic Test:

Here's a test that fetches user data from a hypothetical API endpoint.

test('GET user data', async ({ request }) => {
  const response = await request.get('https://jsonplaceholder.typicode.com/users/1');
  expect(response.status()).toBe(200);
  const userData = await response.json();
  expect(userData.id).toBe(1);
  expect(userData.name).toBe('Leanne Graham');
});

Breaking Down the Test

test: Defines a test case.
request.get: Sends a GET request to the specified URL.
expect: assertions to check if the response meets our expectations.
response.status(): Checks the HTTP status code of the response.
response.json(): Parses the response body as JSON.

Handling Different HTTP Methods

Playwright supports various HTTP methods. Here are examples of POST, PUT, and DELETE requests.

POST Request:

test('POST create new user', async ({ request }) => {
  const response = await request.post('https://jsonplaceholder.typicode.com/users', {
    data: {
      name: 'John Doe',
      username: 'johndoe',
      email: 'johndoe@example.com'
    }
  });
  expect(response.status()).toBe(201);
  const newUser = await response.json();
  expect(newUser.name).toBe('John Doe');
});

PUT Request:

test('PUT update user data', async ({ request }) => {
  const response = await request.put('https://jsonplaceholder.typicode.com/users/1', {
    data: {
      name: 'Jane Doe'
    }
  });
  expect(response.status()).toBe(200);
  const updatedUser = await response.json();
  expect(updatedUser.name).toBe('Jane Doe');

});

DELETE Request:

test('DELETE user', async ({ request }) => {
  const response = await request.delete('https://jsonplaceholder.typicode.com/users/1');
  expect(response.status()).toBe(200);
});

Handling Authentication

Many APIs require authentication. Here’s how to handle it in Playwright:

test('GET user data with authentication', async ({ request }) => {
  const response = await request.get('https://api.example.com/protected-resource', {
    headers: {
      'Authorization': `Bearer YOUR_ACCESS_TOKEN`
    }
  });
  expect(response.status()).toBe(200);
});

Using Fixtures for Setup and Teardown

Fixtures help set up and tear down resources required for your tests. Here's an example:

test.describe('API tests with setup and teardown', () => {
  let userId;

  test.beforeAll(async ({ request }) => {
    const response = await request.post('https://jsonplaceholder.typicode.com/users', {
      data: { name: 'Test User' }
    });
    const user = await response.json();
    userId = user.id;
  });

  test.afterAll(async ({ request }) => {
    await request.delete(`https://jsonplaceholder.typicode.com/users/${userId}`);
  });

  test('GET created user data', async ({ request }) => {
    const response = await request.get(`https://jsonplaceholder.typicode.com/users/${userId}`);
    expect(response.status()).toBe(200);
  });
});

Running Your Tests

You can run your Playwright tests using the Playwright Test Runner. Add a script to your package.json:

"scripts": {
  "test": "playwright test"
}

Then, run your tests with:

npm test

Generating Test Depictions

Playwright Test generates reports by default. You can view the test results in the console, but you might want a more detailed report. Playwright provides a built-in HTML report. To enable it, add a playwright.config.js file in your project root:

module.exports = {
  reporter: [
    ['list'],
    ['html', { outputFolder: 'playwright-report', open: 'never' }]
  ],
};

Run your tests again.

After completing the tests, you'll find the HTML report in the playwright-report directory. To open it, run:

npm playwright show-report

This command opens the HTML report in your default browser, providing a detailed view of your test results.

Best Practices for Automation Testing

Here are some best practices to follow when writing automation tests using Playwright in a Next.js application:

Isolation: Each test should be independent and not depend on the state of other tests.
Use Data Attributes: Use data attributes in your components to make it easier to select elements in your tests.
Mock API Calls: Use mocking or stubbing to avoid making actual API requests during tests.
Assertions: Use meaningful assertions to verify the expected behavior of your application.
Page Objects: Create page objects or helper functions to encapsulate the interactions with specific parts of your application.

Continuous Integration with Playwright

Integrate your Playwright tests into your continuous integration (CI) pipeline. This ensures that your tests are automatically executed whenever you push code changes.

Choose a CI platform like GitHub Actions, Jenkins, or Travis CI.
Configure the CI pipeline to run the Playwright tests using Jest.

Playwright vs. Selenium: A Comparison

Feature	Playwright	Selenium
Language Support	Javascript/TypeScript, Python, C#, Java	Java, C#, Python, Ruby, JavaScript, Kotlin
Browser Support	Chromium (Chrome, Edge), WebKit (Safari), Firefox	Chrome, Firefox, Safari, Edge, IE, Opera
Installation and Setup	Single package installation (includes browser binaries)	Separate installations for WebDriver binaries
Speed and Performance	Generally faster (uses DevTools Protocol)	Slower (uses WebDriver protocol)
Parallel Testing	Built-in support for parallel testing	Requires additional configuration/tools (e.g., Selenium Grid)
APIs and Features	Modern API, auto-wait, network interception, mobile emulation	Mature API, similar features but more configuration
Community and Support	Growing community, backed by Microsoft	Established community, extensive documentation
Cross-Browser Testing	Consistent results with the same API for all browsers	It is possible but may have inconsistencies
Test Reporting	Integrated reporting tools	Requires additional tools/libraries (e.g., TestNG, JUnit)
Script Debugging	Built-in tools for debugging (e.g., Playwright Inspector)	Debugging through language-specific IDE/tools

Conclusion

Automation testing using Playwright in a Next.js application can help you catch bugs and ensure the quality of your software. Following the steps in this guide, you've learned how to set up a Next.js project, write automation tests, and integrate them into your CI pipeline. Remember to continuously update and expand your test suite as your application evolves.

Automate the Obvious, Review the Rest: Designing AI-Powered Content Moderation

GeekyAnts India Pvt Ltd — Thu, 29 Oct 2026 03:33:21 +0000

Introduction

A social product does not fail only when the database is slow. It also fails when the feed becomes a place users no longer trust.

We needed a moderation system that could inspect captions, comments, ads, stories, images, and videos as soon as they were created without delaying uploads or posts for human review or automating every borderline case.

The design we shipped is deliberately split. Text goes through NestJS, Hasura event triggers, BullMQ, and the OpenAI Moderation API. Images and videos go through S3, SNS, SQS, AWS Lambda, and Amazon Rekognition. Both pipelines write the same status model, store the AI evidence, and leave gray-area content for the admin panel.

Moderation is not a feature bolted onto the feed. It is a gate the feed has to pass through.

Why Traditional Moderation Doesn't Scale

As content volume grows, moderation becomes less about identifying whether a single piece of content is safe and more about deciding where each piece should go.

Sending every piece of content to a human does not scale, while automatically approving or rejecting everything based on a single model score introduces its own risks.

The system therefore treats moderation as a triage pipeline: clear cases are automated, ambiguous cases are escalated, and the feed only exposes content that has reached an approved state.

Why Content Moderation Is Required

User-generated content is the product. It is also a risk. Without a consistent review path, unsafe or policy-breaking material can reach other users in seconds.

Protect people on the platform from explicit, violent, hateful, or otherwise harmful material.
Keep posts, ads, and stories out of public feeds until they are safe to show.
Meet App Store, Play Store, and trust-and-safety expectations for UGC products.
Create an audit trail: what the model saw, what score it produced, and who made the final call.
Notify the owner when content is rejected, instead of failing silently.

Manual review of every caption and every frame does not scale.A fully automated approve-or-reject model also creates risk: models are confident on extremes and uncertain in the middle. The system is built around that fact.

What the System Does

Every piece of moderated content starts as pending. Background work then produces a decision: approved, under_review, or rejected.

Approved content can appear in feeds and listings. Rejected content stays out of the public product. Under-review content stays out of the public product until a moderator confirms or overrides the AI decision. That is the contract.

The Architecture: Two Pipelines, One Status Model

We did not force text and media through the same worker. Text is small, frequent, and already in PostgreSQL at the moment it is written. Media is large, arrives in S3, and is expensive to inspect. Each path uses the infrastructure that fits it.

Shared data model content_moderation_status on the source row: pending, approved, under_review, rejected. A content row as the generic moderation record. moderation_results for provider, risk_score, labels, ai_decision, and final_decision. reviewed_by on the result row when a human takes the last step. Feeds that list posts, ads, stories, or media filter on content_moderation_status = approved. The existing status columns (active, trash, uploaded, and so on) still mean what they always meant. Moderation is a separate lifecycle on purpose.

Do not overload “active” to mean “safe.” Safety needs its own column.

Text Moderation: Hasura, BullMQ, and OpenAI

When a user writes text, the row is saved first. The HTTP request does not wait for OpenAI. Hasura fires an event; NestJS creates the content record and enqueues a job; a worker calls the model.

Text request path

Insert / update row
↓
Hasura event trigger
↓
POST /v1/event/content-moderation
↓
Create content row
↓
BullMQ job MODERATE_TEXT
↓
OpenAI /v1/moderations
↓
Save result + update status

The worker

The content-moderation processor runs with limited concurrency and a rate limit so OpenAI traffic stays controlled. Empty text is approved without a call.

Failed jobs go to a dead-letter queue. Hasura retries the webhook (three attempts, ten-second interval, sixty-second timeout) if Nest is briefly unavailable.

How Text Scores Are Calculated

OpenAI returns a flag plus a category_scores map, each value between 0 and 1. We do not treat the boolean flag as the decision. We convert the highest category score into a 0–100 risk score and classify from that number.

Text scoring

OpenAI category_scores
→
Take the maximum score
→
riskScore = round(max × 100)
→
Compare to thresholds
→
approved / under_review / rejected

Risk below 40 → approved. The content can enter the feed.
Risk from 40 up to (but not including) 75 → under_review. A human must decide.
Risk 75 or above → rejected. The content stays hidden and the owner is notified.

Only categories that meet or exceed the approved threshold are stored as labels, as percentages. The admin later sees the violated categories, not a wall of near-zero scores. Thresholds are configurable through MODERATION_APPROVED_THRESHOLD and MODERATION_REJECTED_THRESHOLD; the production defaults are 40 and 75.

How We Calibrated the Thresholds

Scores of 40, 75, or 90 only become production policy after you measure what those cutoffs do to false positives, false negatives, and human workload. The value of these thresholds comes from the calibration loop that produced and maintains them.

Starting points, not final answers

We began with provider semantics and product risk posture, then validated against labelled samples rather than shipping intuition.

Text (OpenAI): start near the middle of the score distribution. Auto-approve only when the max category score is clearly low; auto-reject only when it is clearly high; leave a wide under_review band for the first weeks.
Images/video (Rekognition): start stricter on auto-approve than text (review at 75%, reject at 90%), because a false approval in a feed is costlier than an extra item in the admin queue.
Environment variables keep cutoffs changeable without rewriting the decision engine.

Labelled test set and threshold sweeps

Before locking production defaults, we built a small labelled set covering three buckets: clearly safe, borderline / ambiguous, and clearly violating. For each item we stored the model’s raw scores, the would-be decision at candidate cutoffs, and a human gold label.

Calibration experiment

Collect labelled sample (safe / borderline / violate)
↓
Run OpenAI or Rekognition scores
↓
Sweep candidate thresholds
↓
Measure auto-approve FP, auto-reject FN, review volume
↓
Pick cutoffs that protect the feed without drowning admin
↓
Re-check against human final_decision in production

What “good” looked like for us

Near-zero false auto-approves on the labelled violate set (missed harm is the expensive error).
Bounded false auto-rejects: overturns in admin or appeals should stay rare; if many auto-rejects become approved, raise the reject line or widen review.
A review band that is large enough to catch ambiguity but small enough that queue age stays within SLA.
We treat the AI-human disagreement rate on under_review items as a first-class metric rather than noise.

Text settled at 40 / 75 after sweeps showed that scores below ~40 almost never carried human-confirmed violations in our sample, while scores at or above ~75 almost always did. The middle band produced the majority of human overrides, which is exactly where we want people in the loop.

Media settled at 75 / 90 based on the same calibration process but with a different cost curve: Rekognition confidences in the mid-70s were often ambiguous on a single frame, so we refuse to auto-approve there. Sustained or high-confidence hits (including the video frame policy) justify auto-reject without waiting for a moderator.

Thresholds are hypotheses. Production disagreement rates are the experiment.

We revisit cutoffs when operational signals move: rising appeals after auto-reject, reports against content that was auto-approved, or a review queue that ages past SLA. Calibration is continuous, not a one-time spreadsheet.

Image and Video Moderation: S3, Lambda, SQS, Rekognition

The media never waits in the Nest process for Rekognition. The user uploads to S3 under users/uploads/. An ObjectCreated notification is published to SNS, and the media path continues asynchronously.

Why not two Lambdas on one SQS queue

SQS is a competing-consumer queue: each message is delivered to exactly one consumer. If two Lambda event-source mappings share a single queue and each “filters” by message shape, a message can land on the wrong function. That function may skip and still succeed—deleting the message—or fail and retry forever. Either way, delivery semantics are not safe for mixed payload types.

We therefore do not rely on two independently filtering consumers on one queue. The media pipeline is a two-stage design: an ingest queue owned by the scorer Lambda, and a results queue owned by the worker Lambda. SNS fans events into the correct stage; it is not used as a substitute for competing consumers.

What the first Lambda does

Images (.jpg, .jpeg, .png, .bmp, .webp): DetectModerationLabels and RecognizeCelebrities, then evaluate thresholds and send a result message to SQS.
Videos (.mp4, .mov): StartContentModeration with an SNS notification channel. Rekognition works asynchronously; the worker finishes the job later.
Files matching a thumbnail filename or path pattern are still labeled because ad thumbnails can be the image users see first.

What the worker Lambda does

Look up the media row by S3 key, retrying three times with a three-second delay so the upload API can finish inserting the row.
Inserts content and moderation_results.
Updates media.content_moderation_status.
Propagates the worst status to the parent post, ad, or story.
For videos, pages through GetContentModeration until every label is collected, then applies the frame policy.

SQS is the buffer. Lambda is the computer. Rekognition is the eyes. PostgreSQL is the record.

How Image Scores Are Calculated

Rekognition returns a list of moderation labels, each with a confidence from 0 to 100. We keep every label name mapped to a rounded confidence, then take the maximum confidence as the risk score.

Image scoring

DetectModerationLabels
↓
Build label → confidence map
↓
riskScore = round(max confidence)
↓
Apply 75 / 90 thresholds
↓
Optional celebrity override

Confidence below 75% → approved.
Confidence from 75% up to 90% → under_review.
Confidence 90% or above → rejected.

The defaults are MODERATION_REVIEW_THRESHOLD = 75 and MODERATION_REJECT_THRESHOLD = 90. They use a stricter review boundary than the text moderation thresholds because a false approval on a photo is more damaging in a feed than an unnecessary review.

The policy also calls out hard categories Rekognition can surface—explicit material, sexual activity, violence, hate symbols, drugs and tobacco, visually disturbing content. High confidence in those labels is what the 90% reject line is designed to catch without a human delay.

Video: Async Jobs and Frame Policy

A video cannot be scored from a single still. Rekognition runs a content-moderation job across the file, publishes completion to SNS, and that notification arrives on the same SQS queue the image path already uses.

Video path

S3 video object
↓
StartContentModeration
↓
Rekognition job
↓
SNS success notification
↓
Worker fetches all labels (paginated)
↓
Frame policy + thresholds
↓
Write status

Each label is tied to a timestamp. We still compute riskScore as the maximum confidence, but the decision also depends on how often and how long the violation appears.

Frame policy

A frame is violating if its label confidence is at least the review threshold (75%).
Five or more violating frames, or a cluster lasting three seconds or more, meets the frame policy.
If any label reaches the rejection threshold (90%+), the video is rejected.
If labels are only in the review band and the frame policy is met, the video is also rejected—repeated mid-confidence hits are treated as a pattern, not a coincidence.
If labels are only in the review band and the frame policy is not met, the video goes to under_review.
No qualifying labels → approved.

One risky frame is a question. Five frames or three seconds is an answer.

Parent Aggregation and Feed Visibility

A post, ad, or story can have several media items plus text. The parent must not appear “approved” while one of its images is still pending or rejected.

Worst-status rollup

→ Media A approved
→ Media B under_review
→ Caption approved
→ Parent = under_review

Statuses are ordered pending < approved < under_review < rejected. The database function apply_aggregate_moderation_status applies that order to user_post, hey_ads, and users_story. If any child is rejected, the parent is rejected.

List APIs for the product surface follow one rule: show content only when content_moderation_status is approved. Under-review and rejected items are not a client-side filter. They never qualify for the public listing query.

What Happens When Content Is Rejected

When content_moderation_status becomes rejected, a Hasura event on the content table calls POST /v1/event/content-rejected-notify. That queues an email through SendGrid and an in-app/push notification through Firebase, including the content type and rejection reason. The user is told the content did not pass review. They are not left guessing why it vanished.

Rejection side effects

Status → rejected
↓
Hasura content_rejected_email
↓
Nest notify webhook
↓
Email + push
↓
Still hidden from feed

The Last Step: Admin Panel Review

Automation is the first reviewer, not the last. Anything in under_review—text, image, or video—is a case a person has to close. That is the admin panel’s job.

Human review path

What the moderator needs on screen

A queue, not a scavenger hunt: one list of under_review (and optionally pending) items across posts, comments, ads, stories, and media.
The source itself: caption text, image, or playable video.
The machine evidence: provider (Openai or Rekognition), risk score, labels, celebrity names when present, video frame summary when present.
Actions: approve, reject. Optionally escalate, assign, or add an internal note.
Filters by content type, date, and score so a small team can work the riskiest items first.

On approval, both content_moderation_status and final_decision are set to approve. On rejection, the status becomes rejected, the existing rejection notification path fires, and the item still never reaches the feed. The AI decision remains on the row as ai_decision so we can measure how often humans agree with the model.

under_review is not a soft approve. It is a closed door until a person opens it.

The Production Feedback Loop

Shipping thresholds and queues is not the end of moderation work. The system has to be operated. We treat the following signals as the minimum set for running AI moderation in production—and as the inputs that decide when to retune cutoffs.

Operational signals

Moderation latency: time from enqueue (Hasura event or S3 ObjectCreated) to written decision, tracked as p50/p95 separately for OpenAI text and Rekognition image/video.
Review-queue age and SLA: count and age of under_review items; alert when the oldest item or median wait exceeds the team’s SLA.
Provider failures: OpenAI and Rekognition error rates, throttles, empty results, and dead-letter / retry volume on BullMQ and Lambda.
AI–human disagreement rate: share of rows where ai_decision ≠ final_decision after admin review, broken down by provider and content type.
False positives and appeals: auto-reject later overturned to approve; user appeals or support tickets after rejection notifications.
False negatives and reports: user reports or admin discoveries on content that was auto-approved—these are the most expensive misses.
Threshold drift: weekly or biweekly review of the above rates;if disagreement rates, appeals, or queue age move outside defined bands if disagreement, appeals, or queue age move outside bands, re-run the labelled sweep and adjust env thresholds.

Operate → measure → retune

Ship with calibrated defaults
↓
Watch latency, queue age, provider health
↓
Compare ai_decision to final_decision
↓
Track appeals and reports
↓
Detect drift
↓
Re-sweep thresholds or widen review

Storing ai_decision alongside final_decision and reviewed_by is what makes this loop possible. Without that pair, you can run a model, but you cannot prove whether the model and the product still agree.

A moderation system without operational signals is a deployment, not a practice.

The Deployment Pipeline

The media Lambdas are not updated by hand in the console as a normal workflow. Each function has a GitHub Actions workflow on the develop branch, scoped to its own directory.

Lambda CI/CD

Push to develop
↓
OIDC to AWS
↓
npm ci --omit=dev
↓
Zip the function
↓
Create or update Lambda
↓
Wait + verify

deploy-image-moderation.yml watches lambda/image-moderation/**.
deploy-image-moderation-worker.yml watches lambda/image-moderation-worker/**.
Runtime is Node.js 22. The scorer Lambda is sized for a short Rekognition call (about 60 seconds, 256 MB). The worker is sized for video pagination and database work (about 5 minutes, 512 MB).
Secrets stay in GitHub Secrets: queue URL, database connection, and deployment role. Thresholds and regions live in deploy config.

Text moderation ships with the NestJS service: the Hasura trigger metadata, the webhook, the BullMQ processor, and the OpenAI client. Changing a threshold is a configuration change, not a rewrite of the decision engine.

Edge Cases the Pipeline Had to Absorb

Race with inserts: the worker retries media lookup because S3 can notify Lambda before Nest commits the media row.
Ad thumbnails: if the object is a thumbnail rather than a media row, the worker can still attach the result to hey_ads.
Unsupported types: unknown extensions are skipped rather than auto-approved by accident.
Idempotency: if media is already moderated, a later result may only escalate to a worse status, never quietly downgrade it.
Aggregation after the fact: parent status is recomputed from children so mixed media cannot hide a reject.

What This Changed in the Product

Outcome

Upload / post returns immediately
↓
Background AI inspects content
↓
Clear cases resolve themselves
↓
Gray cases wait for admin
↓
Feed only shows approved
↓
Rejected owners are notified

Text, image, and video share one status vocabulary.
OpenAI and Rekognition are isolated behind queues, so model latency is not user latency.
Scores are explicit numbers with published thresholds, not hidden boolean flags.
The admin panel is the last reviewer for under_review, not an optional extra screen.
Evidence is stored: labels, scores, provider, and the human override.

The important change was not simply adding another moderation API. It was moving moderation from an isolated check into a product lifecycle. Content can return from the upload or post request without waiting for model inference, while remaining outside public listings until the moderation lifecycle resolves. Clear cases move automatically, uncertain cases reach the admin queue, and the final decision is recorded alongside the machine evidence.

This also creates a feedback loop: the AI decision remains stored alongside the human final decision, allowing the team to measure where automated decisions and human decisions agree or differ.

The Bigger Lesson: Automate the Extremes, Staff the Middle

A moderation system that only auto-rejects the obvious still leaks. A system that auto-approves anything the model did not scream about still leaks. The useful design is a three-way split.

The moderation mindset

Ingest without blocking the user
↓
Score with the right model for the medium
↓
Approve the clearly safe
↓
Reject the clearly unsafe
↓
Queue the rest for admin
↓
Keep the feed on approved only

We used OpenAI for language because it already classifies the categories we care about in text. We used Rekognition for pixels because it already understands moderation labels, video timestamps, and celebrity faces. We used Lambda and SQS because media events are bursty and should not live inside the API process. We used Hasura and BullMQ because text events are already database events. We used an admin queue because a score of 62 is not a moral fact. It is a request for a person.

The goal of AI moderation is not to remove humans from the loop. It is to make sure humans spend their time where human judgment matters most.

So before a caption, a photo, or a video is allowed to represent the product, it has to survive the pipeline: pending, scored, decided, and—if the model is unsure—reviewed. That is how content moderation works here: as a path every piece of user content has to follow before it hits the feed, rather than as a slogan in a policy document.

What’s Next? Engineer AI Moderation for Production

The bigger lesson goes beyond content moderation. Production AI features need more than an accurate model. They need clear decision boundaries, asynchronous processing, human review paths, auditability, monitoring, and a way to improve as real-world behavior changes.

The same principle applies whether AI is moderating content, evaluating documents, prioritizing cases, or assisting users inside a product. Automate decisions where confidence is high, route ambiguous cases to people, and retain enough evidence to understand why each decision was made.

That is what turns an AI capability into a system a product team can operate at scale.

At GeekyAnts, our AI Engineering teams design and build production-ready AI systems around real product workflows, combining model integration with scalable architecture, observability, deployment, and human-in-the-loop controls.

Building an AI-powered feature that needs to move from model API to production system? Talk to our AI engineering team.

Mobile App Deep Linking With React Native, NestJS and AWS

GeekyAnts India Pvt Ltd — Tue, 27 Oct 2026 05:11:40 +0000

Build a self-hosted deep linking and referral system for React Native without a third-party SDK. This guide covers Android App Links, iOS Universal Links, an S3 and CloudFront setup, and a NestJS backend that routes users to the right app store.

Third-party attribution services like Branch, AppsFlyer, and Firebase Dynamic Links offer turn-key deep linking out of the box. However, they frequently bring vendor lock-in, recurring SaaS subscription overhead, privacy and data residency compliance challenges, and sudden deprecations (such as the sunsetting of Firebase Dynamic Links).

Firebase Dynamic Links is the clearest example: Google shut the service down on August 25, 2025, and every page.link URL stopped working on that date.

This guide walks through building a fully self-hosted, vendor-agnostic deep linking and referral system using native OS domain verification, AWS infrastructure (S3, CloudFront, Application Load Balancer), a NestJS backend, and a React Native mobile client.

What Is Deep Linking?

Deep linking routes a user directly to a specific screen or action within a mobile app when clicking a standard HTTPS URL, bypassing the default home screen.

Tap a normal link on your phone and your browser opens the matching webpage. A deep link works the same way, except it opens a specific screen inside an app instead.

Example: an invite link

A friend sends you a link to join an app.

Normal link: takes you to a generic webpage or app store listing
Deep link: if the app is already installed, it skips the browser completely and drops you straight into the exact screen you need, like a sign-up form with your friend's invite code already filled in

Why this matters

Without deep linking, joining through a link looks like this:

Tap the link
Land on a generic webpage
Manually open the app
Find the right screen
Type in the code by hand

With deep linking, all five steps collapse into one tap.

The hard part: app not installed

If the app isn't on the phone yet, the link still needs to do something useful. Usually that means sending the person to the App Store or Play Store, then remembering their original intent (like that invite code) once they install and open the app for the first time.

This is the harder problem the rest of the guide solves, and it's usually why teams reach for paid third-party tools instead of building it themselves.

Types of Mobile Links

Mobile platforms support more than one kind of link. This guide uses verified HTTPS links because they are the only type that the operating system ties to a domain you own.

Link type	Example	How the OS trusts it	Limitation
Custom URI scheme	`myapp://join?code=ABC123`	No verification. Any app can register the same scheme	Fails with an error when the app is not installed
Android App Links	`https://invite.app.com/join?code=ABC123`	Android checks assetlinks.json on your domain through the Digital Asset Links API	Opens in the browser if verification fails
iOS Universal Links	`https://invite.app.com/join?code=ABC123`	iOS checks apple-app-site-association on your domain	Opens in Safari if the app is missing or the file cannot be fetched

App Links and Universal Links share one URL for both platforms. The same link also works as a normal web page when the app is not installed, which is what makes the store redirect in this guide possible.

Core Use Cases

The rest of this guide builds the referral onboarding case end to end. The same setup supports the other three cases once you add their paths to the manifests and the app router.

Use Case	Example URL	Expected Target Screen
Referral Onboarding	`https://invite.app.com/join?code=ABC123`	Pre-filled Registration Screen
Product Sharing	`https://store.app.com/products/8930`	Specific Product Details Page
Push Notifications	`https://app.com/notifications/102`	Direct Chat / Message Context
Marketing Campaigns	`https://app.com/promos/summer2026`	Discount / Landing Screen

Direct vs. Deferred Deep Linking

Understanding operating system boundaries is essential before designing the architecture.

Direct Deep Linking

The mobile application is already installed. The mobile operating system (Android or iOS) intercepts the HTTP request, matches the domain signature against its installed applications, and directly opens the app without touching the web browser or backend web servers.

The OS only intercepts links that a user taps. On iOS, a Universal Link typed or pasted into the Safari address bar opens the web page instead of the app. A tap on a link to the same domain inside Safari also stays in the browser. Share links from messages, email, notes or other apps to get the expected behavior.

Deferred Deep Linking

The application is not installed. The browser handles the URL, sending the request to your backend. The user is redirected to the Google Play Store or Apple App Store. Because native Android App Links and iOS Universal Links do not natively pass context across an installation boundary, the original URL parameters are lost upon install.

Scenario	Who handles the link	Does the referral code survive?
App installed	Android or iOS opens the app directly	Yes. The app reads it from the URL
App not installed, Android	Browser, then CloudFront, then NestJS, then Play Store	Yes, through the Play Install Referrer
App not installed, iOS	Browser, then CloudFront, then NestJS, then App Store	No. The user enters it in the app during a grace period

Architecture Distinction: Without third-party SDKs, this architecture provides store routing with server-side/manual attribution (via Play Store Install Referrer on Android and manual user/web account grace periods on iOS) rather than seamless, native OS-level deferred deep linking.

How the Deep Linking Architecture Works

One CloudFront distribution sits in front of two origins. Static verification files come from a private S3 bucket, and every dynamic request on the domain goes to the NestJS service behind the Application Load Balancer. This keeps the files that Android and iOS read separate from the redirect logic, on a single domain and a single HTTPS certificate.

Component Responsibilities

AWS / Stack Component	Primary Responsibility
Amazon Route 53	Manages DNS A records (Alias to CloudFront).
Amazon CloudFront	CDN handling HTTPS termination, edge routing, and origin caching.
Amazon S3	Private origin hosting static .well-known OS verification manifests.
AWS ALB	Routes dynamic requests (/join, API routes) to the NestJS cluster.
NestJS API	Evaluates User-Agent headers, processes referral logic, and executes store redirects.
React Native App	Handles incoming deep link events via native Linking listeners.

A request to /.well-known/* is served from S3. Every other path goes to the ALB and then to NestJS, which decides where to send the visitor.

Hosting the Verification Files on Amazon S3

This creates the storage for the two files that Android and iOS download to verify your domain. The bucket stays private, and CloudFront is the only service allowed to read it.

Step 1: Create a Private Amazon S3 Bucket

Open the Amazon S3 Console and click Create bucket.
Name the bucket (e.g., app-domain-verification-prod).
Keep Block all public access enabled.
Set Object Ownership to Bucket owner enforced.

S3 Bucket Structure
└── .well-known/
    ├── assetlinks.json
    └── apple-app-site-association

Step 2: Generate OS Verification Manifests

Create a local directory named .well-known containing the two required manifest files. Each file lists the apps that are allowed to open links on your domain.

Android Verification (.well-known/assetlinks.json)

[
  {
    "relation": [
      "delegate_permission/common.handle_all_urls"
    ],
    "target": {
      "namespace": "android_app",
      "package_name": "com.company.app",
      "sha256_cert_fingerprints": [
        "your-sha-fingerprint-here"
      ]
    }
  }
]

Pro Tip: Retrieve your SHA-256 fingerprint for production builds via Google Play Console under App Signing > App signing key certificate. The same page in Play Console also shows a ready-made Digital Asset Links JSON snippet for your app. The sha256_cert_fingerprints field is an array, so you can add the upload key fingerprint as a second entry if testers install builds outside the Play Store.

iOS Verification (.well-known/apple-app-site-association)

Note: Do not append .json to the file name.

{
  "applinks": {
    "details": [
      {
        "appIDs": [
          "9JA6382026.com.company.app"
        ],
        "components": [
          {
            "/": "/join*"
          }
        ]
      }
    ]
  }
}

The value in appIDs is your Apple Team ID followed by the app's bundle identifier. The components array controls which paths open the app. Here only paths that start with /join are claimed, so every other page on the domain keeps opening in the browser.

Apple sets three hosting rules for this file. It must be served over https:// with a valid certificate, it must return directly with no redirects, and it must sit in the .well-known directory. Since iOS 14, devices do not fetch the file from your server. They fetch it from an Apple-managed CDN, which requests your file within 24 hours. Expect a delay before changes reach devices.

Step 3: Upload Object & Set MIME Types

Upload the .well-known folder to your S3 bucket. Ensure the HTTP Content-Type metadata for both files is explicitly configured as application/json:

aws s3 cp .well-known/ s3://app-domain-verification-prod/.well-known/ \
  --recursive \
  --content-type "application/json"

CloudFront caches these files at the edge. When you change either file later, create an invalidation so the new version is served right away:

aws cloudfront create-invalidation \
  --distribution-id <your-distribution-id> \
  --paths "/.well-known/*"

Configuring CloudFront and Route 53

One CloudFront distribution serves both origins. Static verification files come from S3, and every other request on the domain goes to the NestJS service behind the ALB.

Step 1: Configure Origins & Origin Access Control (OAC)

Navigate to CloudFront > Distributions > Create Distribution.
Origin 1 (S3): Select your S3 bucket endpoint.
- Under Origin Access, select Origin Access Control (OAC) and create an OAC control setting.
- Apply the provided JSON policy to your S3 bucket policy to allow CloudFront access while keeping the bucket completely private.
Origin 2 (ALB): Add your Application Load Balancer domain (e.g., nestjs-alb-12345.ap-south-1.elb.amazonaws.com) using HTTPS.

Important: When CloudFront connects to an origin over HTTPS, the origin's certificate must include a domain name that matches the Origin Domain. If it does not match, CloudFront returns 502 Bad Gateway. You cannot issue a certificate for the default elb.amazonaws.com name. Create a DNS record such as origin-api.app.com that points to the ALB, attach an ACM certificate for that name to the ALB's HTTPS listener, and enter origin-api.app.com as the Origin Domain.

This certificate is created in the ALB's own Region.

Step 2: Configure Path-Based Cache Behaviors

Set up path routing rules to separate static verification from dynamic application requests:

Priority 1 (/.well-known/*): Target: S3 Origin. Caching Enabled (Managed-CachingOptimized).

Priority 2 (/join or Default *): Target: ALB Origin.
Cache Policy: Managed-CachingDisabled (TTL = 0). Dynamic redirects must never be cached by CloudFront edge nodes.
Origin Request Policy: Must forward Query Strings (code) and Headers (User-Agent, or CloudFront Device Detection headers: CloudFront-Is-Mobile-Viewer, CloudFront-Is-IOS-Viewer, CloudFront-Is-Android-Viewer).

Note: By default, CloudFront strips query parameters and overwrites the viewer User-Agent with Amazon CloudFront. Forwarding these headers/queries is required for NestJS platform detection.

Step 3: SSL Certificate & DNS Alias

Request an AWS Certificate Manager (ACM) public certificate for invite.app.com.
- Crucial Rule: The certificate must be created in the us-east-1 (N. Virginia) region to attach to CloudFront.
In CloudFront Settings, add invite.app.com under Alternate domain name (CNAME) and select the ACM certificate.
In Route 53, create an A Record for invite.app.com, choose Alias, and point it to the CloudFront distribution target (dXXXXXX.cloudfront.net).

Step 4: Protect ALB Origin (Direct Access Prevention)

To prevent users or malicious traffic from bypassing CloudFront and hitting the ALB directly:

In CloudFront Origin Settings for the ALB, add a custom HTTP header: X-Origin-Verify: <your-secret-token>.
In your ALB Listener rules (or NestJS middleware), block any incoming request that does not contain this exact X-Origin-Verify header value with a 403 Forbidden.
Restrict the ALB security group so it only accepts traffic from the AWS-managed prefix list com.amazonaws.global.cloudfront.origin-facing.

The prefix list and the header protect against different things. The prefix list blocks traffic that does not come from CloudFront. The header blocks traffic from other CloudFront distributions that could also reach your ALB. AWS recommends using both together.

If you are building a new stack, CloudFront VPC origins are another option. They let CloudFront reach an ALB in a private subnet, so the ALB has no public endpoint at all. AWS announced VPC origins in November 2024 at no additional cost.

Verifying Infrastructure Deployment

Run standard curl checks to verify headers and response bodies before proceeding with client-side integration:

# 1. Verify Android Verification Asset
curl -i https://invite.app.com/.well-known/assetlinks.json

# Expected Output:
# HTTP/2 200
# content-type: application/json
# x-cache: Hit from cloudfront

# 2. Verify iOS Verification Asset
curl -i https://invite.app.com/.well-known/apple-app-site-association

# Expected Output:
# HTTP/2 200
# content-type: application/json

The first request after an upload or invalidation returns x-cache: Miss from cloudfront. Run the command again to see the cached Hit response. A 301 or 302 status on either file is a failure, since both platforms reject redirects.

Two more checks confirm what the platforms themselves see:

# 3. Ask Google's Digital Asset Links API to read your Android file
curl "https://digitalassetlinks.googleapis.com/v1/statements:list?source.web.site=https://invite.app.com&relation=delegate_permission/common.handle_all_urls"

# 4. Check the copy of your iOS file held on Apple's CDN
curl -i https://app-site-association.cdn-apple.com/a/v1/invite.app.com

Check 3 returns your package name and fingerprint when Google can parse the file. Check 4 returns the file once Apple's CDN has fetched it, which can take up to 24 hours after the first upload.

Configuring Android App Links and iOS Universal Links

The server files say which app may open your links. The app must also declare which domain it handles. Both sides have to match before Android or iOS will open the app.

Android Configuration (AndroidManifest.xml)

Add the auto-verify intent filter inside your <activity> block (typically MainActivity):

XML

<activity
  android:name=".MainActivity"
  android:launchMode="singleTask"
  android:exported="true">

  <intent-filter android:autoVerify="true">
      <action android:name="android.intent.action.VIEW" />
      <category android:name="android.intent.category.DEFAULT" />
      <category android:name="android.intent.category.BROWSABLE" />

      <data
          android:scheme="https"
          android:host="invite.app.com"
          android:pathPrefix="/join" />
  </intent-filter>
</activity>

Three attributes matter most here:

android:autoVerify="true" tells Android to check assetlinks.json on install. On Android 6.0 and higher, the system verifies the hosts in your intent filters automatically when the app is installed.
android:launchMode="singleTask" reuses the running app instance. A link tapped while the app is open then reaches the existing Linking listener instead of starting a second copy of the screen.
android:pathPrefix="/join" limits the app to referral paths, which matches the /join* rule in the iOS file.

iOS Configuration (Xcode Capability)

Open your iOS workspace in Xcode.
Under Target > Signing & Capabilities, click + Capability and select Associated Domains.
Add the domain entry using the applinks: prefix:

applinks:invite.app.com

Handling Deep Links in React Native

The app can receive a link in two states. In a cold start, the link launches a closed app. In a warm launch, the link arrives while the app runs in the background. React Native's Linking API covers both: getInitialURL() for cold starts and the url event for warm launches.

Handle cold-start launches (getInitialURL) and background warm launches (addEventListener):

// src/services/DeepLinkHandler.ts
import { useEffect } from 'react';
import { Linking } from 'react-native';
import AsyncStorage from '@react-native-async-storage/async-storage';

export const PENDING_REFERRAL_KEY = '@pending_referral_code';

export function parseReferralCode(url: string | null): string | null {
  if (!url) return null;

  try {
    const parsedUrl = new URL(url);
    if (
      parsedUrl.hostname === 'invite.app.com' &&
      parsedUrl.pathname === '/join'
    ) {
      return parsedUrl.searchParams.get('code');
    }
  } catch (error) {
    console.error('Failed to parse deep link URL:', error);
  }
  return null;
}

export function useDeepLinkHandler() {
  useEffect(() => {
    // 1. Handle background/running app deep links
    const subscription = Linking.addEventListener('url', async ({ url }) => {
      const code = parseReferralCode(url);
      if (code) {
        await AsyncStorage.setItem(PENDING_REFERRAL_KEY, code);
        // Trigger navigation or dispatch action to store
      }
    });

    // 2. Handle cold-start deep links
    async function checkInitialUrl() {
      const initialUrl = await Linking.getInitialURL();
      const code = parseReferralCode(initialUrl);
      if (code) {
        await AsyncStorage.setItem(PENDING_REFERRAL_KEY, code);
      }
    }

    checkInitialUrl();

    return () => {
      subscription.remove();
    };
  }, []);
}

''While iOS requires server-side or grace-period attribution, Android allows us to extract the original referral code automatically from the Play Store install metadata upon first launch.''

Handling Android Deferred Links (Play Store Install Referrer)

On Android, when a user installs the app via a Play Store link containing &referrer=..., the referral code is preserved in the Play Store metadata. Use react-native-play-install-referrer to fetch it on first launch:

import { Platform } from 'react-native';
import AsyncStorage from '@react-native-async-storage/async-storage';
import PlayInstallReferrer from 'react-native-play-install-referrer';
import { PENDING_REFERRAL_KEY } from './DeepLinkHandler';
async function checkAndroidInstallReferrer() {
  if (Platform.OS !== 'android') return;

  const isFirstLaunch = await AsyncStorage.getItem('@first_launch_done');
  if (!isFirstLaunch) {
    PlayInstallReferrer.getInstallReferrer(async (installReferrerInfo, error) => {
      if (!error && installReferrerInfo?.installReferrer) {
        const params = new URLSearchParams(installReferrerInfo.installReferrer);
        const code = params.get('utm_content');
        if (code) {
          await AsyncStorage.setItem(PENDING_REFERRAL_KEY, code);
        }
      }
      await AsyncStorage.setItem('@first_launch_done', 'true');
    });
  }
}

Google keeps the referrer data for 90 days, and it does not change unless the app is reinstalled. Google also advises calling the API only once, during the first run after install. The @first_launch_done flag handles this. The flag is now written inside the callback, so a launch that closes before the Play Store responds will try again on the next start.

Building the NestJS Redirect Backend

When users without the app click the deep link, CloudFront routes the request to your NestJS server behind the ALB. The backend detects the platform and redirects the user to the appropriate App Store.

NestJS Controller (referral.controller.ts)

import { Controller, Get, Query, Req, Res, BadRequestException, Headers } from '@nestjs/common';
import { Request, Response } from 'express';

@Controller('join')
export class ReferralController {
  private readonly PLAY_STORE_URL = 'https://play.google.com/store/apps/details?id=com.company.app';
  private readonly APP_STORE_URL = 'https://apps.apple.com/app/id1234567890';
  private readonly WEB_FALLBACK_URL = 'https://app.com/download';

  @Get()
  handleReferralRedirect(
    @Query('code') code: string,
    @Req() req: Request,
    @Res() res: Response,
    @Headers('cloudfront-is-android-viewer') isAndroidCloudFront?: string,
    @Headers('cloudfront-is-ios-viewer') isIosCloudFront?: string,
  ) {
    if (!code || !/^[A-Za-z0-9_-]{3,16}$/.test(code)) {
      throw new BadRequestException('Invalid or missing referral code.');
    }

    const userAgent = req.headers['user-agent'] || '';

    // Check CloudFront Device Headers first, fallback to User-Agent
    const isAndroid = isAndroidCloudFront === 'true' || /Android/i.test(userAgent);
    const isIOS = isIosCloudFront === 'true' || /iPhone|iPad|iPod/i.test(userAgent);

    if (isAndroid) {
      // Appends Play Store install referrer parameter
      return res.redirect(302, `${this.PLAY_STORE_URL}&referrer=utm_source%3Dreferral%26utm_content%3D${encodeURIComponent(code)}`);
    }

    if (isIOS) {
      return res.redirect(302, this.APP_STORE_URL);
    }

    return res.redirect(302, `${this.WEB_FALLBACK_URL}?code=${encodeURIComponent(code)}`);
  }
}

Production Controls: Abuse & Replay Prevention

Rate Limiting: Protect the /join endpoint using @nestjs/throttler (e.g., max 10 requests per minute per IP) to prevent referral code enumeration attacks.
Referral Claim Limits: Enforce database constraints limiting one claimed referral code per user account and unique device ID.
Fraud Checks: Flag accounts claiming codes where the referrer and referee share the same IP address or payment method.

Fallback Strategies for Uninstalled Users

Because iOS Universal Links and standard App Links do not preserve URL query params across fresh store downloads without a vendor SDK, choose one of these production-ready custom fallbacks:

Option A: 7-Day In-App Grace Period

Allow newly registered users to manually claim or enter a referral code within 7 days of account creation.

Claims Service (referral-claim.service.ts)

import { Injectable, BadRequestException, ConflictException } from '@nestjs/common';
import { PrismaService } from '../prisma/prisma.service';

@Injectable()
export class ReferralClaimService {
  constructor(private readonly prisma: PrismaService) {}

  async claimReferral(userId: string, code: string) {
    const user = await this.prisma.user.findUnique({ where: { id: userId } });
    if (!user) throw new BadRequestException('User not found.');

    // 1. Enforce 7-Day Expiration Policy
    const gracePeriodMs = 7 * 24 * 60 * 60 * 1000;
    const isEligible = Date.now() - new Date(user.createdAt).getTime() <= gracePeriodMs;

    if (!isEligible) {
      throw new BadRequestException('The 7-day grace period for applying a referral code has expired.');
    }

    // 2. Validate Referral Code Existence & Ownership
    const referralCodeOwner = await this.prisma.referralCode.findUnique({ where: { code } });
    if (!referralCodeOwner) {
      throw new BadRequestException('Invalid referral code.');
    }

    if (referralCodeOwner.ownerId === userId) {
      throw new BadRequestException('Users cannot claim their own referral code.');
    }

    // 3. Atomically Record Claim
    return this.prisma.$transaction(async (tx) => {
      const existingClaim = await tx.referralClaim.findUnique({ where: { userId } });
      if (existingClaim) {
        throw new ConflictException('A referral code has already been applied to this account.');
      }

      const claim = await tx.referralClaim.create({
        data: {
          userId,
          referralCode: code,
          referrerId: referralCodeOwner.ownerId,
        },
      });

      // Credit rewards to both parties...
      return claim;
    });
  }
}

Common Pitfalls

Symptom	Primary Cause	Solution
App opens in browser instead of app	Fingerprint mismatch or missing domain configuration.	Check assetlinks.json SHA-256 fingerprint. On iOS, verify applinks: is present in Xcode Capabilities.
CloudFront returns 403 Forbidden	S3 Bucket Policy is blocking CloudFront OAC.	Update S3 bucket policy with proper AWS:SourceArn matching CloudFront distribution ARN.
/join request hits wrong origin	Priority order misconfiguration in CloudFront Cache Behaviors.	Ensure /.well-known/* behavior is listed above Default (*) behavior in CloudFront console.
Universal Link fails on iOS Simulator	Scheme/domain not registered in Simulator environment or bad test invocation.	Test Universal Links on iOS Simulator via CLI using xcrun simctl openurl booted "`https://invite.app.com/join?code=ABC123`". Ensure Associated Domains capability is added to the Xcode scheme.

Production Test Matrix

The Play Store only records the referrer when the install starts from the redirected link. Test the Android path with the full /join redirect instead of opening the store listing directly.

Platform	App Status	Test Trigger URL	Expected Behavior	Verification Step
Android	Installed	`https://invite.app.com/join?code=TEST1`	Intercepted by OS; opens app directly to onboarding screen.	App opens without browser redirect; code=TEST1 present in app state.
Android	Not Installed	`https://invite.app.com/join?code=TEST1`	CloudFront routes to NestJS; 302 redirects to Play Store with &referrer=....	Inspect Network tab; confirm Play Store URL contains referrer query parameter.
iOS	Installed	`https://invite.app.com/join?code=TEST1`	Intercepted by iOS Universal Links; opens app directly.	App opens directly from Safari/Notes/Messages without opening a web page.
iOS	Not Installed	`https://invite.app.com/join?code=TEST1`	CloudFront routes to NestJS; 302 redirects to Apple App Store.	User is redirected to App Store; code applied post-install via 7-day grace period fallback.

Self-Hosted vs Third-Party SDK

The table below compares this self-hosted setup with a third-party SDK. Use it to decide whether the setup covers your attribution needs.

Capability	Custom AWS Architecture	Third-Party SDK (Branch / AppsFlyer)
Direct Deep Linking (App Installed)	Native	Native
Zero Vendor Subscription Cost	Yes	High SaaS Tiers
Data Privacy & GDPR Compliance	100% Data Ownership	Third-party Tracking
Zero External SDK Overhead	Native Modules Only	Large SDK Footprint
Automatic Install Fingerprinting	Requires Web Fallback / Grace Period	Probabilistic / Deterministic Engine

The self-hosted setup suits teams that need branded links, referral onboarding and full control of user data. A third-party SDK is the better fit when marketing needs automatic install attribution on iOS across paid ad networks.

Need Help Building This?

Setting up deep linking infrastructure from scratch takes real engineering time: AWS configuration, platform-specific verification files, backend routing logic, and ongoing maintenance as app store requirements shift.

If your team needs hands-on support, GeekyAnts' Mobile Engineering services cover React Native architecture, backend integration, and infrastructure setup like the one in this guide. Our engineers are active contributors to the React Native ecosystem and have shipped 300+ React Native apps for global enterprises.

Conclusion

This setup gives you deep links that run on your own domain and your own AWS account. Android and iOS verify the domain through two static files on S3, CloudFront splits those files from the dynamic /join route, and NestJS sends each visitor without the app to the correct store. The React Native app reads the code on every launch path, the Play Install Referrer carries it through Android installs, and the grace period, web fallback or invite matching recover it on iOS.

With the checklist complete and the test matrix passing, the same infrastructure can carry product, notification and campaign links by adding their paths to the manifests and the app router.

Build an MCP-Powered QA Agent with Playwright and TypeScript

GeekyAnts India Pvt Ltd — Tue, 27 Oct 2026 05:10:54 +0000

Build an MCP QA agent that runs Playwright tests, validates APIs, and analyzes failures through a controlled, auditable tool layer with human approval.

Connecting AI to Playwright, APIs, test evidence, and CI/CD.

Abstract

AI-assisted software testing is moving beyond code generation toward agents that can inspect applications, execute tests, analyze failures, and interact with engineering tools. Model Context Protocol (MCP) provides a standardized interface for AI applications to discover and invoke tools exposed by external systems. The current MCP TypeScript SDK v2 exposes tools, resources, and prompts.

This article presents a code-first QA architecture using TypeScript, Playwright, Zod, structured outputs, permission boundaries, evidence collection, and human approval. The objective is controlled access to deterministic QA capabilities rather than unrestricted autonomy.

Introduction

A modern QA workflow spans browser automation, API validation, logs, traces, test reports, test data, source control, and CI/CD. An AI model can reason across these systems, but reasoning alone does not give it safe or structured access to them.

MCP provides a standardized protocol boundary between an AI host and external tools and data. The engineering challenge is therefore not simply connecting MCP to Playwright; it is designing a QA tool layer with explicit capabilities, validated inputs, controlled environments, observable outputs, and safe failure behavior.

This article focuses on a practical architecture: an MCP server exposes selected QA operations while Playwright and API clients remain the deterministic execution layer. The AI can request an operation, but the server owns validation and authorization.

Architecture

The MCP TypeScript SDK supports servers exposing tools, resources, and prompts. Tools are appropriate for actions and side effects; resources are appropriate for read-oriented data such as reports or configuration.

The SDK also supports stdio and Streamable HTTP transports. See the official MCP SDK documentation for the current v2 implementation.

Project Structure

qa-mcp-agent/
├── src/
│   ├── server.ts
│   ├── tools/
│   │   ├── playwright.ts
│   │   ├── api.ts
│   │   └── evidence.ts
│   ├── security/
│   │   └── permissions.ts
│   └── utils/
│       └── sanitize.ts
├── tests/
│   └── checkout.spec.ts
├── artifacts/
├── package.json
└── tsconfig.json

Creating the MCP Server

For a current TypeScript implementation, use the v2 server package and Zod for input validation.

npm init -y
npm install @modelcontextprotocol/server zod @playwright/test

import { McpServer } from "@modelcontextprotocol/server";

const server = new McpServer({
  name: "qa-mcp-server",
  version: "1.0.0"
});

The current v2 SDK documents @modelcontextprotocol/server as the server package and uses Zod schemas for typed validation.

Expose a Controlled Playwright Tool

import * as z from "zod/v4";

server.registerTool(
  "run_playwright_test",
  {
    title: "Run Playwright Test",
    description: "Execute an approved Playwright test file",
    inputSchema: { testFile: z.string() },
    outputSchema: {
      passed: z.boolean(),
      output: z.string()
    }
  },
  async ({ testFile }) => {
    validateTestFile(testFile);
    const result = await runPlaywrightTest(testFile);
    const output = { passed: result.passed, output: result.output };
    return {
      content: [{ type: "text", text: JSON.stringify(output) }],
      structuredContent: output
    };
  }
);

MCP tool schemas validate arguments before the handler executes, and structured outputs can be validated against an output schema. Clients can discover tools with listTools and invoke them with callTool.

Keep Playwright Deterministic

import { execFile } from "node:child_process";

function runPlaywrightTest(testFile: string) {
  return new Promise<{ passed: boolean; output: string }>((resolve) => {
    execFile(
      "npx",
      ["playwright", "test", testFile],
      { timeout: 120000 },
      (error, stdout, stderr) => {
        resolve({
          passed: !error,
          output: `${stdout}\n${stderr}`
        });
      }
    );
  });
}

The server should not pass arbitrary shell strings from the model into an operating-system shell. Use an allowlist and validated arguments.

Permission Boundaries

const ALLOWED_TESTS = new Set([
  "tests/login.spec.ts",
  "tests/checkout.spec.ts",
  "tests/cart.spec.ts"
]);

function validateTestFile(testFile: string) {
  if (!ALLOWED_TESTS.has(testFile)) {
    throw new Error(`Test file is not approved: ${testFile}`);
  }
}

The security boundary belongs in deterministic application code, not in the model prompt.

API Testing as Another Tool

server.registerTool(
  "validate_api",
  {
    title: "Validate API",
    description: "Send an HTTP request and validate the response",
    inputSchema: {
      method: z.enum(["GET", "POST", "PUT", "DELETE"]),
      url: z.string(),
      expectedStatus: z.number()
    }
  },
  async ({ method, url, expectedStatus }) => {
    const response = await fetch(url, { method });
    const output = {
      status: response.status,
      passed: response.status === expectedStatus
    };
    return {
      content: [{ type: "text", text: JSON.stringify(output) }],
      structuredContent: output
    };
  }
);

In production, URL allowlists, authentication handling, request-body validation, rate limits, and environment restrictions should be added before exposing mutation-capable API tools.

Use Resources for Test Evidence

MCP resources are a better fit for read-oriented evidence than action tools. A QA server can expose resources such as test-results://latest or trace://checkout.

server.registerResource(
  "latest-results",
  "test-results://latest",
  { title: "Latest Playwright Results", mimeType: "application/json" },
  async (uri) => {
    const results = await readLatestResults();
    return {
      contents: [{
        uri: uri.href,
        mimeType: "application/json",
        text: JSON.stringify(results)
      }]
    };
  }
);

Failure Analysis

server.registerTool(
  "analyze_test_failure",
  {
    title: "Analyze Test Failure",
    inputSchema: {
      testName: z.string(),
      error: z.string(),
      consoleErrors: z.array(z.string()),
      networkFailures: z.array(z.string())
    },
    outputSchema: {
      classification: z.enum([
        "APPLICATION", "TEST", "NETWORK",
        "DATA", "ENVIRONMENT", "UNKNOWN"
      ]),
      evidence: z.array(z.string())
    }
  },
  async (input) => {
    const result = await analyzeFailure(input);
    return {
      content: [{ type: "text", text: JSON.stringify(result) }],
      structuredContent: result
    };
  }
);

Separate observed evidence from AI-generated hypotheses. The AI can classify a failure and recommend an investigation path, but the underlying evidence should remain inspectable.

The UNKNOWN State

type Diagnosis =
  | "APPLICATION" | "TEST" | "NETWORK"
  | "DATA" | "ENVIRONMENT" | "UNKNOWN";

const analysisInstruction = `
Analyze only the supplied evidence.
Do not invent missing information.
If evidence is insufficient, return UNKNOWN.
Separate observed evidence from hypotheses.
`;

Allowing UNKNOWN is an important reliability control. It prevents incomplete logs from being converted into false certainty.

A QA Workflow

Human Approval for High-Risk Operations

type RiskLevel = "LOW" | "MEDIUM" | "HIGH" | "CRITICAL";

const TOOL_RISK: Record<string, RiskLevel> = {
  read_test_results: "LOW",
  run_playwright_test: "LOW",
  validate_api: "LOW",
  create_test_data: "MEDIUM",
  modify_test: "HIGH",
  delete_test_data: "CRITICAL"
};

function requiresApproval(risk: RiskLevel) {
  return risk === "HIGH" || risk === "CRITICAL";
}

The AI can request an action, but the policy layer determines whether it is permitted. Human approval is an explicit control point rather than a suggestion embedded in a prompt.

Auditability

interface AuditEvent {
  timestamp: string;
  agent: string;
  tool: string;
  arguments: unknown;
  environment: string;
  decision: "ALLOWED" | "BLOCKED" | "APPROVAL_REQUIRED";
  approvedBy?: string;
}

Every tool invocation should produce an auditable event so teams can determine which agent called which tool, against which environment, and with what authorization decision.

Connecting the Client

const { tools } = await client.listTools();

const result = await client.callTool({
  name: "run_playwright_test",
  arguments: {
    testFile: "tests/checkout.spec.ts"
  }
});

The MCP client API supports discovery and invocation of tools and access to resources. Playwright also provides an MCP server for browser automation, which can be used when the use case is direct browser interaction.

Testing the MCP Server

test("blocks unapproved test", async () => {
  await expect(
    runPlaywrightTest("tests/../../secret.spec.ts")
  ).rejects.toThrow("Test file is not approved");
});

test("requires approval for data deletion", async () => {
  const decision = authorize({
    tool: "delete_test_data",
    environment: "production"
  });
  expect(decision).toBe("BLOCKED");
});

The MCP server itself is production software. Its schemas, authorization rules, environment restrictions, error handling, and tool contracts should therefore be covered by automated tests.

Evaluation: Avoid Unsupported Benchmark Claims

A strong implementation should be evaluated rather than supported by invented numbers. Compare conventional QA with MCP-assisted QA using the same scenarios and report the actual environment and measurement method.

Metric	Measurement
Execution time	Repeated wall-clock runs
Diagnosis time	Failure to actionable hypothesis
Correct classification	Correct diagnoses / total failures
Human interventions	Manual approvals or interventions
Blocked unsafe actions	Policy violations prevented
Tool-call failures	Invalid or failed calls
Cost	LLM/tool/API cost per workflow

If numerical performance results are reported, include the actual MCP/Node/Playwright versions, machine or CI environment, workload, number of runs, warm-up method, and measurement definition. Do not present illustrative figures as measured evidence.

Production Architecture

Trade-offs

More tools increase capability but also increase the action space the model must select from. More autonomy can reduce manual effort but increases the impact of incorrect decisions. More evidence can improve diagnosis but can increase data exposure and model cost. More guardrails improve safety but can introduce friction for unusual workflows.

The engineering objective is therefore not maximum autonomy. It is useful autonomy within an explicitly defined risk boundary.

Conclusion

MCP gives QA engineers a standardized way to expose testing capabilities to AI systems, but the protocol itself does not define a safe QA architecture. That responsibility belongs to the implementation.

A production-oriented QA MCP server should expose narrow, validated capabilities; keep browser and API execution deterministic; separate tools from read-only evidence; enforce environment and permission boundaries outside the model; require human approval for high-impact actions; and record enough evidence to make agent behavior auditable.

The important shift is from “AI controls the test environment” to “AI operates through a controlled QA interface.” That distinction provides the foundation for building AI-assisted testing systems that remain useful, observable, and governable as their autonomy increases.

References

MCP TypeScript SDK v2 documentation; MCP client tool/resource APIs; MCP resources and tools documentation; Playwright MCP documentation. These are the current primary technical references used for the implementation.

Railway Environment Variable Incident: What Laravel & Cloud Application Teams Should Do Now

GeekyAnts India Pvt Ltd — Tue, 27 Oct 2026 05:10:02 +0000

On October 21, 2026, Railway disclosed that an unauthorized third party may have accessed the systems that store its customers' environment variables. That exposes a problem that rarely comes up when engineering teams discuss cloud security: what happens when the infrastructure you depend on to recover from an incident is itself affected by that incident?

If your team deploys on a platform like Railway, you have probably already been asked the same question internally: what do we do now?

This post is a practical answer. It's written for Laravel and cloud application teams, and it deliberately avoids alarm. Incidents like this are a reasonable moment to take a breather to check your own setup, and most of what follows is good hygiene you would want in place for your team anyway.

What Exactly Happened?

The facts below come from Railway's own disclosure and from press coverage published on the first day after it. Details are still developing, so treat this as a snapshot:

Railway said it detected unusual activity on October 21 in the systems that store customer environment variables.
Its initial assessment was that an unauthorized party may have accessed encrypted variables and, for some of them, obtained the decrypted contents. Reporting describes the affected values as including API keys, database credentials and private registry credentials.
Railway paused deployments while it investigated. The Defiant reported that this left customers unable to push updated applications carrying replacement credentials. Enterprise builds were restored late Wednesday, while customers on other plans were still waiting to redeploy.
Railway's early analysis suggested the impact may be limited to a subset of users, but it still recommended that every customer rotate every stored secret and review service logs for unusual activity. It has not published a detailed scope.
Railway described the incident as part of a series of attacks on platform companies aimed at compromising AI inference providers, and said it had notified upstream providers so they could proactively revoke potentially exposed keys.

One claim is still being examined as this blog was written. A developer has alleged that roughly $370,000 was stolen through secrets stored on Railway. On-chain analysts have tracked funds moving from a suspected attacker's wallet, but the link between that wallet and the Railway incident has not been confirmed.

Why Environment-Variable Exposure Matters?

Environment variables are the standard way to keep configuration out of code. In practice, they're also where applications keep the keys to everything else: database passwords, payment and email credentials, cloud storage keys, signing secrets and third-party API tokens.

That makes them an unusually high-value target. An attacker who reads a single set of variables doesn't need to find a vulnerability in your application. They already hold credentials that your application is trusted to use, and those credentials often work from anywhere on the internet.

Secrets Are Becoming the New Security Perimeter

There is also a structural point here, and it's the main lesson we'd draw from this incident. Secrets are the new perimeter. Attackers increasingly go after the layer that stores and manages credentials, rather than the services those credentials unlock. It's often the softer target, and it can be just as valuable if the stolen keys grant access to paid AI accounts, databases or cloud resources. Your application can be well-written and fully patched, and its security still depends on every platform that holds a copy of its secrets.

The practical conclusion is not "stop using managed platforms." They are vital in removing a great deal of operational burden for growing teams, and most teams are safer for using them. The lesson here is to design in a manner that a single vendor's bad day is survivable, because the credentials it holds are scoped, short-lived and quick to replace.

What Credentials To Review And Rotate Post the Railway Security Incident?

Railway's guidance was to rotate everything stored as an environment variable, and that's the right default for anyone affected. The harder part is knowing what "everything" includes. For a typical Laravel application, work through this list:

Application key (APP_KEY). This encrypts sessions, cookies and any data you encrypt with Laravel's encrypter. Rotating it will invalidate sessions and make previously encrypted data unreadable unless you plan for it. Recent Laravel versions support graceful rotation through APP_PREVIOUS_KEYS, which lets old values keep decrypting while new ones are written.
Database and cache credentials. DB_PASSWORD, any read-replica credentials, and REDIS_PASSWORD. Also check whether the database is reachable from the public internet; if it is, a leaked password is directly usable.
Cloud and storage keys. AWS, GCP or S3-compatible access keys, plus any service-account files stored as variables. Railway noted that it would handle resets for some platform-managed items, such as storage buckets and GitHub connections, but you should confirm that rather than assume it.
Third-party service tokens. Payment gateways (Stripe, Razorpay and similar), email and SMS providers, push and broadcasting services (Pusher, Ably), search and analytics keys.
AI and LLM provider keys. OpenAI, Anthropic and similar keys deserve priority. They're easy to monetize, they can run up large bills quickly, and Railway has said AI inference access appears to be the attackers' target.
Auth and signing material. OAuth client secrets, Passport or Sanctum-related keys, JWT signing secrets, and webhook signing secrets.
Registry and deployment credentials. Private container registry tokens, package registry tokens (for example private Composer repositories), deploy keys and Git provider tokens.
Wallet and signing keys. If any private key that can move money was ever stored as a variable, treat it as the most urgent item and move funds to a new key.

A sensible order is: anything that can spend money or move data first, then anything that grants access to customer data, then everything else. After rotating, check the logs of each connected service for activity you don't recognize in the window around the disclosure, and keep the old credentials revoked rather than just replaced.

Secret-Management Best Practices to Consider

Incident response is the expensive way to learn about secrets. These practices make the next one cheaper.

Keep a secrets inventory. You can't rotate what you can't list. For each secret, record what it unlocks, where it's stored, who owns it and when it was last rotated. Most teams are surprised by how many forgotten keys turn up.

Use a dedicated secrets manager for high-value secrets. Tools such as HashiCorp Vault, AWS Secrets Manager or Doppler give you access control, audit logs and rotation workflows that a plain variable store doesn't. For the most sensitive material, such as signing keys, use a KMS or HSM so the raw key never lands in your application's environment at all.

Prefer short-lived and dynamic credentials. A database password that expires in an hour is a much smaller liability than one that has been valid for three years. Where your stack supports it, issue credentials on demand rather than storing them.

Apply least privilege everywhere. Use separate keys per service and per environment. Give database users only the permissions the application needs. Put spend caps and usage limits on AI and payment keys, so that a leaked key has a ceiling on the damage it can do.

Make a stolen secret less useful. Put databases and caches behind private networking or IP allowlists, so a leaked connection string is not enough on its own. Where providers support it, restrict API keys by IP or referrer.

Keep secrets out of code, images and logs. Check your Git history, build logs, container layers and error reporters. In Laravel, remember that debug pages and verbose exception output can echo environment values if APP_DEBUG is left on in production.

Practice rotation before you need it. Treat rotation as a routine, scripted task. If it only happens during an incident, it will be slow, manual and error-prone.

CI/CD and deployment security

The Railway incident shows a point that is easy to miss: rotation is a deployment problem and not just a missed item on a security checklist. Telling teams to rotate secrets is only useful if they can then ship the new values. When deployments were paused, rotating a key and getting that key into a running application became two separate problems.

A few things make a real difference here:

Keep a path to redeploy that doesn't depend on a single provider. That doesn't require multi-cloud complexity. It can be as simple as a documented, tested way to stand up your application elsewhere, with infrastructure defined as code.
Scope CI/CD secrets tightly. Pipeline variables should be limited per environment, protected so they are only available on trusted branches, and masked in logs. Avoid one broad token that can touch every project.
Prefer short-lived identity over stored keys in pipelines. OIDC-based federation to your cloud provider lets a pipeline obtain temporary credentials without a long-lived key sitting in a variable.
Separate secrets by environment. Production credentials should not be present in staging or preview environments, where controls are usually looser.
Pin and review what your pipeline runs. Lock third-party actions and dependencies, and review changes to workflow files like code, since a modified pipeline is a direct route to your secrets.
Make rotation a one-command or one-pipeline operation. The goal is that rotating a credential and redeploying with it takes minutes, not an afternoon of coordination.
Keep audit trails. Know who changed a secret or a deployment setting and when.

How To Think About Production Security in Light of Security Incidents?

It's tempting to respond to an incident like this in one of two ways: switch vendors, or bring everything in-house. Neither is usually the answer. Switching vendors moves the risk. Running your own infrastructure after a platform breach can make things worse, because you become the next potential breach, often with a smaller security team than the platform you left. A calmer approach is to ask a different set of questions about every system that touches production.

When a credential is exposed, the real concern is how much access that single key was allowed to have. Rotating it is the immediate fix. But if the same key can expose sensitive data or disrupt several services, that points to an architectural problem we should have addressed much earlier.

— Saurabh Sahu, Chief Technology Officer (CTO)

Assume any single platform can have a bad day. The question is not whether your vendors are perfect but what happens to you when one of them isn't. If the answer is "we rotate a few scoped credentials and carry on," you're in good shape. If it's "everything we own is exposed," then the design itself needs a harder assessment.

Shrink the blast radius by design. Scoped keys, private networking, spend limits and separated environments are not exotic. Together they turn a potential catastrophe into a contained, boring incident.

Treat detection as part of the system. Alerts on unusual API usage, billing spikes and unfamiliar logins mean you may notice misuse before any vendor's email arrives. In some reports of this incident, teams had already rolled their keys before the notification reached them.

Do vendor due diligence that matches what you store. What a platform holds should determine how much scrutiny it gets. Ask where secrets are encrypted, who can decrypt them, what the incident communication process looks like, and how quickly you can recover if the platform is unavailable.

Keep your crown jewels elsewhere. Anything that can move large sums of money, sign on your behalf or unlock all customer data should live in a purpose-built secrets system, and ideally not within a general-purpose variable store.

Rehearse. Run a short exercise: pick a key, assume it leaked, and time how long it takes to revoke, replace and redeploy. The gaps you find in an hour of practice are the ones that would otherwise appear during a real incident.

How We Approached It Across 60+ Client Projects

The Railway disclosure wasn't the only upstream problem our teams were tracking this week. A critical PHP vulnerability, a Node.js remote-code-execution issue and a certificate authority key problem were all on the list too. None of it was code we had written, but all of it was ours to handle, because our clients' production systems depended on it.

Here is what that looked like in practice:

We found out what runs where. The hard part wasn't patching. It was answering "which projects are affected?" quickly.
We put everything in one tracker. Every project, its stack, its version and whether it's safe. Our DevOps team owns the sheet and makes sure every upgrade is closed, not just started.
We called the highest-risk client first. They heard it from us before they read about it somewhere else.
We resisted the wrong lesson. Moving to self-managed infrastructure after a platform breach would not have made our clients safer. It would have swapped one dependency for another risk.

You can't stop PHP, Node or a vendor from failing. The only number you own is time-to-safe: how fast you know, how fast you patch, and how fast the client hears from you.

— Saurabh Sahu, Chief Technology Officer (CTO)

That is the idea behind everything in this post. Rotating a secret, patching a runtime and telling a client are all the same measurement: how quickly you go from "something happened" to "we know where we stand."

So here is a question worth asking your own team: how long would it take to answer "what version of Node (or PHP) is every production service on?" If the answer is more than an hour, fix that before you fix anything else.

Where Geekyants Can Help

Most teams are familiar with an ideal security stance. The difficulty is finding the time to do it while also shipping features. This is where we typically work with clients:

Production security reviews. We audit how secrets, access and deployments are handled across your application and infrastructure, and give you a prioritized list of fixes rather than a long report.
Cloud architecture and modernization. We help teams move from single-vendor, static-credential setups to architectures with secrets management, private networking, infrastructure as code and portable deployments.
DevOps and CI/CD hardening. We build pipelines where rotation and redeployment are routine, and where credentials are short-lived and scoped.
Laravel engineering. If your application is built on Laravel and you need senior help with configuration, key rotation, queue and cache setup or a broader hardening pass, you can hire Laravel developers from GeekyAnts who work on production systems every day.

If you're working through your own rotation right now, or you'd like a second pair of eyes on your setup once the dust settles, we're happy to talk it through.

A short checklist for this week

Rotate every secret stored on the affected platform, starting with anything that can spend money or access customer data.
Check the logs and billing of each connected service for unfamiliar activity.
Confirm your database and cache are not reachable from the open internet.
Build a secrets inventory with an owner and a rotation date for each entry.
Move your highest-value secrets into a dedicated secrets manager or KMS.
Add spend caps and usage alerts to AI and payment keys.
Test that you can redeploy quickly with new credentials, and write the steps down.
Keep one tracker of every project, its stack, its versions and its status, so you can answer "what runs where?" in under an hour.

Building a Real-Time Photo Sync Video Experience Using 100ms SDK and Reactive State Management

GeekyAnts India Pvt Ltd — Fri, 23 Oct 2026 10:08:14 +0000

This blog explains how to build a real-time video experience with synchronized photo sharing, chat, and reactive state management using the 100ms SDK.

Building real-time video experiences traditionally required deep expertise in WebRTC, signaling infrastructure, media synchronization, and browser-specific handling. With modern SDKs like 100ms, teams can now focus more on product experience and less on low-level media infrastructure.

In this article, we will walk through how to build a real-time video calling experience with synchronized photo sharing and chat, powered by the 100ms JavaScript SDK, a reactive state architecture, and a modular frontend design.

We will cover:

Core concepts of 100ms architecture
Pre-join (green room) experience
Joining and managing a live video room
Real-time photo synchronization across participants
Chat integration using broadcast messaging
State management strategies for responsive UI
Synchronization correctness, conflict resolution, and reconnect handling

High-Level Architecture

Before diving into the implementation, it’s worth understanding how the system components interact.

On the backend, when a meetup is created, the server calls the 100ms Rooms API to provision a room and generate authentication tokens for invited participants. Each participant receives a token tied to the same room ID, ensuring everyone joins the same session.

On the frontend, a single HMSReactiveStore instance acts as the centralized source of truth for media state, participants, tracks, and messages. Both the Green Room and the Live Call screens subscribe to slices of this store rather than maintaining disconnected local state.\
The SDK exposes three key primitives:

Store → Reactive room state
Actions → Methods for operations such as join, mute, or messaging
Notifications → Event listeners for peer updates and room events

This architecture provides a predictable data flow for real-time applications and keeps media state, participant state, and UI subscriptions centralized. The synchronized-photo feature still requires explicit conflict-resolution and lifecycle handling, which we address below.

Why a Pre-Join (Green Room) Matters

A pre-join experience is critical for user confidence and system readiness.\
100ms provides a preview API that allows users to:

Test camera and microphone
Set initial mute/unmute states
Validate network readiness and authentication before joining

Key Implementation Flow

Fetch authentication token from backend
Call preview() with user config
Attach local video track to UI
Allow user to toggle mic and camera

await hmsActions.preview({
 authToken: '<token from backend>',
 userName: 'Alice',
});

Once preview is successful:

Local tracks are created
Permissions are verified
Settings persist into the join phase

This reduces friction during room entry.

Setting Up the 100ms SDK

The 100ms JavaScript SDK ships through @100mslive/hms-video-store.\
Everything flows from a single HMSReactiveStore instance:

// Initialize once — treated as a module-level singleton
import { HMSReactiveStore } from '@100mslive/hms-video-store';
const hmsManager = new HMSReactiveStore();
hmsManager.triggerOnSubscribe(); // fire callbacks immediately with current state
const hmsStore = hmsManager.getStore();
const hmsActions = hmsManager.getActions();
const hmsNotifications = hmsManager.getNotifications();

Calling triggerOnSubscribe() is important because it ensures subscribers immediately receive the current SDK state instead of waiting for the next state update.

Reading State with Selectors

The store exposes state through memoized selectors rather than raw state objects. This enables efficient reactive subscriptions.

import {
  selectPeers,
  selectIsLocalVideoEnabled,
  selectIsLocalAudioEnabled,
  selectLocalVideoTrackID,
  selectLocalPeer,
  selectTrackByID,

  selectPeerByID
} from '@100mslive/hms-video-store';

// One-time read
const isVideoOn = hmsStore.getState(selectIsLocalVideoEnabled);

// Reactive subscription — returns an unsubscribe function
const unsubscribe = hmsStore.subscribe((enabled) => {
  // runs whenever local video enabled/disabled
  updateVideoButton(enabled);
}, selectIsLocalVideoEnabled);

Subscribing to fine-grained selectors helps avoid unnecessary re-renders. In our implementation, this was useful for participant and track updates because changes could be scoped to the affected selector rather than causing broad UI updates.

The Green Room: Pre-Join Experience

The Green Room acts as a staging area before users join the room.

Attaching Local Video

Once preview mode starts, we subscribe to the local video track and attach it to a <video> element.

// Wire local video into a <video> element
hmsStore.subscribe(async (trackId) => {
  if (trackId && videoElement) {
    await hmsActions.attachVideo(trackId, videoElement);
  } else if (videoElement) {
    videoElement.srcObject = null;
  }
}, selectLocalVideoTrackID);

If the user disables their camera, the video track disappears and the UI reflects the change.

Toggling Camera and Microphone

Media controls during preview work the same way as they do during a live call.

// Toggle camera
const next = !hmsStore.getState(selectIsLocalVideoEnabled);
await hmsActions.setLocalVideoEnabled(next);

// Toggle mic
const next = !hmsStore.getState(selectIsLocalAudioEnabled);
await hmsActions.setLocalAudioEnabled(next);

Because the UI subscribes to the store, button states remain synchronized.

Transitioning to the Live Call

When the user clicks "Join", we stop the preview session first by calling hmsActions.leave(). This releases the preview peer from the 100ms session and prevents a ghost local peer from remaining alongside the new peer after the user joins. After leave completes, we navigate to the Live Call route.

The camera/microphone preferences chosen in the Green Room are persisted to a Pinia store (with localStorage persistence) so the Live Call can restore them on mount without asking the user again.\
To join a room, 100ms requires:

Auth token
Username

The join flow is straightforward:

await hmsActions.join({
  authToken: savedToken,
  userName:  'Alice',
});

// Restore preferences from Green Room
await Promise.all([
  hmsActions.setLocalVideoEnabled(preferredCameraOn),
  hmsActions.setLocalAudioEnabled(preferredMicOn),
]);

After joining:

The SDK emits events such as ON_JOIN, ON_PEER_UPDATE, and ON_TRACK_UPDATE
Reactive store subscriptions keep the UI aligned with the resulting SDK state

Rendering the Peer Strip

All connected peers are available via the selectPeers selector. We subscribe to it and re-render the participant strip whenever anyone joins or leaves. The local peer is sorted to the front; remote peers are sorted by join time.

Each peer tile has a <video> element whose DOM reference is stored in a Map keyed by peer ID. When a peer's video track changes (e.g. they turn their camera on), we call attachVideo to bind the new track. When a peer leaves, we call detachVideo to clean up.

// Subscribe to the peer list — fires on every join/leave
hmsStore.subscribe((peers) => {
  renderParticipants(peers);
}, selectPeers);

// Attach video for a specific peer
async function attachPeerVideo(peer, videoEl) {
  if (peer.videoTrack) {
    await hmsActions.attachVideo(peer.videoTrack, videoEl);
  }
}

This provides:

UI updates when participants join or leave
Consistent track state across components

The store acts as a single source of truth for SDK-backed room state, while application-specific state such as the current synchronized photo is handled separately with explicit synchronization rules.

Showing Live Audio and Video Status

For remote peers, track enabled/disabled state is read from the store using selectTrackByID. For the local peer, we use selectIsLocalVideoEnabled and selectIsLocalAudioEnabled directly. This allows the mic icon next to each participant tile to update in real time as participants toggle their devices, without polling.

await hmsActions.attachVideo(trackId, videoElement);

Important considerations:

Handle track changes dynamically
Detach video when track is removed
Retry attaching in case of async delays

Media toggling is handled via actions:

await hmsActions.setLocalVideoEnabled(true);
await hmsActions.setLocalAudioEnabled(false);

State is synced using selectors:

selectIsLocalVideoEnabled
selectIsLocalAudioEnabled

Persisting Preferences

Camera and microphone preferences are stored locally and reused:

During preview
During join

This creates a consistent user experience across sessions.

Real-Time Chat via Broadcast Messages

Rather than spinning up a separate WebSocket or a third-party chat service, we used 100ms's built-in messaging primitives. hmsActions.sendBroadcastMessage(text) delivers a message to every peer in the room. The SDK stores all messages in HMSStore and exposes them via the selectHMSMessages selector.

// Send a chat message
await hmsActions.sendBroadcastMessage(userInput);

// React to new messages
hmsStore.subscribe((messages) => {
  // Filter to 'chat' type only — other types are used internally (e.g. photo sync)
  const chatMessages = messages.filter(m => m.type === 'chat');
  renderMessages(chatMessages);
}, selectHMSMessages);

One important design decision: messages are typed. The default type for a regular chat message is 'chat'. We use a custom type 'PHOTO_SYNC' for the carousel sync messages described in the next section. The chat panel filters to type === 'chat' so photo-sync events never appear as chat bubbles.

Unread Message Badge

We track how many chat messages were visible the last time the panel was open (lastReadCount). When a new message arrives while the panel is closed and the sender isn't the local peer, we set an unread flag that drives a red dot on the chat button. When the panel opens, we reset the count.

Synced Image Carousel Across All Peers

The synchronized carousel required a separate state-management approach. We wanted participants to converge on the same photo when someone navigated the carousel. We built the synchronization on top of 100ms's broadcast message API, with no additional signaling infrastructure.

An important distinction is that these messages are navigation events, not durable shared state. Broadcast messaging works for transient actions such as ‘move to photo 5,’ but it should not serve as the canonical state for late joiners. If the application needs the current photo to survive missed messages or reconnects, the 100ms Session Store is a better fit for a canonical session-level state.

Defining the Sync Message Shape

We define a typed interface for the payload so that both the sender and all receivers agree on the structure:

interface PhotoSyncMessage {
  type:       'navigate_photo';
  photoId:    string; // Primary resolution: unique ID of the photo
  navigatorId: string; // which peer triggered the change
  timestamp:  number; // Diagnostic metadata (clocks may drift, do not use for ordering)
  sequenceId: number; // Monotonic counter to reject stale messages

Note on Synchronization Correctness

To prevent concurrent navigation conflicts (e.g., two users changing photos simultaneously) and distributed clock drift issues, this architecture enforces an authoritative navigator model. The presenter acts as the authoritative navigator. Navigation initiated by other participants is ignored by receivers, preventing simultaneous navigation from producing conflicting shared state. Each accepted navigation event also carries a monotonic sequence ID, allowing receivers to reject stale events.

Sending a Navigation Event

Whenever the local user taps a thumbnail or presses a carousel arrow, we broadcast a PHOTO_SYNC message alongside updating local state:

const AUTHORITATIVE_ROLE = 'presenter';
// Module-level sequence tracker 
let localSequenceId = 0;

async function handlePhotoSelected(photo) {
  // Only the authoritative navigator can initiate synced navigation 
  if (currentUser.roleName !== AUTHORITATIVE_ROLE) { return; }
  localSequenceId++;
  store.setCurrentPhotoId(photo.id);  // local update
  await hmsActions.sendBroadcastMessage(
    JSON.stringify({
      type:        'navigate_photo',
      photoId:     photo.id,
      navigatorId: currentUser.id,
      timestamp:   Date.now(),
      sequenceId:  localSequenceId
    }),
    'PHOTO_SYNC'         // custom type keeps it separate from chat
  );
try { 
  await hmsActions.sessionStore.set('canonical_photo_id', photo.id); 
} catch (error) { 
  console.warn('Failed to update canonical session state', error); }
}

Receiving and Applying Navigation Events

On the receiver side, we hook into hmsNotifications.onNotification(), filter to NEW_MESSAGE events with our custom type, parse and validate the payload, and resolve the target photo by its stable ID before updating local state.

// Track the latest applied sync sequence for the room
let lastSyncSequence = 0;
// Define which role is allowed to navigate to prevent simultaneous conflicts
const AUTHORITATIVE_ROLE = 'presenter'; 

hmsNotifications.onNotification((notification) => {
  if (notification.type !== 'NEW_MESSAGE') return;

  let payload;
  try { payload = JSON.parse(notification.data.message); } 
  catch { return; }

  if (payload.type !== 'navigate_photo') return;

  // --- 1. AUTHORITATIVE NAVIGATOR CHECK ---

// Restrict synchronization to an authoritative peer role.
// This prevents multiple participants from independently
// updating the shared navigation state.

  const senderPeer = hmsStore.getState(selectPeerByID(payload.navigatorId));
  if (senderPeer?.roleName !== AUTHORITATIVE_ROLE) {
    console.warn('Rejected navigation event from non-authoritative peer');
    return; 
  }

  // --- 2. DETERMINISTIC SEQUENCE ORDERING ---
  // Reject out-of-order/stale messages from the authoritative peer
  if (payload.sequenceId <= lastSyncSequence) {
    console.warn('Rejected stale navigation sequence');
    return;
  }

  // --- 3. STABLE ID RESOLUTION (No Index Fallback) ---
  // Match strictly by stable photoId. If lists diverge and ID is unknown, 
  // do not guess using index. Ignore event and rely on Session Store recovery.
  const targetPhoto = photos.find(p => p.id === payload.photoId);
  if (!targetPhoto) {
    console.warn('Photo ID not found in local state, ignoring event.');
    return;
  }

  // Apply state and update tracker
  store.setCurrentPhotoId(targetPhoto.id);
  lastSyncSequence = payload.sequenceId;
});

Syncing New Joiners

A participant who joins mid-session could see the carousel start at index 0 while the rest of the room has moved to another photo. For late joiners, the implementation relies on the 100ms Session Store instead of allowing multiple peers to broadcast competing states. When a user joins, they simply read canonical_photo_id from the Session Store to land on the correct image.

Persisting Media Preferences Between Screens

The user selects their camera and mic state in the Green Room. These preferences need to persist through the route change to the Live Call screen and a page refresh.

We store preferredCameraOn and preferredMicOn in a Pinia store configured with selective localStorage persistence. Only these two fields are persisted; sensitive data like the HMS auth token is kept in memory only and is re-fetched on every session.

// Pinia store — persist only media prefs
persist: {
  key:     'app-meetup',
  storage: localStorage,
  paths:   ['preferredCameraOn', 'preferredMicOn'],
}

When the Live Call screen mount, we read these two values back and call setLocalVideoEnabled / setLocalAudioEnabled immediately after join() resolves. This means the peer publishes with the camera and microphone state configured in the Green Room.

Live Session Teardown

Proper session teardown prevents stale participant and media state.

If a participant disconnects without completing session teardown, other users may see stale or ‘ghost’ participants for a short period.

We handle cleanup in three places:

Explicit leave button
Browser unload events
Component unmount lifecycle hooks

For browser lifecycle handling, do not rely on unload alone. Modern browsers may skip or limit unload processing, so lifecycle handling should account for pagehide, beforeunload, and freeze where appropriate. These events should be treated as best-effort lifecycle signals rather than a guarantee that an awaited asynchronous leave() operation will finish before the page disappears.

async function explicitTeardown() {
  unsubscribers.forEach(unsub => unsub?.());
  // For explicit leave button, we safely await
  await hmsActions.leave();
}

// For browser lifecycle (closing tab, backgrounding, freezing)
function setupBrowserLifecycleTeardown() {
  const bestEffortCleanup = () => {
    unsubscribers.forEach(unsub => unsub?.());
    // Do not await leave() during lifecycle events; it's a best-effort fire-and-forget
    hmsActions.leave().catch(console.error);
  };

  // Modern alternatives to the deprecated 'unload' event
  window.addEventListener('pagehide', bestEffortCleanup);
  window.addEventListener('beforeunload', bestEffortCleanup);
  document.addEventListener('freeze', bestEffortCleanup);
}

For an explicit leave action or route transition, the async leave() call can be awaited normally. For browser termination events, cleanup should be best-effort and should not assume asynchronous work will always complete. The application should also remove subscriptions and detach UI references during component teardown, while relying on the SDK/browser lifecycle for connection termination.

Performance and UX Considerations

1. Avoid Duplicate State

Use the reactive store instead of local component state wherever possible.

2. Lazy Rendering

Attach video tracks only when elements are mounted.

3. Optimistic UI Updates

Update the UI when the user changes a photo, then synchronize the change.

4. Graceful Error Handling

Token failures
Permission issues
Network interruptions
Reconnects and late joiners
Unknown or stale PHOTO_SYNC messages

For synchronized navigation, network interruption is especially important: a participant may miss transient broadcast messages. The design should recover from canonical session-level state when the product requires convergence after reconnect.

Synchronization Validation and Failure Cases

Before describing a real-time synchronization flow as production-ready, it is important to validate failure modes rather than relying only on the happy path. Useful acceptance checks include:

Rapid navigation: repeatedly move forward/backward and verify that older messages do not overwrite the latest accepted sequence.
Simultaneous navigation: have two participants navigate at the same time and verify that the defined authority/conflict-resolution rule produces one deterministic final state.
Late join: join after the room has already moved through several photos and verify that the participant converges to the canonical current photo rather than starting at index 0.
Reconnect: interrupt a participant's network connection, restore it, and verify that the participant recovers the current session state.
Divergent photo ordering: use stable photoId matching rather than assuming every participant's array has identical ordering.
Browser lifecycle: leave through the explicit button, route away, background/freeze the page, and close the tab to verify that subscriptions and media references are cleaned up as far as the browser lifecycle permits.

These checks make the synchronization guarantees explicit. When behavior has not been measured under a defined participant count or network condition, describe the implementation as a maintainable foundation rather than making an absolute scalability or production-readiness claim.

Extensibility Opportunities

This architecture can support additional capabilities:

Screen sharing
Role-based permissions
Recording and streaming
Collaborative annotations
Session-level shared state using the Session Store
Deterministic conflict resolution for collaborative controls

Key Takeaways

Building this feature reinforced several architectural patterns for real-time applications:

1. Centralized reactive state simplifies synchronization

Maintaining a single HMSReactiveStore instance prevents UI desynchronization and keeps media state consistent across screens.

2. Preview mode improves pre-join UX

The Green Room gives users confidence and control before entering the live session.

3. Broadcast messaging can support multiple event types

We used the same messaging layer for chat and photo-navigation events by namespacing message types. For durable shared state, such as the current photo for late joiners, session-level state is a better model.

4. Fine-grained subscriptions improve performance

Track-level subscriptions prevent unnecessary participant list re-renders.

5. Clean teardown logic matters

Explicit transitions should await leave(), while browser lifecycle cleanup must be treated as best-effort and designed around modern lifecycle events rather than assuming unload will complete asynchronous work.

Conclusion

Real-time collaboration is no longer limited to basic video calls.

Combining 100ms’s reactive SDK architecture with a structured frontend state model supports interactive shared experiences without requiring a separate synchronization layer for every feature.

The synchronized-media feature also highlights an important engineering lesson: real-time UX requires more than broadcasting an event. Correctness depends on defining the source of truth, resolving concurrent updates, rejecting stale messages, handling late joiners and reconnects, and managing cleanup across browser lifecycle states.

With those considerations in place, the patterns described here provide a maintainable foundation for collaborative experiences involving shared media, live messaging, and reactive participant management.\
As real-time applications expand beyond traditional conferencing, reactive synchronization, centralized media state, and event-driven collaboration provide a foundation for shared digital experiences.

For teams building real-time video and interactive media products, GeekyAnts provides Media & Entertainment software development services for developing and scaling digital media experiences.

Stop Automating Everything: A Balanced Quality Engineering Approach to Testing

GeekyAnts India Pvt Ltd — Mon, 12 Oct 2026 08:55:20 +0000

Balanced quality engineering places automation, API testing, exploratory work, and AI where each gives the most value, so teams ship faster without trading away user-perceived quality.

Key Takeaways

Automate by Risk: Focus automation on business-critical, high-risk workflows rather than raw test-case count, since more tests do not mean better quality.
Cheapest Effective Layer: Validate business logic through API tests where possible and reserve UI automation for the user journeys that matter most.
Keep Exploratory Testing: Human testers surface unexpected behaviour, usability gaps, and edge cases that predefined tests never anticipate.
Measure Quality, Not Volume: Track defect detection, leakage, flakiness, maintenance effort, and investigation time instead of counting automated tests.
AI as an Accelerator: Let AI assist with test generation, failure analysis, and maintenance while engineers stay responsible for quality decisions.

Why Passing QA Tests Can Still Ship Poor Experiences

Software teams are expected to release faster while maintaining reliable digital experiences. Automation has become a critical part of achieving this goal.

A typical automated regression workflow looks like:

This provides fast and repeatable feedback, but passing tests do not necessarily mean that users are experiencing a high-quality product.

For example, an automated checkout test may verify that an order is successfully created while missing:

confusing error messages,
poor mobile usability,
inaccessible controls,
unclear navigation,
unexpected user workflows.

This creates an important distinction:

Technical correctness and user-perceived quality are related, but they are not the same thing.

Moving From Test Automation to Quality Engineering

Traditional testing often follows:

A Quality Engineering approach distributes quality throughout the lifecycle:

The goal is not simply to find more bugs. It is to provide fast, reliable quality feedback at the appropriate layer.

What a Balanced Testing Model Looks Like

A balanced QE strategy can combine five capabilities:

Each layer has a different purpose.

Testing approach	Primary purpose
Unit testing	Validate isolated logic
API testing	Validate services and business rules
UI automation	Validate critical user journeys
Exploratory testing	Discover unexpected behavior
UX testing	Validate the user experience
AI assistance	Accelerate testing activities

Deciding What to Automate

Not every scenario should become a UI automation test. Strong automation candidates are generally:

repeatable,
deterministic,
frequently executed,
business-critical,
expensive to execute manually.

For example:

Login
Checkout
Payment
Order creation
Permission validation
API contracts

A useful decision process is:

The objective should not be 100% automation.

It should be meaningful risk coverage.

Validate Business Rules Through APIs, Not the UI

One common mistake is validating every business rule through the UI.

Suppose an application exposes:

POST /api/orders

The business logic can be validated directly:

const response = await request.post("/api/orders", {
  data: {
    productId: 123,
    quantity: 2
  }
});

expect(response.status()).toBe(201);

const body = await response.json();

expect(body.quantity).toBe(2);
expect(body.status).toBe("created");

The UI test can then focus on whether a user can successfully complete the workflow. This creates a useful separation:

This can reduce unnecessary UI automation and make regression suites faster and easier to maintain.

What Human Testers Catch That Automation Misses

Automation executes what engineers explicitly define.

Humans can investigate what was not anticipated.

For example, an exploratory checkout charter might include:

Explore checkout with focus on:

invalid inputs
browser back button
session expiry
duplicate submission
slow network
mobile viewport
unexpected navigation

A tester might discover:

Field	Observation
Action	Submit payment twice quickly.
Expected	One order.
Actual	Two orders created.
Risk	Duplicate order/payment.

A standard happy-path automation test might never discover this scenario. This is why exploratory testing should not disappear when automation increases.

Why a Green Suite Isn't Proof of a Good Experience

Consider:

await page.getByRole("button", {
  name: "Place Order"
}).click();

await expect(
  page.getByText("Order confirmed")
).toBeVisible();

The test proves that the order can be placed.

It does not necessarily prove that:

the button is easy to find,
the error messages are understandable,
the workflow is accessible,
the mobile experience is usable,
the customer understands the payment status.

This is where human-centered testing complements automation.

Metrics That Show Whether the Strategy Works

A balanced QE strategy should be measurable.

Instead of measuring only:

Number of automated tests

track metrics such as:

Execution time
Defects detected
Defect leakage
Flaky tests
Maintenance effort
Failure investigation time
Exploratory findings

A practical experiment can use a fixed set of application workflows:

Registration
Login
Product Search
Cart
Checkout
Payment
Order Confirmation
Invoice Download

Then evaluate them using:

UI Automation
API Testing
Exploratory Testing
UX-focused Testing

The experiment should record the same measures for each testing approach. Until real results are available, it is better to define the measurement plan rather than show placeholder values.

Metric	What to record
Execution time	Time required to complete the selected workflow or test set
Unique defects detected	Distinct defects found by each testing approach
Maintenance effort	Time spent updating or repairing tests as the application changes
Investigation time	Time required to understand and classify a failed test or observed issue
Defect leakage	Defects that escaped earlier test layers and were found later
Flaky-test rate	Intermittent automated failures that do not consistently reproduce
Exploratory findings	Unexpected behaviors, usability problems, or risk scenarios discovered through human investigation

Charts Worth Publishing Once You Have Real Data

Once the experiment has produced real measurements, two graphs can communicate the results clearly without making the article visually heavy:

Execution Time by Testing Approach — compare the measured execution time for UI automation, API testing, exploratory testing, and UX-focused testing.
Unique Defects Detected by Testing Approach — compare the number of distinct defects found by each approach, ideally with defect categories called out where useful.

Do not publish these charts with illustrative or fabricated values. Generate them only from the completed experiment and label the units, sample size, and test conditions so readers can interpret the results correctly.

These graphs should reinforce an important point:

Different testing approaches detect different categories of risk.

Keeping Automation Reliable Enough to Trust

Automation becomes less valuable when teams stop trusting test results.

A test that behaves like this:

Run 1 → PASS
Run 2 → PASS
Run 3 → FAIL
Run 4 → PASS
Run 5 → PASS

is potentially flaky.

Avoid solving this by adding arbitrary waits:

await page.waitForTimeout(5000);

Instead, synchronize with application state:

await expect(
  page.locator('[data-testid="order-confirmation"]')
).toBeVisible();

The principle is:

Wait for conditions, not time.

Automation should be treated as production-quality software, with code review, maintainability, monitoring, and refactoring.

Where AI Helps — and Where It Shouldn't Decide

AI can extend the QE workflow without replacing deterministic testing.

A practical architecture is:

For example, AI can analyze:

Playwright Trace
Screenshot
DOM
Console Logs
Network Errors
Test History

and suggest a failure category:

Test defect
Application defect
Environment issue
Data issue
Network issue

However, the engineer should validate the result.

A useful operating model is:

This keeps AI as an accelerator rather than an uncontrolled decision-maker.

Fitting the Approach Into Your CI/CD Pipeline

The balanced approach fits naturally into a modern CI/CD pipeline:

Different test layers provide feedback at different speeds, allowing teams to avoid using expensive UI tests for problems that can be detected earlier.

Trade-Offs Between Testing Approaches

There is no single testing technique that covers every quality dimension.

Approach	Strength	Limitation
UI Automation	End-to-end regression	Maintenance cost
API Testing	Fast business validation	Doesn't validate UX
Exploratory	Finds unexpected issues	Less repeatable
UX Testing	User experience	Requires human effort
AI Testing	Faster analysis/generation	Requires validation

The right strategy combines these capabilities according to product risk.

A Reference Architecture You Can Adapt

The complete model can be summarized as:

This model does not attempt to eliminate human testing.

Instead, it places automation, APIs, AI, and human reasoning where each provides the most value.

Conclusion: Getting the Balance Right

Modern Quality Engineering should not be a competition between manual testing and automation.

Automation provides speed, repeatability, and scalable regression coverage. API testing provides efficient validation of business behavior. Exploratory and human-centered testing provide contextual reasoning and help identify problems that deterministic automation may miss. AI can further accelerate these activities by assisting with generation, analysis, and maintenance.

The most effective Quality Engineering strategy is one that uses technology to accelerate testing without losing sight of the person ultimately using the software.

The goal is not maximum automation. The goal is maximum meaningful quality feedback for the engineering effort invested.

From Rolling Deployments to Zero-Downtime Releases

GeekyAnts India Pvt Ltd — Wed, 07 Oct 2026 09:17:22 +0000

Executive Summary
In digital banking, deployment availability is a business requirement as much as a technical one. A deployment that interrupts access to online banking can affect customers attempting to log in, view balances, initiate transfers, or perform other critical operations.

Our earlier rolling deployment model introduced problems around version coexistence. Every new frontend build generated build-specific resources such as runtime.1234.js and main.1234.js. During a rolling restart, old and new pods could coexist. A request for a resource from the new build could reach an older pod where that resource did not exist, producing a 404 response.

Static resources were cached at the CDN for about four minutes. We needed to consider application availability and version consistency across the application shell, static resources, CDN, and backend pods.

We moved to a Blue-Green deployment strategy. The new version is deployed into a separate environment, its pods start and pass readiness checks, and the environment is validated before customer traffic is switched. The existing environment continues serving production traffic until the cutover.

In our environment, the observed ALB traffic transition from Blue to Green was about 40–180 ms. The key benefit is not faster application startup. Pod creation, image pulling, startup, and readiness occur outside the customer-facing cutover window. The previous environment also remains available so that traffic can be routed back to it if a rollback is required.

A race condition remained during the ALB transition under high request volumes. A request for a build-specific static resource could reach the previous environment and receive a 404 response. We addressed this with a targeted NGINX retry for the affected static resources, with a delay of about one second before the retry.

Why We Needed a Different Deployment Strategy 1.1 The Rolling Deployment Model A rolling deployment replaces application instances in stages. This avoids replacing every instance at once, but it also means different application versions can coexist during the deployment.

1.2 Build-Specific Static Resources
Every application build generates resources associated with that build. For example:

Build 100:
runtime.100.js
main.100.js
styles.100.css

Build 101:
runtime.101.js
main.101.js
styles.101.css

The browser receives an application shell that references a particular build's resources. If a request for runtime.101.js reaches a Build 100 pod, the resource does not exist there, and the server can return a 404 response.

The pod itself could be healthy. The failure was caused by a version mismatch between the resource requested by the client and the application instance handling the request.

1.3 CDN Caching Added Another Timing Dimension
Static resources were cached at the CDN for about four minutes. The application origin could move to a new version while cached resources from the previous version remained available at the edge.

Deployment correctness had to account for three independent timing domains: application and pod readiness, ALB traffic convergence, and CDN cache lifetime.

Blue-Green Deployment Architecture Blue-Green deployment introduced two complete application environments. Blue is the active version, and Green is the new version.

The principle is simple: deploy and validate Green before allowing it to receive production traffic. This separates deployment preparation from production traffic cutover.

What Happens During a Blue-Green Deployment 3.1 Build and Image Creation The CI/CD pipeline creates an immutable image for the release. The frontend resources generated by that build are part of the same release artifact.

This preserves a clear relationship between the application version and the resources it serves.

3.2 Creating the Green Pods
The new release is deployed to Green without terminating the active Blue environment.

The Green pods go through image pull, container startup, application initialization, and readiness checks before they become eligible to receive traffic.

3.3 Pod Readiness
Readiness is a key mechanism in this architecture. Kubernetes distinguishes a running container from a container that is ready to accept traffic. A failed readiness probe causes the pod to be treated as not ready for normal service traffic. A startup probe can be used when an application needs more initialization time before liveness or readiness checks begin.

For a banking application, readiness should represent the minimum state in which the application can process production requests without creating service risk. Depending on the implementation, this can include application initialization, configuration loading, dependency availability, and other required startup conditions.

3.4 Green Validation
Once the Green pods are Ready, the new environment can be validated before customer traffic is moved.

Confirm that the pods and application are healthy.
Run API smoke tests against the new environment.
Confirm that the required static resources are available.
Check connectivity to critical dependencies.
Test business-critical flows.
Review application and infrastructure metrics.
This allows the team to validate Green while Blue continues serving production traffic.

How the ALB Traffic Switch Works The Application Load Balancer (ALB) provides the boundary for switching production traffic between Blue and Green. An ALB listener evaluates routing rules and forwards traffic to target groups. Target groups contain registered targets and can have their own health checks.

Before the cutover, Blue is the active target group and Green is prepared without receiving normal production traffic. Once Green has passed the deployment gates, the routing configuration changes so that production requests are directed to Green.

The Critical Difference: Deployment Time vs Customer-Impact Time A key distinction in Blue-Green deployment is the difference between deployment time and customer-impact time. The new pods still need to pull the image, start the container, initialize the application, and pass readiness checks.

What changes is where that work occurs.

Blue continues serving customers while Green is being prepared. This removes the time-consuming preparation work from the customer-facing cutover path.

The customer-facing transition is limited to the routing window rather than the full duration required to create and warm the new pods.

The improvement should not be described as “the deployment became X% faster.” The deployment preparation time still exists, but it does not need to translate into customer downtime.

Rollback: Turning Recovery Into a Traffic Switch Blue-Green changes the rollback mechanism.

Normal state:

BLUE = Build 100
GREEN = Build 101
Traffic = 100% Green
If monitoring identifies a production problem after the cutover, Blue can remain available as the previous known-good version.

Instead of rebuilding and redeploying the previous image, the deployment can route traffic back to the running Blue environment.

This changes rollback from a deployment operation into a traffic-routing operation, which can reduce mean time to recovery (MTTR).

The Production Challenge After Blue-Green 7.1 The ALB Transition Window Blue-Green solved the major version-coexistence problem, but production testing exposed a smaller race condition during the traffic switch.

In our environment, the observed ALB transition from Blue to Green was about 40–180 ms.

Before:
Blue = 100%
Green = 0%

Transition:
Blue = 100% -> 0%
Green = 0% -> 100%

After:
Blue = 0%
Green = 100%
Under high request volume, requests can arrive during this short interval. A request that arrives during the transition can encounter the previous routing state.

7.2 Static Resource 404 During the Transition

This was not a persistent application failure. It was a transient race condition caused by the interaction between build-specific resource names and traffic convergence.

NGINX Retry as the Last-Mile Resilience Layer To address this transient condition, a targeted retry mechanism was introduced at NGINX for the affected static resources.

The one-second delay was longer than the observed ALB transition window. This gave the traffic transition time to complete before the retry.

This mechanism was scoped to static-resource requests. A generic retry policy should not be applied to state-changing banking operations such as fund transfers without idempotency and transaction-safety guarantees.

CDN and Resource Versioning Build-specific resource names remain part of the design. Instead of overwriting a stable path such as /main.js, each build produces a unique resource path.

/main.100.js
/main.101.js
/main.102.js
This allows multiple versions of a resource to coexist in the CDN. The application shell should reference the resource set belonging to its build.

The CDN cache lifetime of about four minutes is treated as a separate concern from the ALB transition. Build-specific resources provide coexistence, while the NGINX retry addresses the short traffic-convergence race.

Zero-Downtime as an End-to-End Property The implementation showed that zero-downtime (ZDT) is not a property of Kubernetes or the load balancer alone.

A system can have healthy pods and still return customer-visible errors if resource versioning, caching, or routing transitions are not handled correctly.

Our final approach combines several mechanisms:

Blue-Green deployment
+
Pod readiness
+
Build-specific static resources
+
CDN caching strategy
+
ALB traffic cutover
+
NGINX static-resource retry
+
Fast rollback
=
Zero-Downtime Release Architecture

Business and Operational Impact Before the move toward ZDT, deployments were scheduled during low-traffic periods such as night-time maintenance windows. This was a risk-management decision. If a deployment could cause service interruption, a period with fewer customers using the platform reduced the number of users exposed to that interruption.

Before:
Release Ready
|
v
Wait for Night
|
v
Maintenance Window
|
v
Deployment

Blue-Green changed the release model.

After:
Release Ready
|
v
Deploy Green
|
v
Validate
|
v
Switch Traffic

Because Blue remains available while Green is prepared, a new deployment does not require the application to be taken offline. Subject to change-management, monitoring, and business controls, teams can perform deployments during normal operating hours, including the morning, instead of restricting them to night-time maintenance windows to avoid deployment downtime.

This changes the operational model: application availability no longer dictates the deployment window.

Deployment Sequence
Developer merges release |
CI/CD builds application |
Build-specific static resources generated |
Immutable container image created |
Image pushed to registry |
Green deployment created |
Green pods start |
Startup / initialization |
Readiness probes pass |
Green validation / smoke tests |
Green target group confirmed healthy |
ALB routing switched |
Traffic moves Blue -> Green |
Monitor application + business metrics |
Keep Blue available for rollback |
Decommission Blue after confidence window
Key Engineering Lessons
13.1 Blue-Green Requires Coordinated Deployment Controls
Blue-Green depends on the interaction between deployment orchestration, readiness, traffic management, resource versioning, CDN behavior, observability, and rollback. Two application environments alone do not address the dependencies across the release path.

13.2 Move Preparation Work Outside the Cutover Path
Image pulling, pod creation, startup, and readiness can take minutes. Blue-Green allows the previous version to continue serving traffic while the new environment completes this work.

13.3 Rollback Should Be Designed Up Front
Keeping the previous environment available allows the system to route traffic back to a known-good version without rebuilding and redeploying that version.

13.4 Short Transition Windows Can Expose Race Conditions
A short routing transition can still expose a race condition when request volume is high. Deployment design must account for requests that arrive while traffic moves between environments.

13.5 Static Resources Are Part of the Release Architecture
Frontend resources form part of the production deployment architecture. Their versioning, CDN lifetime, and relationship with the application shell must be considered as part of the release design.

Final Architecture

Conclusion The move from rolling deployments to Blue-Green deployment addressed the version mismatch that could occur when build-specific frontend resources reached pods running an older build. Blue-Green separates release preparation from traffic cutover by allowing the new environment to complete startup, readiness checks, and validation while the existing environment continues serving customers.

The implementation showed that zero-downtime deployment depends on controls across the release path. Resource versioning, CDN caching, traffic switching, readiness checks, retry behavior, and rollback design must work together to protect customer requests during a release.

Keeping the previous environment available also allows recovery through a traffic switch instead of another deployment. This release model reduces the need to schedule deployments around maintenance windows while preserving a path back to the previous version.

Building Release Architectures for Production Reliability
Zero-downtime deployment depends on more than switching traffic between two environments. Pod readiness, static-resource versioning, caching, traffic routing, monitoring, and rollback must work together to protect customer-facing applications during a release. For teams building or modernizing applications with these production requirements, GeekyAnts provides custom software development services covering architecture, CI/CD, deployment, and post-launch support.

References
Kubernetes: Liveness, Readiness, and Startup Probes
https://kubernetes.io/docs/concepts/workloads/pods/probes/
AWS Elastic Load Balancing: Application Load Balancer
https://docs.aws.amazon.com/elasticloadbalancing/latest/application/introduction.html
AWS Elastic Load Balancing: Target Groups
https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-target-groups.html
AWS Elastic Load Balancing: Listener Rules
https://docs.aws.amazon.com/en_en/elasticloadbalancing/latest/application/listener-rules.html

The Deprecation Notice Nobody Reads

GeekyAnts India Pvt Ltd — Wed, 07 Oct 2026 08:49:51 +0000

This blog explains why API deprecation notices often fail and how staged retirement, telemetry, brownouts, and scheduled enforcement can make API migrations more controlled and predictable.

Every API team eventually writes the same message

"This endpoint will be removed on [date]. Please migrate before then." It goes in a changelog, a blog post, maybe an email. Traffic to the deprecated endpoint can show little movement until the day it is switched off, at which point support queues fill up with people who swear they never saw the notice.

They may not have seen it. A warning with no consequence attached is easy for a busy team to defer. The problem with sunset timelines is not communication. Most contain nothing that creates a cost for delay until the exact day they stop being a warning and become an outage.

The flat line before the cliff

The pattern shows up across API providers: deprecation is announced, documentation is updated, and usage of the old endpoint barely dips. One cited account of a typical deprecation window reported that six months after announcing a sunset date, the old version was still carrying 40% of total traffic, with a team left asking why nobody had migrated.

A deprecated endpoint that still returns 200 OK carries no urgency. Migration work competes with every other roadmap item, and "the thing that still technically works" reliably loses that competition until it stops working at all. A blog post is not an effective notice if ignoring it carries no immediate consequence.

Two timelines, two outcomes

Timeline	Without a strategy	With a strategy
Month 0	Endpoint removed, no warning	Deprecation announced; Deprecation/Sunset headers added; migration guide published
Month 3	47 integrations already broken	Migration reminders sent to consumers still calling the endpoint
Month 6	Support queue flooded	First announced brownout window, a scheduled, published maintenance-style window during which the endpoint is intentionally degraded
Month 9	Emergency rollback underway	Quota tightened for remaining consumers; brownout windows lengthened per the published schedule
Month 12	Trust damaged, repeat incidents	Endpoint retired, returns 410 Gone; near-zero breakage

What changes structurally is that, by month 6, the endpoint behaves differently for consumers who have not migrated. That change is published on a schedule before it takes effect rather than discovered by consumers in production.

Why longer notice periods don't fix this alone

Some major API providers use deprecation or version-support windows measured in years, giving customers time to include migration work in planning cycles.

But two years solves a planning problem, not an urgent problem. It does nothing to make migration feel pressing in month three, nine, or eighteen because nothing about the API has changed yet. The calendar moved; the consequence didn't.

Staged, announced retirement

The fix isn't a louder notice, and it isn't quiet degradation either. It's a retirement plan the API enforces on a schedule the consumer knows about. Every stage is disclosed in the changelog and response headers before it takes effect rather than discovered in production.

A staged retirement can follow five stages:

Headers-only window. Deprecation and Sunset headers present on every response. No behavioral change. This window should be long enough to cover at least one full planning cycle for your consumers.
Warnings surfaced in-band. A deprecation notice is added to response bodies, or a Link header points to the migration guide, so tooling rather than memory catches it.
Announced brownout windows. Published, scheduled windows (e.g., "the deprecated endpoint will return elevated latency and occasional 429s every Tuesday 2–4pm UTC starting date") during which the endpoint behaves like it will after full retirement. These are calendared and documented before they start, exactly like a maintenance window.
Progressively tighter quotas for remaining consumers, identified by telemetry, not by chance.
Hard retirement. 410 Gone, with the migration link still present.

At every stage, a consumer hitting the new behavior can look at a public schedule and confirm exactly which stage is in effect and when the next one begins. Nothing about it is a surprise, and nothing about it is silent.

What changes for internal versus public APIs

For an internal API with a handful of known consuming teams, the friction curve can be steep and communication can be direct. A Slack message plus a dashboard showing which service is still calling the old path can help get a fix shipped within a sprint. The consuming teams are known, and a sharper cutoff is more practical because the organization also controls the fallback.

Public and partner-facing APIs are the harder case. The consumer may be unknown until they show up in usage telemetry, may not follow the changelog, and may be running an integration nobody at their organization has touched since it was built. That is where headers, published schedules, and graduated enforcement provide value. They turn an anonymous, unreachable consumer's client code into something that reports its own deprecation status back to whoever looks at the logs.

Comparing three retirement strategies

	Communication-only	Hard sunset	Staged enforcement (brownouts)
What it is	Changelog/email notice, no behavior change until removal day	Fixed date, endpoint simply stops working	Published schedule of increasing, announced friction before final removal
Best fit	Low-risk internal APIs, few known consumers	Emergency/security-driven deprecations	Public and partner APIs with unknown or numerous consumers
Retry-storm risk	Low until cutover, spikes hard at deadline	High; everyone fails at once	Distributed across scheduled windows; retry behavior can be observed before it's catastrophic
SLA / contractual exposure	Rarely satisfies partner deprecation clauses alone	Highest exposure if the date arrives early	Lower; each stage documented and pausable
Rollback strategy	Trivial (nothing changed yet)	Difficult; re-enabling under load is its own incident	Straightforward; each stage is a feature flag
Operational cost	Low upfront, high at cutover	Low upfront, very high at cutover	Higher upfront, spread more evenly
When it fails	Consumers defer migration	Consumers who missed the notice get zero warning	Poorly tuned severity or cadence can trigger retries without Retry-After and backoff

No single strategy is correct in isolation. The choice depends on how well you know your consumers, what your contracts require, and how much operational machinery you're willing to build before you start the clock.

Standards-correct signals

RFC 9745 defines Deprecation as a Structured Fields Date, not an HTTP-date string. That means the value should look like @1735689600 (an @-prefixed Unix timestamp), not the output of toUTCString(). Sunset (RFC 8594) does use an HTTP-date. Mixing the two formats can break client tooling. A malformed value may fail to parse as expected, which can disrupt automated deprecation tracking.

Production-oriented middleware, with per-consumer rollout, a rollback switch, and telemetry hooks built in:

// Formats a JS Date/timestamp as an RFC 9745 Structured Field Date: "@" function toStructuredFieldDate(msTimestamp) { return @${Math.floor(msTimestamp / 1000)}; }

function deprecationMiddleware({ deprecationDate, sunsetDate, migrationLink, stages, // ordered array of { startsAt, mode, config } isEnforced, // (consumerId) => bool — per-consumer rollout / allowlist telemetry, // { recordHit(stage, consumerId), recordOutcome(stage, consumerId, result) } killSwitch, // () => bool — global rollback }) { return function (req, res, next) { const now = Date.now(); const consumerId = req.consumerId; 

res.setHeader('Deprecation', toStructuredFieldDate(deprecationDate));
res.setHeader('Sunset', new Date(sunsetDate).toUTCString());
res.setHeader('Link', `<${migrationLink}>; rel="deprecation"`);

telemetry.recordHit('headers', consumerId);

if (killSwitch()) {
  return next(); // incident-response rollback: headers stay, no enforcement
}

const activeStage = stages
  .filter((s) => now >= s.startsAt)
  .sort((a, b) => b.startsAt - a.startsAt)[0];

if (!activeStage || !isEnforced(consumerId)) {
  return next();
}

switch (activeStage.mode) {
  case 'retired': {
    telemetry.recordOutcome('retired', consumerId, '410');
    return res.status(410).json({
      error: 'gone',
      message: 'This endpoint was retired. See the Link header for the migration guide.',
    });
  }
  case 'brownout': {
    const { rejectRatio, retryAfterSeconds } = activeStage.config;
    if (Math.random() < rejectRatio) {
      telemetry.recordOutcome('brownout', consumerId, '429');
      res.setHeader('Retry-After', String(retryAfterSeconds));
      return res.status(429).json({
        error: 'deprecated_brownout',
        message: `Scheduled brownout in effect per the published retirement schedule. See ${migrationLink}.`,
      });
    }
    telemetry.recordOutcome('brownout', consumerId, 'pass');
    return next();
  }
  case 'quota': {
    req.deprecationQuotaTag = activeStage.config.quotaKey;
    return next();
  }
  default:
    return next();
   }
 }; 
}

Stages here are explicit, scheduled, published data rather than a continuous random function tied to elapsed time. Enforcement is per-consumer and gated by an allowlist so a rollout can ramp in stages instead of switching on for every consumer at the same time. A kill switch supports incident response, and every enforcement decision is recorded to telemetry so it can feed the decision loop below.

Telemetry as the backbone

"Track usage per consumer" isn't a strategy on its own. The strategy is the decision loop that telemetry feeds:

Identify every consumer still hitting the deprecated path, from API keys/auth identity, not just aggregate request counts.
Segment by risk and traffic. A partner integration processing payments and a dormant internal script hitting the endpoint twice a day need different handling and different owners.
Notify owners, not just via the changelog. The segmented list makes targeted outreach possible.
Measure migration after each stage, before deciding to advance. The gate for moving to the next stage should be a measured drop in remaining traffic plus confirmed outreach, not just a date on a calendar.
Enforce the next stage only for consumers who haven't moved. Scope enforcement to the remaining, already-notified cohort rather than applying it to everyone the moment a date arrives.

This loop is also what tells you when to pause: if a critical, contractually-protected partner is still on the old path with an active migration in progress, the gate holds them back from the next enforcement stage while the rest of the cohort proceeds.

Testing the strategy before shipping it

Before rolling out a staged retirement for a public or partner API, test the full sequence of headers, warnings, one brownout window, quota tightening, and retirement against a low-traffic internal endpoint with a small, known set of consumers. Compress the sequence into a shorter window than a real public deprecation would use.

What to Capture at Each Stage

Request volume to the deprecated path, before and after each stage transition
Number of distinct consumers still active, and how that count changes after outreach vs. after the first brownout
Retry behavior during the brownout window, including whether clients back off per Retry-After or retry at a rate that begins to resemble an attack
Support/Slack messages generated per stage
Time from "notified" to "migrated" per consumer, segmented by whether they moved after notice alone or only after a brownout

The test report should show the share of consumers who migrated after headers and direct outreach, the share who required a brownout window, the number of integrations requiring one-on-one follow-up, and any unexpected brownout behavior.

What actually breaks

Staged retirements do not fail cleanly, so teams need to plan for the ways they can fail.

A brownout window can trigger a retry loop that amplifies load instead of shedding it if clients do not respect Retry-After. That is why the header has to be present on every throttled response, not just the first.

A batch job can fail without drawing attention if it does not check for 429 at all. The problem may surface when someone notices missing data downstream.

A dormant integration with no listed owner can remain invisible until an enforcement stage breaks it. That is why the consumer registry needs an ownership field checked before enforcement begins, not after.

None of these are arguments against staged enforcement. They are arguments for building the telemetry and allowlist machinery described above before turning any stage on. If something behaves in an unexpected way, it should show up in a dashboard instead of a support ticket.

Why this matters now

API environments now include internal microservice contracts, partner integrations, and a growing layer of AI agents and tools that can call APIs without a person reading a changelog. A deprecation strategy that depends on a person noticing an email doesn't scale to that world. One built into the protocol through headers, a published schedule, and telemetry-gated enforcement does.

Sunset timelines don't fail because nobody announced them. They fail because, for most of the window, nothing about using the deprecated path changed. The fix isn't writing a better warning, and it isn't making things worse without notice until the deadline. It's a retirement plan the API keeps, announced one stage at a time. By the retirement date, consumers should know it is coming because they have seen it on the schedule, not because they experienced the change without warning.

Build APIs for Change, Not Just Launch

API reliability extends beyond keeping an endpoint available. Teams also need a plan for how APIs evolve, how consumers migrate, and how older versions are retired without creating avoidable production failures. Staged retirement, telemetry, and controlled enforcement make deprecation part of the API lifecycle rather than a last-minute cutoff. Explore how GeekyAnts approaches scalable systems and API architecture through its Backend Engineering capabilities.