DEV Community: GeekyAnts India Pvt Ltd

The Hidden Complexity Behind GitOps’ “Push to Deploy”

GeekyAnts India Pvt Ltd — Fri, 13 Nov 2026 11:39:57 +0000

A GitOps platform that provisioned isolated customer environments straight from Git, and the four problems that surfaced once it was running: dependencies that weren’t ready when resources existed, configuration changes that broke existing tenants, successful deploys that hid broken integrations, and resource limits that blocked work the cluster could handle. The piece documents each fix, the ~110-second provisioning run, and is careful about which numbers are measured observations and which are calculations.

What Building a GitOps Platform Actually Taught Us About Reliable Infrastructure

Getting an application into Kubernetes was the easy part. Making sure its database was ready, its configuration pointed to the right environment, its secrets existed, and its deployment could recover from failure took considerably more work.

We spent roughly two weeks and 194 commits building a GitOps platform that provisioned isolated customer evaluation environments directly from Git. Adding a client configuration could trigger the creation of an entire environment without an engineer manually applying Kubernetes resources.

The provisioning flow worked. Operating it exposed a different set of problems: migrations ran too early, applications started before their dependencies were reachable, configuration changes broke existing environments, and deployments could get stuck waiting for the very changes needed to fix them.

These weren't problems with GitOps itself. They exposed the assumptions underneath our automation.

This is what building the platform taught us about making infrastructure provisioning reliable, and what our production observations and resource analysis revealed about its limitations.

TL;DR

We built a GitOps platform in roughly two weeks, using ApplicationSets, Argo CD, and Helm, to provision isolated customer evaluation environments.
A documented cold-provisioning run reached Synced+Healthy in about 110 seconds.
Production incidents exposed a Redis/NATS startup race that left readiness checks returning HTTP 503 for more than three minutes across two client environments.
We added a TCP-connectivity gate across 23 Deployments, along with explicit deployment ordering and changes to migration recovery.
A historical CPU observation reported 39m used against 1,230m reserved, though the original record carries an unresolved inconsistency.
Recalculating from the current Helm chart and live node capacity gives a projected ~10.2 CPU-bound or ~10.9 memory-bound environments per node.

The central lesson: declaring the desired state is only part of reliable GitOps. Dependencies, configuration compatibility, resource allocation, and recovery behavior matter just as much.

The Platform’s Numbers: Build, Failures, Capacity

Six figures frame the build and what operating it revealed. Each one is a historical observation or a calculation, not a repeated benchmark, and the sections that follow explain where each came from and what it does and does not prove.

Metric	Result
Initial platform build	~2 weeks
Engineering activity	~194 commits
Cold provisioning	~110 seconds
Readiness failures	HTTP 503 for over 3 minutes
Deployments using the dependency-readiness gate	23
Projected capacity	~10 CPU-bound or ~11 memory-bound environments per node

The provisioning time and CPU figures are historical observations, not repeated benchmark results. The capacity figures are calculations based on current resource requests and node allocatable capacity, not results from a concurrent load test.

The Original Provisioning Design

The requirement was straightforward: adding a client to Git should provision an isolated evaluation environment without an engineer manually creating Kubernetes resources.

Each environment needed its own application stack, supporting services, configuration, secrets, and HTTPS. Some prospects retained their environments for weeks and occasionally used real data, making isolation and persistence as important as provisioning speed.

We built the workflow around three components:

ApplicationSets generated Argo CD Applications from each client’s configuration, so one config entry became one environment definition.
Helm rendered the actual Kubernetes resources from that definition, turning templates into concrete manifests.
Argo CD reconciled the cluster toward the desired state declared in Git, applying whatever the manifests described.

The provisioning flow looked like this:

This removed manual provisioning, but it didn't guarantee that dependencies were ready, configuration changes were compatible with existing tenants, or a running application was functionally correct.

Those gaps became apparent as we operated the platform. They fall into four areas, and the rest of this article works through them in order.

1. Ordering the Resources Was Not the Same as Them Being Ready

Getting the resources to come up in the right order turned out to be easier than getting each one to actually be usable when the next one needed it. Two separate failures made that distinction clear, and a third showed that recovery had the same problem.

Database Migrations Ran Before the Database was Ready

Our initial database migration used a pre-install hook that assumed the database would already be available. Sometimes the database was still starting when the migration tried to connect. The resource existed, but that did not mean it was accepting connections.

We replaced the implicit, hook-based ordering with explicit Argo CD sync waves, which run groups of resources in a numbered sequence rather than all at once:

# Dependencies
argocd.argoproj.io/sync-wave: "0"

# Migrations and initialization
argocd.argoproj.io/sync-wave: "1"

# Application workloads
argocd.argoproj.io/sync-wave: "3"

This established the intended sequence: dependencies first, migrations and initialization next, and application workloads afterward.

However, sync waves control resource ordering; they don't independently guarantee that a database is ready to accept connections. Argo CD's progression also depends on resource health assessment and hook behavior.

A database can exist and still be unavailable. Resource creation and dependency readiness are different conditions.

Redis and NATS Introduced Another Startup Race

We encountered a more persistent version of this problem in the knowledge-service and parsing-service.

Our production incident notes document two client environments where these services encountered ECONNREFUSED while connecting to Redis and NATS. Their readiness endpoints continued returning HTTP 503 for more than three minutes.

The immediate workaround was to delete the affected pod manually.

The important detail was that sync waves had already placed Redis and NATS before the application workloads. Ordering alone had not prevented the race.

The problem exposed two separate concerns:

The application could start before its dependencies accepted connections.
The application's recovery behavior did not restore readiness as expected after the failed connections.

We introduced a wait-for-redis-nats init container across 23 Deployments. It waits for TCP connectivity before allowing the application container to start.

The distinction matters: sync waves control ordering; startup guards check connectivity.

Neither replaces functional validation. An open TCP port doesn't guarantee that a database can execute queries or that a broker can process messages. The init-container gate addresses startup timing, while application-level recovery must handle dependencies becoming unavailable after startup.

The failure and mitigation can be understood as two different startup paths:

Deployment ordering and dependency readiness address different failure modes. Sync waves establish resource order; the init-container gate waits for TCP connectivity before the application starts.

What the Evidence Establishes

The production incidents demonstrate that the startup race occurred and that manual intervention was needed. The implementation record documents the subsequent connectivity gate.

They do not provide a controlled before-and-after comparison, repeated test runs, or a measured change in deployment success rate. We therefore treat this as an operational incident and an implementation change, rather than a quantified reliability benchmark.

That distinction matters when evaluating infrastructure improvements: a real failure and a plausible fix are valuable engineering evidence, but they are not the same as a controlled experiment.

Migration Jobs Needed a Safe Recovery Strategy

Existing Kubernetes Jobs couldn't always be updated and rerun in the way our deployment workflow required.

For affected Jobs, we used Argo CD's Replace=true and Force=true sync options to allow replacement and recreation rather than relying on ordinary updates.

Force replacement can delete and recreate a Job, causing its work to execute again. For database migrations, that is safe only when the operation is repeatable or has a deliberate recovery procedure.

These flags don't make a migration idempotent. They change how resources are managed, not whether the work they perform can safely run twice.

The lesson was that deployment recovery must account for the work a resource performs, not just whether Kubernetes can recreate it.

Ordering and recovery dealt with resources coming up in the wrong sequence. The next set of failures came from the opposite direction: changes that applied cleanly and still broke environments that were already running.

2. A Configuration Change Could Break Existing Tenants

A change that worked perfectly on a fresh environment was not guaranteed to work on an old one. The platform had tenants that predated the change, and three separate incidents showed how a clean-looking update could still reach back and break them.

The Frontend Artifact Contained the Production API Endpoint

We built the frontend centrally, with the production API endpoint embedded in the generated JavaScript. The endpoint appeared 14 times across seven files in the build artifact.

Reusing that artifact in a customer environment could direct requests toward production even when the Kubernetes deployment appeared healthy.

Rebuilding the frontend for every client was cleaner, but each build took several minutes and conflicted with our roughly two-minute boot target.

We introduced a startup-time replacement mechanism that substituted the expected production endpoint with the client-specific endpoint.

The startup script first checked that the expected production string existed. If it didn't, the container refused to start.

This was a fail-closed workaround, not a complete security control. It protected against the expected string being absent, but couldn't prove that every request used the correct endpoint or rule out other hardcoded endpoints.

Runtime configuration or a properly environment-specific build would be a stronger long-term design.

The incident reinforced an important distinction: a healthy deployment doesn't prove that the application is communicating with the correct environment.

Missing Secrets Caused Nine Deployment Deadlocks

A chart change introduced a required secret or configuration value that existing client environments didn't all have.

Affected pods entered CreateContainerConfigError before the application started, leaving no useful application logs to inspect.

Argo CD waited for the workload to become healthy. The current sync could remain stuck while the fix was waiting in a later Git revision, preventing the normal deployment workflow from applying the change needed to resolve the problem.

We encountered this pattern nine times.

We changed our deployment discipline: whenever a chart introduced a new required secret key, we updated existing client configurations in the same commit. We used ./scripts/reseal-all.sh to reseal the key across existing tenant configurations.

This made the chart change and its required configuration part of the same change set.

A chart that works on a fresh installation can still break every existing tenant, so backward compatibility had to become part of the deployment process.

Tenant Isolation Required Deliberate Trade-offs

We chose a namespace-per-client model, with separate application workloads and supporting services such as databases and caches.

We kept cleanup manual because deleting an environment during an active customer evaluation was considered worse than paying for an unused one. The trade-off was that abandoned environments could continue consuming resources.

Later, we separated development and production at the cluster level. Namespace separation alone wasn't enough to protect production from an operator accidentally running a destructive command against the wrong cluster.

Namespaces remained one layer of isolation, not a complete security boundary. RBAC, secret access restrictions, network policies, and other controls still needed to match the threat model.

3. A Successful Deployment Did Not Guarantee a Working Integration

Kubernetes can report a workload as healthy and Argo CD can report it as synced while the thing the workload is supposed to connect to is still broken. As the platform grew past 20 services, the gap between “deployed” and “actually working” became its own class of problem.

Certificate Issuance Became a Bottleneck

Initially, we issued certificates per client. As environment provisioning became more frequent, certificate authority rate limits became a constraint.

We moved first to a manually issued wildcard certificate, then to automated wildcard issuance using cert-manager with DNS-01 validation. Certificate material was propagated into individual client namespaces.

Each step traded one risk for another. The manual wildcard certificate eased the immediate issuance bottleneck but introduced renewal risk, since a human had to remember to renew it. Moving to cert-manager automated renewal and cut the manual work, while adding dependencies on the certificate controller, the DNS automation, its credentials, and secret propagation.

Reliable certificate management takes more than successful issuance. Expiration, renewal status, propagation, and the certificate actually served to the client all have to be accounted for.

Deployed Integrations Weren't Always Working Integrations

As the platform grew to more than 20 services, we encountered problems that weren't obvious from deployment status.

One configuration path could request broader Google permissions than intended. In another case, credentials existed but weren't wired into the service that needed them.

We changed calendar and Gmail permissions to be opt-in per client and became more deliberate about verifying that credentials were connected to the components that depended on them.

Three states are easy to conflate:

The credential exists.
The service is running.
The feature works as intended.

Each requires its own validation. Kubernetes health and Argo CD synchronization are useful signals, but they aren't substitutes for functional checks of external integrations.

The broader lesson was that configuration must be traced all the way from the source of truth to the component that consumes it.

4. Resource Allocation Was Limiting Capacity Unnecessarily

The cluster had CPU to spare and was still blocking work. The limits we had set to keep any one client in check were doing their job too aggressively, and fixing that meant first understanding what the environments actually consumed, then recalculating capacity from the current configuration rather than an old assumption.

Namespace Quotas Blocked Migrations despite Available CPU

We introduced namespace quotas to prevent one client environment from consuming too much of the cluster.

Once several services were running in a namespace, that quota started blocking migration Jobs, even though actual CPU consumption looked low against the reservations.

A historical resource observation recorded approximately 39 millicores of CPU usage against 1,230 millicores reserved. The original record did not preserve the raw command output, an exact measurement timestamp, or repeated samples. It also contained an unresolved inconsistency, with another sentence referring to 1,225m out of 4.

We retain the 39m and 1,230m figures as historical observations rather than treating them as a validated benchmark.

We reduced CPU requests to better reflect observed consumption, removed the namespace quota that was blocking migrations, and retained per-container limits to constrain individual workloads. The configuration included a maximum of four CPUs and 4 GiB of memory per container.

The observation helped identify a potential resource-allocation problem. It did not establish average CPU consumption over time or prove how many environments could run safely under peak load.

Recalculating Capacity Using the Current Configuration

Capacity estimates are only as accurate as the resource configuration behind them. As services are added, enabled, or resized, the total CPU and memory requests per environment change, affecting how many environments a node can accommodate.

We recalculated capacity by summing the CPU and memory requests of all enabled services in the current Helm configuration and comparing those totals against the node's allocatable resources.

The calculation uses two constraints:

CPU-bound capacity: available allocatable CPU divided by total CPU requests per environment.
Memory-bound capacity: available allocatable memory divided by total memory requests per environment.

The resulting estimates indicate which resource is likely to become the scheduling bottleneck first.

These figures represent theoretical capacity, not guaranteed production limits. They do not fully account for workload bursts, operational headroom, scheduling constraints, or actual runtime consumption. Concurrent workload testing is needed to establish practical capacity under real operating conditions.

The key lesson is that capacity planning must follow the current configuration. Estimates derived from an earlier chart version can become stale as the platform evolves, even when the underlying infrastructure remains unchanged.

The four problems were all things that went wrong. It is worth stepping back to the thing that went right, and being just as precise about what that result does and does not prove.

Provisioning an Environment in About 110 Seconds

The platform's README records a cold-provisioning observation of approximately 110 seconds, measured from a Git push until Argo CD reported the environment as Synced+Healthy, starting with an empty namespace.

This demonstrates that the ApplicationSet discovery, Helm rendering, and Argo CD reconciliation workflow could provision an environment without manually creating Kubernetes resources.

The available record describes one measured run. It does not include raw timestamps, a sample count, or a script that reproduces the timing. We therefore present 110 seconds as a recorded cold-provisioning observation, not a guaranteed provisioning time or a statistical performance result.

It is also important to understand what the endpoint represents. Synced+Healthy describes Argo CD's view of the managed resources; it does not independently establish that every application feature, external integration, or customer workflow is functioning correctly.

A complete environment still needed its dependencies, secrets, configuration, certificates, and integrations to work as intended.

Reducing the time to create Kubernetes resources was useful. Making the resulting environment dependable was the harder engineering challenge.

Each section so far has paired a finding with a note on how strong the evidence behind it is. Gathered in one place, that honesty becomes its own point: not every improvement here is backed by the same kind of proof, and it helps to see exactly which is which.

What the Implementation Shows, and What Stays Unmeasured

The platform gave us concrete operational evidence about several failure modes. Not every improvement, though, is backed by a controlled benchmark, and the table below sorts each finding by the kind of evidence behind it.

Finding	Evidence available	What it establishes
Dependency startup race	Two documented client incidents with connection refusals and prolonged HTTP 503 readiness	The failure occurred in production
Redis/NATS connectivity gate	Implementation across 23 Deployments	The startup guard was introduced
CPU utilization versus reservations	One historical `kubectl top` observation	A potential resource-request mismatch
Cold provisioning	One recorded run of approximately 110 seconds	The workflow completed within that observed time
Current node capacity	Live node allocatable resources and current chart requests	A reproducible theoretical capacity calculation

We have not run a controlled A/B experiment comparing dependency startup with and without the readiness gate. The production incidents were uncontrolled observations, not repeated tests under identical conditions.

That means we cannot quantify the gate's effect on startup success rate, median time to readiness, or recovery time from the available records.

A minimal isolated test with a deliberately delayed Redis or NATS dependency would be the next step to quantify that improvement. Such a test should record repeated runs with and without the gate and preserve the actual output.

The same principle applies to capacity. The current calculation is useful for planning, but concurrent workload testing would be necessary to validate a production operating limit.

Distinguishing these evidence levels is part of the engineering discipline: historical observations, calculations, and controlled experiments answer different questions.

Why We Stopped Trusting “Deployed” as “Done”

Four habits carried across the incidents, and each one came directly from a failure above. They share a single idea: a clean deployment is a claim about the system, not proof the system works, so each habit is a way of checking what the deployment status does not.

1. We Evaluated Changes Across the Entire Environment.

A chart change could require a secret update; a service change could affect an integration; and certificate automation could introduce DNS and credential dependencies.

2. We Separated Deployment Status from Application Correctness.

A successful sync didn't prove that a database was reachable, a frontend targeted the right API, or an integration worked.

3. We Designed for Recovery.

A workflow needed a way to apply a fix when a resource was stuck, and a Job replacement strategy needed to account for whether its work could safely run again.

4. We Kept the Resource Model Current

A capacity estimate built on an older chart grows misleading as services and their requests change, so the model has to be recalculated against the configuration actually running.

These weren't isolated bugs. They exposed recurring assumptions about dependencies, configuration, resource allocation, and the lifecycle of existing environments.

A Reproducible GitOps Reference You Can Start From

We created a public reference repository to demonstrate the GitOps provisioning pattern without exposing proprietary project configuration:

GitHub: gitops-environment-provisioning-template

The repository illustrates the core provisioning structure, including client discovery through ApplicationSets, a shared Helm chart, and deployment configuration.

It provides a starting point for engineers exploring how to organize a GitOps-driven environment provisioning workflow.

The implementation makes the desired state explicit. The operational lessons show why that is not sufficient on its own: dependencies need readiness checks, configuration changes must account for existing tenants, and recovery procedures must respect the work performed by each resource.

The historical production incidents and resource observations described in this article come from the original platform's operational records. The repository is a public reference implementation, not a claim that the original production environment or all its measurements can be reproduced from the template.

What “Push to Deploy” Actually Meant

We started with a simple requirement: add a client to Git and provision an environment without manually creating Kubernetes resources.

We got that flow working. The harder work was making it predictable when a database wasn't ready, a broker was still starting, a secret was missing, a certificate needed renewal, or a configuration change broke an existing tenant.

GitOps gave us a way to declare and reconcile the desired state. It didn't remove the need to understand runtime behavior, security boundaries, resource consumption, integration correctness, or recovery.

Our production incidents showed where the initial assumptions failed. Our resource analysis showed why capacity calculations must follow the current configuration. And our provisioning observation demonstrated that the core workflow could complete in roughly 110 seconds under the recorded conditions.

The remaining challenge is to turn those operational improvements into repeatable measurements that quantify their impact.

Reliable provisioning came from three things: making dependencies explicit, being honest about the limits of our evidence, and designing around how the system actually behaved rather than how we expected it to. That was the real work behind “push to deploy.”

This Kind of Work Is Its Own Discipline.

Building provisioning that stays reliable once real environments depend on it, dependency ordering, backward-compatible configuration, recovery that respects each resource, and capacity that tracks the live chart, is the work GeekyAnts does through its DevOps engineering practice, which covers CI/CD, Kubernetes, infrastructure as code, observability, and production reliability.

How We Engineered a Three-Path Reward Processing System for a Real-Time Sports Platform

GeekyAnts India Pvt Ltd — Fri, 13 Nov 2026 10:29:04 +0000

How one score event triggers 20+ reward evaluations across sync, async, and scheduled paths

1. The Problem

A player steps into one of our Core Soccer Tech pods, plays a 90-second game, and scores 847 points. The kiosk needs to respond instantly: show the score, trigger a celebration animation if they hit a personal high, and display any XP (experience points) earned.

But behind that single score, the system has to answer over 20 questions. Did they beat their high score? Did they unlock a new game mode? Did that score push them past a reward threshold? How many of today's seven daily missions did that game advance? What about the eight weekly missions running in parallel? And the eight seasonal missions that span the entire quarter?

All of this from one POST request.

The naive approach is to evaluate everything synchronously inside the score submission endpoint. The kiosk waits, the database holds a long transaction, and every player in the facility feels the lag. At 75+ touches per 90-second game and multiple pods running simultaneously, that is not a theoretical concern. It is a production one.

We needed to split the work. Some rewards have to be synchronous because the kiosk needs the data immediately for animations. Some can be deferred by a few seconds. Some only make sense at the end of a week or quarter.

The result is a three-path reward processing engine: synchronous rewards evaluated inside the score transaction, asynchronous processing via a BullMQ worker for missions and achievements, and cron-driven distribution for leaderboard rewards.

2. Architecture Overview

The backend is a NestJS 11 application on Node.js 24. It talks to PostgreSQL through the Knex query builder (via nestjs-knex); there is no ORM (Object-Relational Mapper), so every query in this article is either a Knex builder call or raw SQL. Schema changes, unique constraints, and database triggers all live in Knex migrations. Background jobs use BullMQ 5 on Redis, and scheduled jobs use @nestjs/schedule.

The reward engine sits between score ingestion and four output tables. Every reward, regardless of its source, writes to one or more of these: player_transaction (the universal ledger of XP and CC, the platform's credit currency), player_metadata (running totals maintained by database triggers), player_achievement (one-time milestones), and player_notification (kiosk alerts).

Three ingestion paths, including classic and weekly pattern games, OnePod batch sessions, and battle rooms, all converge into a single addScore() method. From there, processing splits into three paths based on urgency:

Sync Path (green): Runs inside the database transaction. Returns rewards in the HTTP response so the kiosk can show celebration animations immediately. Handles single game rewards, high score detection, and game mode unlocks.

Async Path (purple): Score events are enqueued to a BullMQ queue after the transaction commits. A background worker evaluates achievements, badges, daily missions, weekly missions, seasonal missions, and level-ups with a concurrency cap of 5 parallel jobs.

Scheduled Path (red): Cron jobs run on fixed schedules in the America/Denver timezone. Every Monday: the weekly pattern rotates at 12:00 AM, weekly leaderboard rewards are distributed at 12:05 AM, and the "top 10 in your age group" seasonal mission is updated at 12:10 AM. Battle season rewards run at 12:05 AM on the first day of each quarter. These jobs rank players and distribute CC, XP, and badges based on leaderboard position.

3. The Unified Entry Point

All three ingestion paths converge into addScore(). This is a deliberate architectural choice: a single method handles score creation and sync reward evaluation inside one database transaction, then dispatches the async event after that transaction commits.

Why a Single Entry Point?

Classic games, OnePod batch sessions (2–4 players ranked together), and battle rooms (deferred scoring until all players finish) have very different ingestion patterns. But once a score is created, the reward logic is identical. By funneling everything through addScore(), we guarantee that every score, regardless of how it was submitted, gets the same reward treatment.

Transaction Boundary

The transaction boundary is the most important design decision in this system. Inside the transaction, we run eight operations in a fixed order:

Fetch the previous high score for this player and game mode (weekly pattern scores excluded).
Increment player.game_count and read the new value back.
Insert the score into soccer_game_score.
Insert the game-specific record (one_pod_score or weekly_pattern_score) if the game style needs one.
Insert a NewGameUnlock achievement if the new game count crosses an unlock threshold.
Evaluate single game rewards (section 4) and, if any qualify, record the XP and write one player_transaction row.
Insert a PersonalHighScore achievement if the score beats the previous high.
Log today's play date into player_daily_play_log.

The transaction callback returns, Knex issues COMMIT, and only then is the ScoreEventPayload enqueued to BullMQ. Enqueueing is deliberately outside the transaction. If the enqueue fails, the score is saved but async rewards are lost, and because the enqueue call is not caught, the kiosk receives a 500 for a score that was in fact persisted. We accepted this tradeoff to keep the transaction short and avoid coupling score persistence to Redis availability.

const result = await this.knex.transaction(async trx => {
  const prevHighScore = await getPrevHighScore(trx, playerId, req.game_mode_id)

  // Step 1
  const [{ game_count }] = await trx('player')
    .where('id', playerId)
    .increment('game_count', 1)
    .returning('game_count')

  // Step 2
  const [createdScore] = await trx('soccer_game_score')
    .insert({
      player_id: playerId,
      game_mode_id: req.game_mode_id,
      score: req.score,
      game_style: req.game_style
    })
    .returning('*')

  // Step 3
  // Steps 4 to 8 run here, all on `trx`
  return {
    createdScore,
    prevHighScore,
    newGameCount: game_count
  }
})

// COMMIT has happened. Now, and only now, hand the event to the worker.

await this.scoreEventsQueue.addScoreEventJob({
  player_id: playerId,
  score_id: result.createdScore.id,
  score: result.createdScore.score,
  game_mode_id: result.createdScore.game_mode_id,
  game_style: result.createdScore.game_style,
  prev_high_score: result.prevHighScore,
  new_game_count: result.newGameCount,
  season: result.createdScore.season,
})

Battle rooms and OnePod sessions insert several scores in one outer transaction, so they pass their own trx into addScore(). Battle completion also passes skipEnqueue = true, collects one payload per player, and enqueues them all after the outer commit.

The ScoreEventPayload The payload sent to the async worker carries everything needed to evaluate all downstream rewards without additional database lookups at the entry point:

{ player_id, score_id, score, game_mode_id, game_style, prev_high_score, new_game_count, season }

prev_high_score is included because the worker needs it for score level achievement checks. Without it, the worker would need to query historical scores, adding latency and database load to every async job.

4. The Sync Path: Immediate Rewards

These rewards are evaluated inside the addScore() transaction and returned in the HTTP response. The kiosk uses them to display immediate feedback: XP earned animations, high score celebrations, and game unlock screens.

Single Game Rewards

Four reward types are evaluated per score, and all can stack:

Reward	Condition	Notes
`play_any_game`	Always qualifies	Base XP for playing
`score_x`	`score >= floor(third_best_score * threshold_pct / 100)`	Needs at least 3 scores in the mode
`score_y`	Same formula, higher `threshold_pct`	Harder tier
`new_high_score`	`score > prev_high_score AND games_in_mode >= min_high_score_games`	`min_high_score_games` comes from global config, default 10

Each row is configured in single_game_reward_config (threshold_pct, xp_reward, animation, and a game_style exclusion list). Weekly pattern games skip this block entirely.

The best-score formula is worth noting: instead of comparing against the player's top score, we compare against their third-best score in that game mode. This prevents a single lucky outlier from permanently raising the bar; a player has to post three strong games before the threshold moves.

// Top 3 in this mode, read after the new score was inserted (so it is included)
const topScores = await trx('soccer_game_score').select('score')
  .where({ player_id, game_mode_id })
  .whereNot('game_style', 'weekly_pattern')
  .orderBy('score', 'desc').limit(3)
const thirdBestScore = topScores.length >= 3 ? topScores[2].score : null

// score_x / score_y
if (thirdBestScore !== null && config.threshold_pct !== null) {
  const threshold = Math.floor((thirdBestScore * config.threshold_pct) / 100)
  qualifies = score >= threshold
}

// new_high_score
qualifies = gamesInMode >= minHighScoreGames && score > prevHighScore

If a player has fewer than 3 scores in a mode, thirdBestScore is null and score_x and score_y are not awarded at all. Only play_any_game fires on those first two games. There is one side effect of including the new score in the top-3 query: on exactly the third game in a mode, the new score can be the third best itself, so score_x and score_y automatically qualify for that game.

High Score Detection and Game Unlocks

If the new score beats the player's previous high score (and the game style is not weekly_pattern), a PersonalHighScore achievement is inserted. This check has no game-count gate, so a player on their third game gets the celebration animation but not the new_high_score XP. Similarly, when game_count crosses an unlock threshold, a NewGameUnlock achievement is created with metadata indicating which game mode (CoreOG, Mirror, Xpert, Clutch, or Mirror_X) was unlocked.

5. The Async Path: BullMQ Worker

After the transaction commits, a ScoreEventPayload is enqueued to a Redis-backed BullMQ queue named scoreEvents. A worker picks it up and processes the steps below sequentially.

Worker Configuration

The worker runs in the same Node.js process as the HTTP server with a concurrency cap of 5 parallel jobs. Successful jobs are removed immediately; the last 100 failed jobs are retained for debugging. This is a pragmatic choice: a separate worker process would add deployment complexity for a system that handles scoring bursts measured in dozens per minute, not thousands.

this.queue = new Queue('scoreEvents', { connection: redisConfig })

this.worker = new Worker<ScoreEventPayload>(
  'scoreEvents',
  async job => this.withDeadlockRetry(() => this.processScoreEvent(job.data)),
  { connection: redisConfig, concurrency: 5 },
)

// called by addScore() after COMMIT
await this.queue.add('processScore', payload, {
  removeOnComplete: true,
  removeOnFail: 100,
})

That is the whole configuration. We do not set attempts or backoff, so BullMQ's defaults apply: a job that throws is moved straight to the failed set with no automatic retry. The only retry in this path is application-level. withDeadlockRetry() reruns the job up to three times when PostgreSQL reports a deadlock (error code 40P01), waiting 50 * attempt + random(0..50) milliseconds between tries. Everything else fails once.

Stalled jobs follow BullMQ defaults too: a job lock lasts 30 seconds and is renewed every 15; if the process dies or blocks the event loop long enough for the lock to expire, the stalled checker (every 30 seconds) puts the job back in the wait list once (maxStalledCount: 1). A second stall causes the job to fail. After a final failure the error is logged with the player_id and score_id, the job sits in the failed set until it is pushed out by the 100-job cap, and nothing replaces it. There is no dead-letter queue and no alerting on the failed set today.

Achievement Evaluation (Steps 1–2)

Step 1 checks whether the new score crossed a score-level tier. These configurable thresholds are defined per game mode and can award XP, credits, and badges. Step 2 checks career total points milestones, using a database trigger-maintained counter on player_metadata. Both use an achievementExists(score_id, type) idempotency check before inserting, preventing duplicates if the same event is ever processed twice. A points-badge evaluation runs between step 2 and the missions.

Mission Processing (Steps 3–5)

This is where the bulk of the work happens. Steps 3 through 5 evaluate daily, weekly, and seasonal missions respectively. They run sequentially within a single job, not in parallel. This was a deliberate choice: sequential execution means a database error in daily missions prevents weekly and seasonal from running for that event, but it also means we never have two mission services competing for database connections from the same job. A level-up check runs last, after the seasonal missions.

6. The Mission System: Three Time Horizons

The mission system operates across three time horizons, each with fundamentally different state management patterns. Missions unlock on a player's fifth game overall: games 1 to 4 are skipped, game 5 creates the mission rows, and processing starts from game 6.

Daily Missions

Seven mission types reset every day. On a player's first score each day, getOrCreateDailyMissions() lazily initializes all seven missions with fresh progress counters. Goals are personalized; the TotalPoints mission, for example, sets its goal to the player's third-best Core OG score multiplied by a configurable factor (minimum 1), ensuring the target scales with skill level.

The ScoreYStreak mission is the most interesting pattern: it requires the player to score above a threshold N times in a row. One miss resets progress to zero, so each consecutive successful score increases the cost of breaking the streak. The backend tracks this with a simple conditional: if score >= threshold, increment progress; else, reset to zero.

Weekly Missions

Eight mission types reset every Monday. The week anchor is computed in the database as date_trunc('week', (current_timestamp AT TIME ZONE 'America/Denver')::date). The most complex is TopNScore, where N is locked when the week's missions are created, based on the player's total game count: experienced players must score in their top 5, newer players in their top 10. This graduated difficulty prevents both sandbagging and frustration.

Weekly missions also track consecutive play days using an append-only player_daily_play_log table. Each day a player plays, a row is inserted with INSERT ... ON CONFLICT (player_id, play_date) DO NOTHING (Knex .onConflict([...]).ignore()) for concurrent access safety. The streak count is computed from this log at evaluation time.

Seasonal Missions

Eight missions spanning an entire quarter, with 16 to 200 threshold milestones each. Unlike daily and weekly missions that track per-score progress, seasonal missions use aggregate counters stored in player_season_data. Each score event increments these counters with a single UPDATE, and the mission service compares current values against threshold tables.

The idempotency mechanism here is elegant: each player's mission row stores a last_completed_threshold_id. When processing, the service iterates through thresholds in sort order, skipping anything at or below this ID, and stopping at the first uncrossed threshold. The ID update and reward insert happen in the same database transaction, so if the transaction rolls back, neither persists.

for (const t of thresholds) { // sort_order ascending
  if (mission.last_completed_threshold_id !== null && t.id <= mission.last_completed_threshold_id) continue
  if (t.threshold_value > mission.current_value) break
  newThresholds.push(t)
}
if (newThresholds.length === 0) continue

await trx('player_seasonal_mission').where({ id: mission.id })
  .update({ last_completed_threshold_id: newThresholds.at(-1).id })
await this.levelUpService.recordXp(trx, playerId, season, totalXp, XpSource.SeasonalMission)
await trx('player_transaction').insert({ player_id: playerId, transaction_type: 'add',
  reward_name: 'seasonal_mission', xp_earned: totalXp, free_live_credits: totalCc })

Rotating Game Modes

Both daily and weekly missions include a "play the rotating game mode" mission type. Daily rotation is deterministic: day of year (Denver local date) modulo 4 maps to Mirror, Xpert, Clutch, or Mirror_X. Weekly rotation is derived from the week's start date and stored in a weekly_mission_rotation table with ON CONFLICT (week_start_date) DO NOTHING for concurrent initialization. This keeps missions fresh without requiring manual configuration.

7. The Scheduled Path: Cron Jobs

Four cron jobs handle scheduled game-pattern updates and time-bounded reward processing. All run in the America/Denver timezone to match the facilities' operational hours. The scheduler is @nestjs/schedule (which wraps the cron npm package), and every job passes the same constant:

export const CURR_APP_TIMEZONE = 'America/Denver'

@Cron('0 0 * * 1', { timeZone: CURR_APP_TIMEZONE }) // Monday 12:00 AM: rotate weekly pattern @Cron('5 0 * * 1', { timeZone: CURR_APP_TIMEZONE }) // Monday 12:05 AM: weekly leaderboard rewards @Cron('10 0 * * 1', { timeZone: CURR_APP_TIMEZONE }) // Monday 12:10 AM: top 10 age group mission @Cron('5 0 1 1,4,7,10 *', { timeZone: CURR_APP_TIMEZONE }) // Jan/Apr/Jul/Oct 1, 12:05 AM: battle season rewards

Daylight Saving Time (DST) is handled by the cron library, which evaluates the expression in the named zone, so "12:05 AM Denver" stays 12:05 AM local across the March and November changes. Daily mission dates, the weekly anchor, and the play log use the same zone inside PostgreSQL (current_timestamp AT TIME ZONE 'America/Denver'); the database session itself runs in UTC. Game-mode rotation and the battle season calendar resolve the same zone in Node.js.

Each job is wrapped in a small withRetry() helper that allows three attempts with a fixed five-minute wait between them. A job that fails all three attempts logs the error and waits for its next scheduled run.

Weekly Leaderboard Rewards – Monday 12:05 AM

Every Monday, the system ranks the previous week's weekly pattern scores. Players are split into age group leaderboards (grouped by age bracket and gender) plus an overall leaderboard. Ranks are matched against a leaderboard_reward_config table that maps rank ranges to CC, XP, and badge rewards. A unique index on (player_id, leaderboard_type, season, week_start_date) prevents duplicate awards.

Top 10 Age-Group Update – Monday 12:10 AM

Five minutes after weekly rewards, a second cron processes the seasonal mission that tracks how many weeks a player placed in their age group's top 10. It reads from the player_leaderboard_reward table that the 12:05 AM job just populated.

The five-minute gap is the only dependency mechanism. The 12:10 AM job does not check whether the 12:05 AM job finished; it simply queries last week's top-10 rows and processes whoever is there. If the first job is still running, or is in its 5-minute retry wait, the second job sees a partial or empty result and finishes early. Two properties make this acceptable rather than dangerous. First, the job recomputes each player's count from player_leaderboard_reward as an absolute value rather than an increment and skips the player when the stored value already matches. This allows the job to be rerun manually without double-awarding. Second, the first job's retry schedule (12:10 and 12:15) means the rows usually exist within minutes. Affected players are picked up the next time they finish in the top 10, when the recount includes the missed week. We accept that a player can see this mission update one week late.

Battle Season Rewards – Quarterly

On the first day of each quarter, battle season rewards process. Two leaderboards are computed: total battle points (sum of points from battle royale and OnePod ranked games) and battle efficiency (points-to-expected ratio, preventing pure volume from dominating). Both leaderboards use the same check-then-insert and unique index pattern for idempotency, with the season start date in week_start_date.

8. Idempotency and Known Risks

No distributed system is perfectly reliable. Here is an honest assessment of where our idempotency holds and where it has gaps.

Where It Holds

System	Mechanism	Status
Achievements	`achievementExists(score_id, type)` before every insert	Fully idempotent
Seasonal thresholds	`last_completed_threshold_id` bookmark, updated in the reward transaction	Safe via transaction
Leaderboard rewards	Check-then-insert inside a transaction, unique index as backstop	Fully idempotent
Level-up rewards	`player_level_reward_history` unique on `(player_id, level_config_id, season)` with `ON CONFLICT DO NOTHING`	Fully idempotent

The achievement check is a single indexed lookup:

private async achievementExists(score_id: number, achievement_type: AchievementType) {
  return !!(await this.knex('player_achievement').where({ score_id, achievement_type }).first())
}

Leaderboard rewards use the unique index from the migration plus an explicit check, because the check lets us skip the CC, XP and notification writes cleanly instead of catching a constraint error:

// migration
table.unique(['player_id', 'leaderboard_type', 'season', 'week_start_date'],
  { indexName: 'uq_player_leaderboard_reward' })
await this.knex.transaction(async trx => {
  const existing = await trx('player_leaderboard_reward')
    .where({ player_id, leaderboard_type, season, week_start_date }).first()
  if (existing) return

  await trx('player_leaderboard_reward').insert({ player_id, leaderboard_type, season, week_start_date,
    rank, cc_awarded: tier.cc_reward, xp_awarded: tier.xp_reward, badge_config_id: tier.badge_config_id ?? null })
  if (tier.cc_reward > 0) {
    await trx('player_transaction').insert({ player_id, transaction_type: 'add',
      free_live_credits: tier.cc_reward, free_virtual_credits: 0, xp_earned: 0 })
  }
  if (tier.xp_reward > 0) {
    await this.levelUpService.recordXp(trx, player_id, season, tier.xp_reward, XpSource.WeeklyPattern)
  }
})

Wherever the article says a row is inserted with conflict protection, the actual statement is PostgreSQL's INSERT ... ON CONFLICT (...) DO NOTHING, written in Knex as .onConflict([...]).ignore(). Reward history rows use .onConflict([...]).merge([...]), which becomes ON CONFLICT ... DO UPDATE SET.

Where It Has Gaps

Daily and weekly missions track progress with increment-based counters, and the seasonal player_season_data counters (games played, total points, days played) are incremented the same way. If a score event is processed twice, those counters get double-incremented. The completion rewards are protected by a separate reward history table and the threshold bookmark, but the progress inflation can cause missions to complete earlier than intended.

With no BullMQ retries configured, a second run of the same event can come from only two places: the in-process deadlock retry, which re-runs processScoreEvent() from the top after a partial write (the function is not one transaction), and BullMQ's stalled-job recovery, which re-queues a job once if its lock expired mid-run.

Known Risks We Accept

Pool exhaustion: The HTTP server and BullMQ workers share a single database connection pool. Under high scoring load, 5 concurrent workers multiplied by multiple queries per job plus HTTP traffic can compete for connections. This has not been a production issue yet, but it could become a scaling constraint if scoring volume increases significantly.

Sequential mission processing: Steps 3 to 5 in the worker run sequentially. A failure in daily missions prevents weekly and seasonal from running for that score event. We chose this over parallel execution to avoid connection pool pressure, but it means one service's bug can cascade.

Single-process cron: The cron jobs have no distributed lock. The API runs as one Node.js process today; running two instances would fire every cron job in both, and the second run would fail on the unique index rather than double-award.

Building Reward Systems That Can Handle Different Processing Needs

A reward system does not need to process every event in the same way. In this case, separating immediate, asynchronous, and scheduled work keeps player-facing responses fast while moving heavier evaluations and time-based rewards to paths better suited to them. The approach has been running in production across multiple Core Soccer Tech facilities, handling over 38 million scored points from 7,000+ players. The architecture also shows why transaction boundaries, idempotency, queue behavior, failure handling, and known system constraints need to be considered together when designing production systems. These are the same concerns that shape backend engineering for applications that need to support growing workloads without making every workflow part of one synchronous request.

Scaling Down Before Scaling Up: Why Bigger Servers Don’t Fix Bad Architecture

GeekyAnts India Pvt Ltd — Fri, 13 Nov 2026 09:27:58 +0000

This blog explores how inefficient backend architecture can cause performance issues even under low traffic, covering practical ways to reduce database load, API latency, and resource usage before scaling infrastructure.

Why Scaling Infrastructure Wasn’t the Answer

When an application becomes slow, the first instinct is often predictable: add more memory, upgrade the server, increase database capacity, or scale vertically.

Sometimes, that works until it doesn’t.

We experienced this firsthand when an application that had been running smoothly on a 64 GB server started crashing after being moved to a basic 2 GB instance.

What made the situation unusual was the workload. There were only 4–5 active users, a limited dataset, and no major traffic spikes. Yet the application experienced database connection spikes, high API latency, frequent request timeouts, and unstable performance.

This wasn’t a scaling problem. It was an architecture problem.

The Problem: When a 64 GB Server Hid the Real Issue

What we observed

The infrastructure changed, but the workload did not.

Our initial assumption was that the smaller server lacked sufficient resources. But the connection behavior and latency suggested something deeper.

Database connections were spiking despite low traffic.
API latency remained high.
Requests frequently timed out.
Connection pooling stabilized the number of connections but did not resolve the latency issue.

The Architecture Before Refactoring

The application used Hasura, Node.js services, PostgreSQL, and multiple Dockerized services. The biggest issue was in the request path for Hasura Actions.

The problematic request lifecycle

The Node.js service was calling back through Hasura instead of using a direct database path where appropriate.

This created a circular dependency between Hasura and Node.js. A single request could therefore involve multiple network hops, GraphQL execution layers, serialization and parsing, and additional database connection usage.

Hasura → Node.js → Hasura → PostgreSQL was doing more work than the request actually required.

Why Connection Pooling Wasn’t the Final Fix

Symptom vs. root cause

PgPool/PgBouncer helped control the number of active database connections, but it did not reduce the amount of work being performed per request.

That distinction mattered. We had solved connection management, not query efficiency or architectural overhead.

Connection pooling controls concurrency; it does not automatically make inefficient work efficient.

Breaking the Loop: Rethinking the Data Access Layer

The simplified architecture

For straightforward database operations, remove unnecessary internal round trips.

Where Node.js remains valuable

We reduced unnecessary Hasura Actions and removed circular Hasura → Node.js → Hasura flows where direct database access was more appropriate.

Hasura continued to handle client-facing GraphQL access, CRUD, filtering, and aggregations. Node.js remained responsible for business logic that genuinely required an application service, authentication flows, external integrations, and asynchronous workflows.

The goal was not to remove Hasura or Node.js. It was to give every layer a clear responsibility.

The Cost of Over-Abstraction

GraphQL, microservices, Hasura Actions, and Node.js each solve real problems. The issue occurs when every request is forced through every abstraction.

More network hops
More serialization/deserialization
Higher latency
Duplicate responsibilities
More database connection churn
Harder debugging and observability

Abstraction should reduce complexity—not increase the number of steps required to complete a request.

Database Optimization: Reducing Work at the Source

1. Eliminate Redundant Queries

Redundant request pattern

Optimized request pattern

We consolidated repeated reads into fewer, better-structured queries. This reduced query count, database pressure, network traffic, and response time.

2. Solve N+1 Query Patterns

N+1 pattern

Optimized pattern

Instead of repeatedly fetching related entities, we used joins, CTEs, and PostgreSQL functions where appropriate.

3. Indexing Strategy

Indexing decision flow

We added indexes based on actual access patterns rather than indexing fields indiscriminately. Foreign keys, join columns, filtering fields, sorting fields, and pagination patterns were the primary candidates.

The goal was predictable query performance. Every index also carries storage and write-maintenance costs.

4. Schema Design and Constraints

Use clear and consistent naming conventions.
Define foreign-key relationships explicitly.
Use NOT NULL, UNIQUE, and CHECK constraints where the data model requires them.
Keep relationships normalized where appropriate.
Review RBAC policies for both correctness and query efficiency.

Database constraints document the data model and enforce important data rules at the source.

PostgreSQL Functions and Triggers: Put the Right Work in the Database

PostgreSQL Functions

When is a PostgreSQL function useful

Functions are useful for complex database-oriented fetching, reusable calculations, aggregations, and returning data in a form that reduces application-layer processing.

Database Triggers

Trigger use case

Triggers are appropriate for deterministic database responsibilities such as maintaining counters, derived values, timestamps, audit records, or data-integrity behavior.

Use triggers for deterministic database responsibilities—not as a hidden replacement for the entire application layer.

Database Events vs. External Workflows

Keep external work out of the critical transaction

Email delivery, push notifications, and other external calls should be handled asynchronously when the user-facing database transaction does not need to wait for the external system.

Views and Materialized Views

Normal Views

Reusable read model

Normal views are useful when you want to encapsulate reusable joins and filtering logic while still reading the current underlying data.

Materialized Views

Precomputed analytics path\

Materialized views are useful when expensive aggregations do not need to be recalculated for every request.

For appropriate workloads, REFRESH MATERIALIZED VIEW CONCURRENTLY can reduce disruption during refreshes. A suitable unique index is required for concurrent refreshes.

Safe refresh strategy

Optimizing Analytics APIs

Avoid repeated runtime calculations

Analytics endpoints often perform expensive joins and aggregations over transactional data. We reduced this work by pre-aggregating data, using materialized views, caching suitable results, and minimizing on-the-fly calculations.

If an expensive value does not need second-by-second freshness, don’t calculate it from scratch on every request.

API Optimization Principles

Pagination and Filtering

Efficient list API

Pagination and filtering should happen as close to the database as possible. Cursor-based pagination can be useful for large or frequently changing datasets.

Reduce Over-Fetching

Minimal response path

Avoid SELECT * and fields that the client does not need. Smaller payloads reduce database work, serialization, network transfer, and client-side processing.

Computed Fields

Avoid repeated runtime computation

Expensive computed fields can introduce latency when evaluated for every record. Where appropriate, use precomputed values, summary tables, materialized views, or caching.

GraphQL: Powerful, but Give It Guardrails

Avoid unnecessarily deep nested queries.
Control expensive computed fields.
Watch for N+1 patterns.
Limit excessive Hasura Actions.
Avoid using GraphQL as an internal service-to-service communication protocol when a more direct mechanism is appropriate.
Return only the data required by the client.

Healthy boundary

Service responsibility

Redis: Cache What Doesn’t Need to Be Recomputed

Read-through cache

Redis is useful for read-heavy endpoints, configuration data, analytics results, frequently requested reference data, and suitable computed counters.

Every cache should have a clear TTL, invalidation, refresh, and consistency strategy.

Effective caching targets data that benefits from reuse and has a defined consistency strategy.

Batch Processing: Email and Notifications

Synchronous anti-pattern

Asynchronous pattern

Background processing

This keeps external service latency out of the user-facing request and allows retries, batching, and controlled failure handling.

Data Lifecycle Management

Scheduled cleanup

Expired sessions, temporary records, old notifications, logs, stale tokens, and other disposable data should have an explicit lifecycle.

Large cleanup operations should be batched and controlled so that maintenance itself does not become a resource spike.

Observability: Measure Before You Optimize

Performance investigation loop

Query logging to identify slow SQL statements
pg_stat_activity to inspect active database sessions
EXPLAIN ANALYZE to inspect query execution plans
API timing to separate application, database, external-service, and network time
GraphQL resolver timing to identify expensive fields

Without observability, you can spend weeks optimizing the wrong layer.

Reducing Backend Downtime

Reliability loop

Performance and availability are connected. Health checks, monitoring, structured logs, graceful degradation, controlled retries, and automated recovery help prevent resource pressure from becoming prolonged downtime.

Retries should recover from failures without amplifying them.

Do We Really Need That Microservice?

Microservice decision

Every service adds container memory, network communication, database connections, deployment complexity, monitoring, and operational overhead.

If multiple services are tightly coupled, communicate constantly, share the same database, and always scale together, consolidation can produce a simpler and more efficient architecture.

The Anti-Patterns We Discovered

Hasura → Node.js → Hasura circular API calls
Using GraphQL as a universal internal communication layer
Over-splitting logic across microservices
Distributing one business flow across too many layers
Treating PostgreSQL as pure storage
Using infrastructure capacity to hide inefficient request paths

Each decision looked manageable in isolation. Together, they created system-wide overhead.

When 2 GB Became an Advantage

What the smaller environment exposed

The 2 GB environment didn’t create all of these problems. It exposed them. The constrained environment forced us to question every connection, query, network hop, container, and runtime calculation.

The 2 GB environment didn’t create all of these problems. It exposed them.

What Changed After the Refactor?

Outcome

More controlled database connections
Lower API latency
Fewer unnecessary database calls
Reduced network hops
Less application-layer processing
More efficient analytics queries
Better separation of responsibilities
Reduced container overhead
More asynchronous background processing
Lower infrastructure requirements

The Bigger Lesson: Scale Down Before You Scale Up

When an application becomes slow, the instinct is to scale. But adding infrastructure to an inefficient architecture can increase its cost without addressing the underlying problem.

If every request makes unnecessary network calls, adding more servers does not eliminate those calls. If every API performs expensive queries, increasing RAM does not make those queries logically simpler. If every service opens unnecessary database connections, increasing database capacity does not fix the request flow.

The optimization mindset

In our case, the answer was to simplify the request path, remove circular Hasura–Node.js communication, reduce unnecessary Actions, move appropriate operations closer to PostgreSQL, optimize queries and indexes, address N+1 patterns, use pagination and database-level filtering, precompute analytics, cache read-heavy workloads, move external work to background processing, and reduce unnecessary services.

The 64 GB server made the system comfortable. The 2 GB server made it honest.

Before scaling infrastructure, reduce unnecessary work across the architecture, request path, and database. Then scale based on measured demand.

Build for Scale at the Architecture Level

Scaling infrastructure works best when the architecture is ready to support it. Clear service boundaries, optimized database access, efficient request paths, caching, and asynchronous processing can reduce resource pressure before additional capacity enters the equation. If your application is facing performance bottlenecks or rising infrastructure demands, explore how backend engineering services can help strengthen the system before the next stage of scale.

My Flutter App Froze With Three Photos on Screen. Here's What I Was Doing Wrong

GeekyAnts India Pvt Ltd — Fri, 13 Nov 2026 08:16:51 +0000

This blog explains how rethinking Flutter’s image-processing architecture fixed severe performance issues and improved rendering efficiency.

How chasing the right optimizations in the wrong place locked up a Flutter carousel that had plenty of hardware to spare.

I had done everything right.

Isolates for heavy processing, caching for repeated renders, viewport-aware lazy loading so off-screen photos were not burning cycles. This was textbook Flutter performance optimization, but the app still lagged, even on a flagship Android phone with 12 GB of RAM.

That is the thing about architectural mistakes. They do not care how clean your implementation is.

The App, and Where It Started Hurting

InstaFramer is an open-source bulk image editor for Instagram creators. The core use case is simple: take a batch of photos, fit them into Instagram's aspect ratios without cropping, and export them. To fill the empty space around the photo, users can pick a white background, a black background, or an extended blur background (the same photo, blurred and stretched to fill the frame).

The editor screen has a swipeable carousel. Select up to 20 photos, swipe through them, tweak the settings, and export.

Even with white or black backgrounds, the carousel was not silky smooth. Performance was passable on a flagship device, but the moment blur entered the picture, it became completely unusable. For each card, showing the real-time blur preview, in which the original photo is centered and scaled over a blurred version of itself, required a heavy and repeated process: decoding the image, resizing it, running a Gaussian blur, and compositing the result every time the settings changed.

I reached for the standard toolkit. Background isolates via compute() for the heavy lifting, a FutureBuilder in each PhotoCard to handle the async result, and Image.memory() to render the final Uint8List. Isolates are Dart's concurrency model; they run in separate memory spaces, so any data passed back to the main thread has to be serialized across that boundary.

On paper, this is correct Flutter. In practice, it was burning memory and CPU cycles far beyond what a background preview pipeline should ever need.

What "Correct" Looked Like

Here is the PhotoCard build method at this stage:

child: FutureBuilder<Uint8List?>(
  future: _generatePreview(widget.photo, widget.settings),
  builder: (context, snapshot) {
    if (snapshot.connectionState == ConnectionState.waiting) {
      return _buildLoadingState(); // spinner, "Processing preview..."
    }
    if (snapshot.hasError || !snapshot.hasData) {
      return _buildErrorState();
    }
    return AspectRatio(
      aspectRatio: widget.settings.aspectRatio.ratio,
      child: Image.memory(snapshot.data!, fit: BoxFit.contain),
    );
  },
),

And _generatePreview was calling imageProcessor.processPreview(photo, settings), which ran in a background isolate, decoded the full image, downsampled it, ran the Gaussian blur, and returned a Uint8List back across the isolate boundary.

For one photo, this is fine. For 20 photos in a carousel, where each swipe could trigger a rebuild and changing the blur intensity slider could invalidate all cached previews simultaneously, this was a performance disaster in slow motion.

The App Froze While I Was Profiling the Freeze

I found out exactly how bad it was during a single debugging session.

At 10:54, I opened the CPU flame chart. The frames were taking close to a second to complete.

At 10:55, the performance overlay confirmed what I was seeing: 34 FPS average, with orange jank bars on nearly every frame.

At 10:57, while I was still looking at the data, this appeared:

The app had frozen while I was profiling the freeze.

Three photos: Not ten, not twenty. Three. And the app was already choking.

A Quick Note on Why Isolates Were Not the Problem

Before going further: isolates are not the villain here. They are the right tool for CPU-heavy work, and they genuinely helped during the export phase (more on that later).

The problem was specifically using isolates to generate UI previews: work that was happening on every render, on every card, in a live scrolling carousel. Every preview was a full round trip: decode on the isolate, Gaussian blur on the isolate, serialize the result into a Uint8List, ship it back across the isolate boundary, and hand it to the GPU to display.

The question I had not asked yet was simple: does this actually need to be a CPU operation at all?

I was not asking it yet. First, I tried to fix what I had.

The Fixes That Almost Worked

The obvious move after a freeze like that is to reduce scope. Stop processing everything at once. Be smarter about when and what to compute. The instincts were sound. The problem was that acting on them meant optimizing a pipeline that should not have existed in the first place.

False Fix 1: Viewport-Aware Lazy Loading

The first patch came from a straightforward diagnosis: the carousel was firing off preview generation for all 20 photos the moment the screen loaded. Even cards the user had not looked at yet were running _generatePreview, spinning up isolates, and allocating memory in parallel.

The fix was viewport gating. Only process photos that are actually visible or immediately adjacent. Every card beyond a distance of 1 from the current index gets a grey placeholder instead.

bool _isPhotoInViewport(int photoIndex, int currentIndex) {
  final distance = (photoIndex - currentIndex).abs();
  return distance <= 1;
}

child: isInViewport
    ? FutureBuilder<Uint8List?>(
        future: _generatePreview(widget.photo, widget.settings),
        ...
      )
    : _buildViewportPlaceholder(...),

This worked. Measurably. Instead of 20 concurrent decode-blur-serialize pipelines competing for memory, there were at most 3. The freeze window narrowed considerably. A 20-photo session that previously locked up immediately became usable for longer.

But the root cause was completely intact. Every time you swiped to a new card, the full pipeline kicked in:

Decode on the isolate → Gaussian blur on the isolate → Serialize the result → Cross the isolate boundary → Allocate a fresh Uint8List → Hand it to the GPU

The viewport gate just reduced how many times it fired.

False Fix 2: LRU Cache + Cache-First Rendering

The second patch addressed a different inefficiency: cards that had already been processed were being reprocessed on every rebuild. The cache was the logical answer.

An LRU cache with maxCacheSize = 12 and priority-based eviction (current photo = 100, adjacent = 50, others = 10). The viewport logic moved from inside PhotoCard up to the carousel level (a cleaner separation that made the cache lookup straightforward). Before falling through to the FutureBuilder, check the cache first.

if (widget.previewCache.containsKey(cacheKey)) {
  return AspectRatio(
    aspectRatio: widget.settings.aspectRatio.ratio,
    child: Image.memory(widget.previewCache[cacheKey]!, fit: BoxFit.contain),
  );
}

// Cache miss, still falls through to FutureBuilder
return FutureBuilder<Uint8List?>(...);

A cache hit was instant. No async, no spinner, no isolate. For photos the user had already visited, this felt genuinely snappy.

For everything else, the full round trip was still happening.

What Both Fixes Had in Common

Each fix was a layer of insulation around the real problem. Viewport gating reduced how many cards triggered the pipeline. The cache reduced how often any given card triggered it. Between the two, the freezes became less frequent and took longer to surface. The blast radius shrank. But neither one touched the underlying assumption: that generating a blurred preview was a CPU task.

Every cache miss, which means every first load of every card, was still doing the same thing: CPU decodes the image, CPU blurs it, CPU serializes the pixels into a Uint8List, that Uint8List crosses the isolate boundary, gets allocated on the main heap, and then gets handed to the GPU to draw.

The GPU was the final recipient of this data every single time. It was rendering the pixels the CPU had spent so much work preparing.

So: why was the CPU doing this at all?

The Question I Had Been Avoiding

Two commits in, the freeze required more photos to trigger. The jank required none.

The assumption underneath all of it: the preview the user sees should be pixel-identical to the exported output. Many image-editing workflows separate fast edit previews from the final processed output. But I had been treating that as someone else's product decision, not a performance insight I could borrow.

That assumption had survived two commits completely untouched. It was time to poke it.

The Test

I rewrote PhotoCard to use Stack + AssetEntityImage + BackdropFilter, ran an export on the same photo through the existing CPU pipeline, and pulled both outputs up side by side on my phone.

Then I zoomed in. Then I zoomed in more. I dragged both images around looking for any sign the GPU compositor had cut a corner somewhere. Color cast, edge softness, blur falloff, anything. Something had to be off. The CPU pipeline was doing actual Gaussian math. The widget tree was just compositing.

There was nothing there.

Same softness. Same falloff. Same color cast. The GPU compositor had been producing a visually identical result the entire time, as a byproduct of normal widget rendering. I had been bypassing it with a full decode-blur-serialize-transfer pipeline, paying the complete memory and compute cost, to produce the same output the widget tree was already giving me for free.

Flutter was already solving my problem. I just was not asking it to.

The Redesign

Here is PhotoCard after the pivot, now using AssetEntityImage from the photo_manager package instead of Image.memory.

class PhotoCard extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return Card(
      clipBehavior: Clip.antiAlias,
      child: AspectRatio(
        aspectRatio: settings.aspectRatio.ratio,
        child: Stack(
          fit: StackFit.expand,
          children: [
            _buildBackground(context),
            Center(
              child: Transform.scale(
                scale: settings.scale,
                child: AssetEntityImage(
                  photo,
                  isOriginal: false,
                  thumbnailSize: const ThumbnailSize.square(800),
                  fit: BoxFit.contain,
                  filterQuality: FilterQuality.medium,
                ),
              ),
            ),
          ],
        ),
      ),
    );
  }

  Widget _buildBackground(BuildContext context) {
    switch (settings.backgroundType) {
      case BackgroundType.extendedBlur:
        return Stack(
          fit: StackFit.expand,
          children: [
            AssetEntityImage(
              photo,
              isOriginal: false,
              thumbnailSize: const ThumbnailSize.square(150),
              fit: BoxFit.cover,
            ),
            BackdropFilter(
              filter: ui.ImageFilter.blur(
                sigmaX: settings.blurIntensity * 0.5,
                sigmaY: settings.blurIntensity * 0.5,
              ),
              child: Container(color: Colors.black.withOpacity(0.1)),
            ),
          ],
        );
      case BackgroundType.white:
        return Container(color: Colors.white);
      case BackgroundType.black:
        return Container(color: Colors.black);
    }
  }
}

BackdropFilter applies the blur via the GPU compositor directly during the render pass, so there's no CPU pixel work involved. The background layer requests an intentionally small ThumbnailSize.square(150) thumbnail. Blur hides the resolution artifacts at this preview size, so 150 px is visually indistinguishable from full resolution once the shader is applied. The foreground (the layer the user actually looks at) gets ThumbnailSize.square(800). Higher fidelity where it matters, smaller where it does not.

blurIntensity * 0.5 as sigma was tuned to match the visual feel of the CPU-rendered export. The side-by-side test confirmed it.

There is no FutureBuilder. No spinner. No _generatePreview. No Uint8List. PhotoCard is now a StatelessWidget because there is genuinely no state left to manage.

The full architecture shift:

// BEFORE
Photo → compute() isolate → decode → Gaussian blur
      → Uint8List → isolate boundary → Image.memory() → GPU

// AFTER
AssetEntityImage (150px thumbnail) → BackdropFilter (GPU shader) → GP compositor

The CPU no longer touches pixel data for previews.

With PhotoCard stateless, allowImplicitScrolling: true on the carousel became safe. Previously, implicit scrolling could wake up off-screen cards and trigger preview processing. After the pivot, the actual comment left in the carousel code reads:

// We don't need complex viewport logic anymore because
// PhotoCard is now just lightweight widgets.

The viewport tracking was deleted. The LRU cache was deleted. The _previewCache parameter that had been threading through EditorScreen into PhotoCarousel into every PhotoCard was gone.

Net diff of this single commit: +304 lines added, -783 lines deleted.

The optimized solution is less code!

Export Is Where Isolates Actually Belong

The GPU pivot fixed previews. Export is a genuine CPU job: decode, blur, composite, encode, write to disk.

Previews needed to get off the CPU entirely. Export needs to use the CPU better.

Batching Instead of Queuing

The original export loop was sequential. One photo at a time, fully awaited, then the next:

for (int i = 0; i < photos.length; i++) {
  final processedBytes = await _imageProcessor.processImage(asset, settings);
  await tempFile.writeAsBytes(processedBytes);
  await Gal.putImage(tempFile.path); // gal package — saves to device gallery
  yield i + 1;
}

This is safe, but it is also slow. On a 10-photo batch, every idle CPU core means more time the user spends staring at a progress bar.

The fix was controlled concurrency. Process three photos at a time with Future.wait, yield progress after each batch completes, then start the next:

final batchSize = 3; // currently hardcoded at 3

for (var batchStart = 0; batchStart < photos.length; batchStart += batchSize) {
  final batchEnd = (batchStart + batchSize).clamp(0, photos.length);
  final batch = photos.sublist(batchStart, batchEnd);

  final batchFutures = batch.asMap().entries.map((entry) async {
    final asset = entry.value;
    Uint8List? originBytes = await asset.originBytes;
    if (originBytes == null) return null;

    Uint8List? processedBytes = await _imageProcessor.processImage(
      originBytes, settings, isExportProcessing: true,
    );
    if (preserveMetadata) {
      processedBytes = _preserveMetadata(originBytes, processedBytes);
    }
    originBytes = null; // GC hint — more on this below

    final tempFile = File('${exportDir.path}/$exportFileName');
    await tempFile.writeAsBytes(processedBytes);
    await Gal.putImage(tempFile.path);
    await tempFile.delete();
    return globalIndex + 1;
  });

  final batchResults = await Future.wait(batchFutures);
  for (final result in batchResults) { yield result; }
}

Why three and not five or eight? Processing more full-resolution images concurrently means more multi-megabyte buffers alive simultaneously. Push it too far and the memory pressure starts to resemble the original preview freeze. Three was a conservative starting point rather than a benchmarked optimum, low enough to stay well clear of the memory pressure that caused the original freeze. It has held up across everything I tested, from a three-year-old midrange Pixel to a current flagship, and I have not had reason to push it higher.

Notice originBytes = null inside the batch closure. The export service loads the raw bytes, uses them for processing and metadata, then explicitly nulls the reference. This is a hint to the Dart garbage collector that the buffer is reclaimable. It does not guarantee immediate collection, but with three multi-megabyte buffers alive concurrently, narrowing the window each one stays pinned should reduce peak memory pressure.

Making Export Feel Instant

Even after batching, there was a UX problem. Tapping "Export" felt unresponsive. The screen would hang on the editor view for a beat before the processing screen appeared. I never fully isolated the cause. The symptom was obvious and the fix was straightforward.

Two things, back to back. First, emit a progress state at 0% before any work starts. This forces the UI to switch to the processing view immediately. Second, yield to the event loop before starting the export pipeline:

emit(PhotosProcessingState(
  current: 0,
  total: currentState.photos.length,
  backgroundType: currentState.settings.backgroundType,
));

await Future.delayed(const Duration(milliseconds: 50));

await for (final progress in _exportService.exportPhotos(...)) {
  emit(PhotosProcessingState(...));
}

The 50 ms delay gives the framework time to render the 0% state before the export pipeline starts competing for resources. With it, the processing view renders first and the heavy work runs behind it. The difference between "nothing happened" and "it is already working" is 50 milliseconds of intentional patience.

One More Pass: BackdropFilter vs ImageFiltered

BackdropFilter got the preview off the CPU, but it was not the cheapest way to do it. BackdropFilter blurs whatever is already painted behind it, which means the engine has to save a layer and read back the framebuffer contents underneath. That machinery exists for effects like frosted glass over something dynamic behind it. In PhotoCard, the only thing being blurred is the single AssetEntityImage sitting directly below it in the Stack. Nothing dynamic behind it, so BackdropFilter's actual capability, blurring arbitrary content behind it, was never being used.

ImageFiltered applies the same blur directly to its own child's paint pass, no readback required. I swapped it in and profiled both under the same stress test: eight full-range drags on the blur intensity slider, 90 sampled frames each, profile mode on the emulator. The average raster time was close: 2.56 ms for BackdropFilter against 2.09 ms for ImageFiltered. The gap that mattered was in the worst frames. BackdropFilter's slowest frame during the drag hit 29.49 ms, past the 16.67 ms budget for 60 FPS. ImageFiltered's worst frame stayed at 6.38 ms. The saveLayer and framebuffer readback that BackdropFilter needs are not a steady tax. They show up as occasional spikes, and a slider drag is exactly the kind of interaction that exposes them.

case BackgroundType.extendedBlur:
  return Stack(
    fit: StackFit.expand,
    children: [
      // Nothing sits behind this layer, so filter the image itself
      // rather than backdrop-filtering the stack.
      ImageFiltered(
        imageFilter: ui.ImageFilter.blur(
          sigmaX: settings.blurIntensity * 0.5,
          sigmaY: settings.blurIntensity * 0.5,
        ),
        child: AssetEntityImage(
          photo,
          isOriginal: false,
          thumbnailSize: const ThumbnailSize.square(150),
          fit: BoxFit.cover,
        ),
      ),
      Container(
        color: Colors.black.withOpacity(0.1),
      ),
    ],
  );

The Actual Lesson

I had done everything right. That was the problem.

The isolates were correct. The cache was correct. The viewport gating was correct. Each one was a legitimate fix for a real inefficiency. Competently implemented, well-reasoned, and aimed squarely at the wrong thing.

None of them asked the one question that actually mattered: does this work belong on the CPU at all?

The GPU compositor handles blur as a byproduct of normal rendering. The thumbnail is already there. The shader runs in place. The CPU was running a full Gaussian pipeline to produce output the widget tree was already generating for free, one frame at a time, without being asked.

Before reaching for isolates, before sizing your cache, before writing a single viewport check, figure out which layer the work actually belongs to.

Flutter already had it handled. I just had to get out of the way.

Building Flutter Apps That Stay Fast Under Real Workloads

Flutter performance problems are not always solved by adding more caching, concurrency, or processing logic. As this experience shows, the larger gain can come from identifying which part of the rendering architecture should handle the work and removing unnecessary processing from the path. For teams working through similar performance and architecture challenges, explore our Flutter app development services, which include application development, performance optimization, integrations, maintenance, and quality assurance.

Building Local LLMs Using Dart FFI And llama.cpp: Beyond Wrapper Packages

GeekyAnts India Pvt Ltd — Fri, 13 Nov 2026 07:35:00 +0000

Build local LLMs in Flutter with Dart FFI and llama.cpp, and see how native bridges, GGUF models, memory management, and token streaming enable private, on-device AI.

Introduction

When I first explored running local LLMs inside Flutter, I had a lot of questions:

Why are people using C++ libraries for AI?
Why not just write everything in Dart?
What exactly is llama.cpp?
What are .so, .dll, and .dylib files?
Why does Flutter suddenly need native code? If you're coming from a Flutter-only background, all of this initially feels very unfamiliar. Most Flutter developers rarely touch:
native memory
C/C++
shared libraries
compilers
ABI architectures
or systems-level programming

But once you understand the architecture, things become surprisingly logical. This article documents my journey building a proof-of-concept meeting summarizer with local LLM inference.

A Quick Disclaimer

This article is not intended to convince anyone to replace existing Flutter local AI packages.

In fact, if your goal is simply to integrate local AI into an application, a well-maintained package may be the fastest way to ship.

My goal here was different.

I wanted to understand what happens beneath those abstractions, how Flutter communicates with native AI runtimes, and what it takes to build that bridge myself.

That curiosity ultimately led me down the Dart FFI and llama.cpp rabbit hole.

POC support scope: The working LLM path in this repository currently targets Android arm64-v8a. The platform descriptions below explain how FFI libraries generally work; the other native llama bridge integrations like iOS or desktop OS’s are not implemented in this POC.

The Problem Statement

Before diving into implementation details, let's start with a simple question.

Why go through all the complexity of Flutter FFI and llama.cpp when we could simply host a GGUF model on a Python FastAPI server and consume it like any normal API?

That's exactly what I wondered initially.

The answer is that local AI solves a completely different problem.

Server-based AI depends on internet connectivity, introduces latency, incurs infrastructure costs, and sends user data to external servers

Local AI via llama.cpp runs the model directly on the device itself, enabling:

Offline LLM inference after the model has been downloaded
Better privacy because prompts and model output do not need to leave the device
Lower latency (no network round trips)
No backend costs (computation happens on user's device)

For applications like meeting summarization, note taking, personal assistants, or enterprise environments with strict privacy requirements, these benefits can be extremely valuable.

That said, local AI isn't always the right choice.

If your application requires very large models, centralized company knowledge, or continuous model updates, a cloud-based architecture may still make more sense.

The important thing is understanding the trade-offs and choosing the right approach for your use case.

Why Are AI Libraries Usually Written in C++?

As Flutter developers, we're used to solving everything in Dart. So why do most AI runtimes seem to be written in C++, Rust, CUDA, or other low-level technologies?

The answer is performance.

Large Language Models spend most of their time performing:

Matrix multiplications
Tensor operations
Memory-intensive computations
CPU and GPU optimizations

Languages like Dart are excellent for application development, but they're not designed for squeezing every bit of performance from modern hardware.

C++ gives developers direct control over:

Memory management
CPU instructions
Threading
SIMD optimizations
GPU integrations

That's why projects such as TensorFlow, PyTorch, ONNX Runtime, and llama.cpp all rely heavily on native code under the hood. Even many Python AI frameworks are actually thin wrappers around C++ implementations.

What Is llama.cpp?

At a high level, llama.cpp is an inference engine for running Large Language Models locally. Think of it as an AI inference engine optimized for local hardware.

It takes:

Model weights (the trained parameters from the LLM, in GGUF format)
Tokens (numerical representations of text)
Prompts (user input)

And performs all the mathematical operations required to generate text.

Some reasons llama.cpp has become so popular include:

Runs directly on CPUs
No Python dependency
Optimized for mobile and embedded devices
Supports quantized models

Cross-platform support

Understanding GGUF Models

When working with llama.cpp, you'll frequently come across GGUF files.

GGUF (GGML Universal File) is a model file format designed to store model tensors and metadata for efficient loading by GGML-based inference engines.

GGUF can store a model’s weights in different levels of precision. Quantization is the process of making those weights smaller by reducing their precision. So, GGUF can store either the original high-precision weights or quantized versions like Q4.

For example, Q4 quantization can reduce a 1B model from roughly 2 GB to roughly 800 MB on disk while maintaining acceptable quality i.e., small enough to be practical on some phones.

Without formats like GGUF, running local LLMs on mobile devices would be far less practical.

One thing I learned quickly is that choosing the right model is often more important than choosing the right framework.

A smaller, well-quantized model that runs smoothly on target devices is usually more valuable than a larger model that struggles with memory and performance constraints.

Existing Flutter Packages for Local AI

Before building my own integration, I explored some of the existing Flutter packages available for running local models. Some examples include:

flutter_gemma
fllama
llama_cpp_dart

These alternatives can be a great starting point if your goal is simply to get local AI working quickly. So why didn't I just use one of them?

For this project, my goal wasn't only to run a model locally. I wanted to understand:

How models are loaded
How memory is managed
How Dart communicates with native code
How token streaming works
How worker isolates fit into the picture
What actually happens inside the FFI layer

Building the bridge myself gave me a much deeper understanding of the architecture and helped me appreciate the amount of work these packages abstract away.

If your objective is shipping a feature quickly, evaluating existing packages first is absolutely a sensible approach.

If your objective is learning how local AI actually works inside Flutter, building a small FFI bridge yourself is an incredibly valuable exercise.

Flutter + llama.cpp Architecture

Each layer has a specific responsibility.

Flutter - Handles UI, state management, and user interactions.
Provider / Services - Contains application logic and coordinates model operations.
Worker Isolate - Keeps heavy inference work away from the UI thread.
Dart FFI - Acts as the bridge between Dart and native code.
Native Bridge - Exposes a stable C API that Flutter can call.
Llama.cpp - Loads models and performs inference.
GGUF Model - Contains the trained weights that generate responses.

Understanding Native Libraries

One thing that confused me initially was seeing files like:

libllama_bridge.so

llama_bride.dll

libllama_bridge.dylib

When we used to install games on Windows, many saw files like: game.dll, audio.dll, etc.

Those .dll files are compiled native libraries. They contain optimized code for:

Rendering
Audio processing
Physics simulation
Networking

The main game executable loads them and uses their functionality.

Platform-Specific Names

Flutter integrations using llama.cpp can follow the same pattern:

Instead of physics.dll, a bridge might be packaged as:

Android/Linux: libllama_bridge.so (Shared Object)
macOS/iOS: Symbols linked directly into the process or a .dylib where platform rules allows it
Windows: llama_bridge.dll

The native library does the heavy work, and Flutter communicates with it via FFI.

Please note that in our POC we have only covered Android arm64 build configs.

How Flutter Talks to C++: Dart FFI

This is where Dart FFI comes in.

FFI stands for: Foreign Function Interface

Which means: "Allow Dart to call functions written in another language." At the Dart-to-native boundary, FFI can call native C APIs without:

Flutter Platform channels
Platform-channel message serialization or
Message passing

[Official Dart FFI Documentation](https://dart.dev/interop/c-interop)

FFI vs Platform Channels: Why Not Platform Channels?

Flutter already provides Platform Channels, so why introduce FFI?

The answer comes down to performance.

AI workloads involve:

Frequent native calls
Large amounts of memory
Continuous token generation

FFI allows Dart to call native functions directly without serialization overhead. A simple FFI binding looks like this:

DynamicLibrary.open() only loads the library. lookupFunction() resolves an exported C symbol and converts it to a callable Dart function. Only then can Dart invoke the native bridge. The repository collects these typed lookups in _BridgeBindings.

Key Concepts That Finally Made Everything Click

Opaque Pointers

One concept that initially felt strange was opaque pointers.

Instead of exposing the actual C++ model object to Dart, native code simply returns a handle.

Dart doesn't know what's inside the session.

It simply passes the handle back whenever it wants to load a model, generate tokens, or destroy the session.

This keeps the object layout and memory management on the native side. It also creates an explicit ownership contract: every successful bridgeSessionCreate() must eventually be paired with bridgeSessionDestroy(), and pointers returned by allocating native functions must be released with the matching native free function.

Streaming Callbacks

One of the coolest parts of the implementation was token streaming.

Instead of waiting for the entire response, llama.cpp generates tokens one by one.

Every generated token flows back into Flutter through this callback. Pointer.fromFunction() converts a Dart callback into something the native C code can call. The userData pointer tells the callback which token stream to send the generated tokens to, so the tokens can appear one by one in the typing effect.

One important limitation is that the callback must run on the same Dart isolate thread. In this POC, that works because generation and the callback happen on the same worker thread. If the native library calls the callback from a different thread, we should use NativeCallable.listener instead.

Worker Isolates

AI inference is computationally expensive.

Running inference on Flutter's main Isolate would quickly make the UI feel sluggish or frozen. To avoid that, I moved all native interactions into a dedicated Worker Isolate.

The UI remains responsive while the model generates tokens in the background.

The Isolate lets the main Dart thread and the worker communicate by sending messages. The main isolate sends a command to the worker, the worker runs the blocking native/FFI work, and then sends back token, completion, or error messages. FFI talks directly to the native code, avoiding platform-channel overhead, while the Isolate keeps the heavy work away from the UI thread.

The POC I Built - Meeting Summarizer

Architecture

The proof of concept I built follows a fairly simple flow.

The user records a conversation.

The app has two transcription options: recorded audio uses local Whisper, while live speech uses the device’s speech recognition. It prefers offline recognition but falls back to online recognition if offline isn’t supported.

That transcript is then passed to the local LLM through the FFI bridge.

As the model generates a summary, tokens are streamed back to Flutter and displayed in real time.

The LLM inference process happens locally on the device.

No server required for LLMs.

Preview of what I build

Challenges I Faced

While the final architecture looks clean, getting there wasn't always smooth.

Model Downloading and Distribution

One thing that surprised me initially was that llama.cpp doesn't ship models - you download GGUF files, store them, and verify checksums. Model management quickly becomes a product concern rather than just an engineering concern.

This was probably one of the first moments where I realized local AI introduces a completely different set of challenges compared to calling a hosted API.

Memory Consumption

The Llama 3.2 1B Q4_K_M model file used in this POC is roughly 800 MB. Total runtime memory is higher and varies with context size, KV cache, runtime buffers, backend allocations, and device. Bigger isn’t better on a phone. Choosing a model becomes a balancing act between:

Quality
Speed
RAM usage
Storage size

FFI Debugging Can Be Painful

A normal Dart error usually gives a helpful stack trace.

A native crash often gives something like:

Segmentation Fault

And that's it.

Most of my debugging time was spent tracking:

Invalid pointers
Memory ownership issues
Double frees
Native lifecycle bugs

FFI is powerful, but it also removes some of the safety nets Flutter developers are used to.

Device Variability

Fast on one phone, slow on another. Testing across multiple devices becomes extremely important when working with local AI.

Local AI Is Not Always Better

This was perhaps the biggest lesson.

It's easy to get excited about running models completely offline.

But local AI isn't a universal replacement for cloud AI.

Cloud-based solutions still have advantages:

Access to larger models
Easier updates
Centralized knowledge
Less device storage usage

The best solution depends entirely on the problem you're trying to solve.

When Would I Actually Use This?

One question I kept asking myself throughout the project was: "This is cool, but would I use it in a real application?"

I think the answer depends on the use case.

Great Candidates

Meeting summarization
Personal journals
Offline assistants
Enterprise privacy-focused applications
Travel apps with limited connectivity

Probably Better on the Cloud

Large-scale document analysis

Complex RAG systems

Company-wide knowledge assistants
Very large models requiring significant compute

Local AI is powerful, but it isn't a universal replacement for cloud-hosted AI.

Project Repository

The complete source code for this proof of concept is available on GitHub:

GitHub Repository:

GitHub Link to Project

The repository contains:

Flutter application code
Dart FFI bindings
Native C/C++ bridge implementation
Worker isolate architecture
Token streaming implementation
Model loading lifecycle management
Meeting summarization proof of concept

If you're interested in exploring the implementation details beyond what is covered in this article, feel free to check out the repository.

Key Learnings From This Project

Memory Management Matters

Dart uses garbage collection; native C/C++ memory must be released manually to avoid leaks and crashes.

Isolates Are Essential

Inference can take several seconds depending on the model and device. Running everything on the UI isolate is not a realistic option.

Streaming Creates Better UX

Showing tokens as they're generated makes the application feel much faster and more responsive.

Native Boundaries Should Be Simple

Keeping a small, stable API between Dart and C++ significantly reduces debugging complexity.

Don't Fight Existing Ecosystems

Building an inference engine from scratch would be an enormous undertaking.

Using llama.cpp allowed me to focus on application development rather than reinventing years of optimization work.

Conclusion

Before starting this project, Flutter FFI and local AI felt intimidating.

There were many unfamiliar concepts, from native libraries and memory management to callbacks and worker isolates.

But once I understood the architecture, it became clear that each layer has a very specific purpose.

The stack I ended up with looks like this:

Flutter for UI
Dart for application logic
FFI for interoperability
C++ for performance-critical operations
llama.cpp for inference
GGUF models for local execution

The result is a working POC on android arm64: speech -> transcript -> streamed local summary, with LLM inference that runs directly on the device without depending on cloud infrastructure.

The learning curve is definitely steeper than consuming a REST API, but the capabilities you unlock - private, offline, low-latency AI running entirely on a user's device - make the journey worthwhile.

Building this POC showed how much Flutter can do when it reaches beyond Dart into native code.

Resources

GitHub Link to Project

Dart FFI Documentation

llama.cpp Repository

GGUF Model Format

Flutter Native Integration

Provider Package

Automation Testing With Playwright Using JavaScript

GeekyAnts India Pvt Ltd — Fri, 06 Nov 2026 05:10:44 +0000

Automation testing is a critical part of software development that helps ensure the quality and reliability of your application. Playwright is a robust end-to-end testing framework that enables you to automate browser actions and interactions. This comprehensive guide will walk you through performing automation testing using Playwright in a Next.js application.

Playwright is a JavaScript library that provides a high-level API to automate browsers like Chromium, Firefox, and WebKit. It allows you to write browser automation scripts that simulate user interactions, navigation, form submissions, and more. Playwright's unique architecture provides a fast and reliable way to test web applications.

Setting Up a Javascript Project

Before we start writing tests, let's set up a Next.js project:

Create a new Javascript project:

mkdir my-js-project
cd my-js-project
npm init -y

This will create a new directory called my-js-project, navigate into it, and initialize a new Node.js project with default settings.

The -y flag in the npm init -y command is a shorthand for "yes". It automatically answers "yes" to all the prompts that npm init would normally ask you, thereby accepting all the default values for the project setup.

Install the Playwright dependencies:

npm install playwright

Writing Automation Tests with Playwright

Let's create a simple automation test for a javascript application. We want to test the navigation from the home page to a blog post page.

const { chromium } = require('playwright');

describe('Next.js Blog Automation Test', () => {
  let browser;
  let page;

  beforeAll(async () => {
    browser = await chromium.launch();
  });

  beforeEach(async () => {
    page = await browser.newPage();
  });

  afterAll(async () => {
    await browser.close();
  });

  it('should navigate to a blog post', async () => {
    await page.goto('http://localhost:3000');
    await page.click('a[href="/blog"]');
    await page.waitForSelector('h1');
    const pageTitle = await page.textContent('h1');
    expect(pageTitle).toContain('Blog Post Title');
  });
});

Let us see the components of the above code in a bit more detail.

When using testing frameworks like Jest, Mocha, Jasmine, or Playwright, the describe block is a fundamental organizational structure that helps group related test cases together. It provides a way to define a logical container for a set of tests with a common context or functionality. The describe block is primarily used to improve your test suite's organization, readability, and maintainability.

The beforeEach function is a part of testing frameworks like Jest and is used to set up a consistent and controlled environment before running each individual test case within a describe block. This is particularly helpful in maintaining a clean state and avoiding interference between test cases.

In testing, it's important to create a controlled environment for each test case to ensure that they run independently and reliably. Sometimes, tests require the same setup steps before execution. Instead of repeating the setup code in every test case, you can use beforeEach to define those common setup actions.

In automated testing, especially when using testing frameworks like Jest, you often encounter lifecycle functions that help you set up and tear down certain conditions before and after tests are run. One important lifecycle function is beforeAll.

The beforeAll function executes a block of code before all the tests within a test suite (also known as a test file or describe block) are run. This block of code typically sets up any common resources or configurations required for all the tests in that suite.

The afterAll function is a part of testing frameworks like Jest, and it is used to perform cleanup tasks or teardown operations after all the test cases within a describe block have been executed. It's the counterpart of the beforeAll function, which is used for setup operations before the test cases.

In testing, there might be scenarios where you need to perform cleanup tasks or reset the environment after all the test cases within a particular test suite have run. afterAll allows you to define these cleanup actions.

Enhancing API Testing with Playwright

Did you know that Playwright, commonly recognized for its powerful browser automation capabilities, can also be utilized for API testing? This versatile tool excels in web application testing and offers robust features to automate and validate API interactions. By leveraging Playwright for API testing, you can streamline your testing processes and ensure your APIs function as expected.

Writing Your First API Test

Let's start with a basic example. We'll test a simple REST API that returns user information.

Start with creating a new file called apiTest.spec.js.
Import Playwright:

const { test, expect } = require('@playwright/test');

Write a Basic Test:

Here's a test that fetches user data from a hypothetical API endpoint.

test('GET user data', async ({ request }) => {
  const response = await request.get('https://jsonplaceholder.typicode.com/users/1');
  expect(response.status()).toBe(200);
  const userData = await response.json();
  expect(userData.id).toBe(1);
  expect(userData.name).toBe('Leanne Graham');
});

Breaking Down the Test

test: Defines a test case.
request.get: Sends a GET request to the specified URL.
expect: assertions to check if the response meets our expectations.
response.status(): Checks the HTTP status code of the response.
response.json(): Parses the response body as JSON.

Handling Different HTTP Methods

Playwright supports various HTTP methods. Here are examples of POST, PUT, and DELETE requests.

POST Request:

test('POST create new user', async ({ request }) => {
  const response = await request.post('https://jsonplaceholder.typicode.com/users', {
    data: {
      name: 'John Doe',
      username: 'johndoe',
      email: 'johndoe@example.com'
    }
  });
  expect(response.status()).toBe(201);
  const newUser = await response.json();
  expect(newUser.name).toBe('John Doe');
});

PUT Request:

test('PUT update user data', async ({ request }) => {
  const response = await request.put('https://jsonplaceholder.typicode.com/users/1', {
    data: {
      name: 'Jane Doe'
    }
  });
  expect(response.status()).toBe(200);
  const updatedUser = await response.json();
  expect(updatedUser.name).toBe('Jane Doe');

});

DELETE Request:

test('DELETE user', async ({ request }) => {
  const response = await request.delete('https://jsonplaceholder.typicode.com/users/1');
  expect(response.status()).toBe(200);
});

Handling Authentication

Many APIs require authentication. Here’s how to handle it in Playwright:

test('GET user data with authentication', async ({ request }) => {
  const response = await request.get('https://api.example.com/protected-resource', {
    headers: {
      'Authorization': `Bearer YOUR_ACCESS_TOKEN`
    }
  });
  expect(response.status()).toBe(200);
});

Using Fixtures for Setup and Teardown

Fixtures help set up and tear down resources required for your tests. Here's an example:

test.describe('API tests with setup and teardown', () => {
  let userId;

  test.beforeAll(async ({ request }) => {
    const response = await request.post('https://jsonplaceholder.typicode.com/users', {
      data: { name: 'Test User' }
    });
    const user = await response.json();
    userId = user.id;
  });

  test.afterAll(async ({ request }) => {
    await request.delete(`https://jsonplaceholder.typicode.com/users/${userId}`);
  });

  test('GET created user data', async ({ request }) => {
    const response = await request.get(`https://jsonplaceholder.typicode.com/users/${userId}`);
    expect(response.status()).toBe(200);
  });
});

Running Your Tests

You can run your Playwright tests using the Playwright Test Runner. Add a script to your package.json:

"scripts": {
  "test": "playwright test"
}

Then, run your tests with:

npm test

Generating Test Depictions

Playwright Test generates reports by default. You can view the test results in the console, but you might want a more detailed report. Playwright provides a built-in HTML report. To enable it, add a playwright.config.js file in your project root:

module.exports = {
  reporter: [
    ['list'],
    ['html', { outputFolder: 'playwright-report', open: 'never' }]
  ],
};

Run your tests again.

After completing the tests, you'll find the HTML report in the playwright-report directory. To open it, run:

npm playwright show-report

This command opens the HTML report in your default browser, providing a detailed view of your test results.

Best Practices for Automation Testing

Here are some best practices to follow when writing automation tests using Playwright in a Next.js application:

Isolation: Each test should be independent and not depend on the state of other tests.
Use Data Attributes: Use data attributes in your components to make it easier to select elements in your tests.
Mock API Calls: Use mocking or stubbing to avoid making actual API requests during tests.
Assertions: Use meaningful assertions to verify the expected behavior of your application.
Page Objects: Create page objects or helper functions to encapsulate the interactions with specific parts of your application.

Continuous Integration with Playwright

Integrate your Playwright tests into your continuous integration (CI) pipeline. This ensures that your tests are automatically executed whenever you push code changes.

Choose a CI platform like GitHub Actions, Jenkins, or Travis CI.
Configure the CI pipeline to run the Playwright tests using Jest.

Playwright vs. Selenium: A Comparison

Feature	Playwright	Selenium
Language Support	Javascript/TypeScript, Python, C#, Java	Java, C#, Python, Ruby, JavaScript, Kotlin
Browser Support	Chromium (Chrome, Edge), WebKit (Safari), Firefox	Chrome, Firefox, Safari, Edge, IE, Opera
Installation and Setup	Single package installation (includes browser binaries)	Separate installations for WebDriver binaries
Speed and Performance	Generally faster (uses DevTools Protocol)	Slower (uses WebDriver protocol)
Parallel Testing	Built-in support for parallel testing	Requires additional configuration/tools (e.g., Selenium Grid)
APIs and Features	Modern API, auto-wait, network interception, mobile emulation	Mature API, similar features but more configuration
Community and Support	Growing community, backed by Microsoft	Established community, extensive documentation
Cross-Browser Testing	Consistent results with the same API for all browsers	It is possible but may have inconsistencies
Test Reporting	Integrated reporting tools	Requires additional tools/libraries (e.g., TestNG, JUnit)
Script Debugging	Built-in tools for debugging (e.g., Playwright Inspector)	Debugging through language-specific IDE/tools

Conclusion

Automation testing using Playwright in a Next.js application can help you catch bugs and ensure the quality of your software. Following the steps in this guide, you've learned how to set up a Next.js project, write automation tests, and integrate them into your CI pipeline. Remember to continuously update and expand your test suite as your application evolves.

Master Cursor Custom Rules to Align Gen AI with Your Code

GeekyAnts India Pvt Ltd — Thu, 05 Nov 2026 06:21:15 +0000

Learn how to use Cursor Custom Rules to guide AI-assisted coding. Explore project rules, .mdc files, setup steps, practical use cases, and best practices.

Editor’s Note: This article was originally written by Ajinkya Vinayak Palaskar, Software Engineer III, and published in May 2025. It was revised and updated by Harrini in October 2026 to reflect the latest Cursor Custom Rules features, rule formats, configuration options, and AI-assisted development practices. The technical content was reviewed by Ajinkya Vinayak Palaskar, Software Engineer III.

Generative AI tools like Cursor are changing the way developers write code, but let’s be honest, the default AI behavior doesn’t always match how you or your team builds software. Whether it’s naming conventions, project structure, or the way you wire up API calls, out-of-the-box AI can feel like working with a junior dev who doesn’t quite get the vibe yet.

That’s where Cursor’s Custom Rules come in. Instead of adapting your codebase to fit AI’s suggestions, you can flip the script and make AI generate code that follows your standards, your structure, and your expectations.

In this blog, we’ll dive into what Cursor’s rules are, how they work, and how you can use them to bring structure, consistency, and actual team alignment to your AI-powered workflow. Just real, dev-focused examples that help you go from “AI that kind of helps” to “AI that codes like your best junior dev (who doesn’t forget lint rules).”

Understanding Cursor’s Custom Rules

At its core, Cursor’s Custom Rules feature gives developers a way to shape the behavior of the AI assistant so it actually respects your coding preferences. It’s not just about giving it tips; it’s about creating structured, repeatable rules that get applied every time you prompt it in certain files or folders.

Cursor supports two types of rules:

Project Rules: These are project-specific rules stored in .cursor/rules/ and committed to version control. They apply to everyone working in the repo, super useful for enforcing team-wide patterns, boilerplate structures, or naming conventions.
User Rules: These live locally and are scoped only to your Cursor environment. Think of them as your personal tweaks or power-ups for one-off or experimental cases. Cursor stores them in your user settings directory.

To sum it up: Project rules = team alignment. User rules = personal productivity boosts.

How Do These Rules Actually Work?

Cursor rules define instructions that guide an agent when it works on a project. Depending on how a rule is configured, it can be applied automatically based on the task, to specific files, or manually when selected.

Rules can include:

Description: Explains what the rule is for.
File patterns: Define which files the rule targets when using file-specific rules.
Instructions: Tell an agent how to work with the code.
File references: Add project context with references such as @src/lib/validators.ts.

Cursor supports four application modes: Always Apply, Apply Intelligently, Apply to Specific Files, and Apply Manually. This lets teams choose whether an instruction should be active everywhere, selected based on relevance, tied to particular files, or triggered explicitly.

One important distinction is that cursor rules guide the agent. They do not automatically apply to every AI feature in Cursor, such as tab completion or inline edit. Rules should therefore complement tests, linting, type checking, CI, and code review rather than replace them.

So instead of writing the same prompt over and over again, like "Please generate a Zod schema and name everything in camelCase," you just write the rule once, and Cursor applies it automatically in the right places.

What About AGENTS.md?

Cursor also supports AGENTS.md for project-level instructions. It can be useful when a project needs straightforward guidance without the more granular scoping available through .cursor/rules/*.mdc. For file-specific or manually triggered instructions, project rules provide more control.

Setting Up Custom Rules in Cursor

Now that we know what rules are and why they matter, let’s get into how to actually set them up. The cursor makes this surprisingly smooth. You don’t need any weird config voodoo, just a well-structured JSON file, a clear prompt, and you’re good to go.

The easiest way to get started is to use the built-in command:

To create a rule, open Customize → Rules → Add Rule in Cursor. Developers can also use /create-rule in Agent to create a rule from the current conversation. The command palette remains another way to access rule creation, with shortcuts varying by operating system.

This will prompt you for a name for the rule and create a markdown file at this location:

.cursor/rules/your-new-rule.mdc
Here’s what a full rule looks like:

description: Enforce Zod schemas in service layer
globs: ["src/services/**/*.ts"]
alwaysApply: true
referencedFiles: ["src/lib/validators.ts"]

When working in service files, always use Zod to validate inputs and outputs. Follow the existing pattern from validators.ts. Don't skip validation unless explicitly told to.

Let’s break this down:

description: Short explanation of what the rule does
globs: File patterns where the rule should apply (e.g., **/*.ts)
alwaysApply: If true, the rule is used without needing manual selection
referencedFiles: (Optional) Files used as examples or context for better AI responses
Body: The actual instructions shown to the AI when working in matching files.

To manage rules,

Open Customize → Rules in Cursor. From there, rules can be added, edited, and managed by scope.
You’ll see both User Rules and Project Rules
Toggle, delete, or update them as needed

Now that you have got the structure down, let’s look at how developers can use Cursor Custom Rules to handle real-world problems.

Use Case 1: Enforcing Consistent API Validation with Zod The Problem

Your team uses Zod for request/response validation across all service files, but devs often forget to include schemas or name things consistently.

Rule Example

description: Enforce Zod schemas in service layer
globs: ["src/services/**/*.ts"]
alwaysApply: true
referencedFiles: ["src/lib/zod-templates.ts"]

Always use Zod to validate both request and response payloads. Reuse schema patterns from zod-templates.ts. Do not allow untyped or loosely typed data to go through the service layer.

This makes the AI generate pre-validated, strongly typed service methods every time, no reminders or rework needed.

Use Case 2: Enforcing Consistent File Structure for Features

The Problem

Every new feature in your app should follow a clean structure like index.tsx, hooks.ts, types.ts, and api.ts, but different devs often structure things differently.

Rule Example

description: Enforce file structure for new feature folders
globs: ["src/features/**/index.tsx"]
alwaysApply: true

When generating or modifying a feature in src/features, make sure the folder contains:

index.tsx as the entrypoint
hooks.ts for all custom hooks
types.ts for local types
api.ts for all API calls

Follow existing naming and organization patterns.

With this rule, when the rule is active in Cursor Agent, it gives the agent consistent instructions for generating or modifying code. It does not automatically govern every AI feature in Cursor.

Use Case 3: Enforcing Unit Test Coverage with Vitest

The Problem\
You're using Vitest, and every utility function should have a corresponding test file. Devs often skip writing them.

Rule Example

description: Ensure utility functions are covered by Vitest
globs: ["src/utils/**/*.ts"]
alwaysApply: true

Whenever a new utility function is created or modified, also create a corresponding .test.ts file using Vitest. Follow the structure and naming conventions used in existing test files.

With this rule in place, AI writes tests along with your utilities, improving test coverage and helping juniors not skip QA steps.

Use Case 4: Auto-Wiring RPC Handlers with tRPC

The Problem

You're using tRPC, and you want every new route to follow a specific handler format with proper input/output typing.

Rule Example

description: Standardize tRPC handler creation
globs: ["src/server/api/**/*.ts"]
alwaysApply: true

When generating new procedures, use z.infer for input/output types. Use the protectedProcedure or publicProcedure wrapper from the existing router setup. Refer to src/server/api/_utils.ts for common logic.

With this, AI consistently generates boilerplate that aligns with your tRPC config without you needing to manually adjust every time.

Best Practices & Tips

Getting started with Custom Rules is easy, but getting them to stick and scale well with your team takes a bit of finesse. Here are some battle-tested tips to help you make the most of them:

Make Rules Iterative, Not Perfect

Don’t try to write the ultimate prompt on Day 1. Start small; even a 1-liner like “Use Zod in all services” can go a long way. Watch how Cursor responds, and improve the rule over time.

Think of rules like code: ship early, refine often.

Be Specific in the Prompt

Don’t just say “use tests"; say “use Vitest and place tests in the tests folder next to the source file." The more specific the language, the more reliable the output.

Use phrasing like "Always start with...”, “Never skip...”, “Follow the pattern from...", etc.

Review Rules as a Team

If you’re on a team, treat your rules like coding conventions. Do quick async reviews, and align on when a rule should apply (alwaysApply: true vs. manual). This helps avoid confusion or overreach from the AI. Tests, linters, type checks, CI pipelines, security checks, and code reviews should enforce requirements that need deterministic validation.

Keep It Human

Don’t over-engineer your prompts. Write it like you’re telling a junior dev sitting next to you. That’s usually the sweet spot.

Final Thoughts

Custom rules in Cursor are low-effort, high-impact. With just a few Markdown files, you can guide AI to follow your team’s coding style, enforce structure, and even auto-suggest best practices, all without writing extra logic or docs.

I have found that whether you’re working solo or on a big team, these rules let you offload the repetitive reminders and keep your codebase clean and consistent.

If you’ve ever wished AI could “just know how we do things around here,” this is how you get there.

FAQs

Are Cursor Custom Rules still supported in 2026?

Yes, Cursor Custom Rules are still supported in 2026. They provide the agent with persistent instructions for coding standards, project patterns, workflows, and other development preferences. Project rules are stored as .mdc files in .cursor/rules/, while user rules apply across projects through Cursor’s customization settings.

How can Cursor Rules reference project files?

Cursor Rules can reference project files with @filename so Agent can use them as additional context. For example, a rule can reference @src/lib/validators.ts when asking an agent to follow an existing validation pattern. This helps keep generated code aligned with established project conventions.

When should we apply a cursor rule?

A cursor rule is useful when an instruction is needed only for certain tasks but should not be tied to specific files. Cursor uses the rule’s description to decide when it is relevant, which makes this approach useful for project patterns, workflows, or domain-specific guidance.

Automate the Obvious, Review the Rest: Designing AI-Powered Content Moderation

GeekyAnts India Pvt Ltd — Thu, 29 Oct 2026 03:33:21 +0000

Introduction

A social product does not fail only when the database is slow. It also fails when the feed becomes a place users no longer trust.

We needed a moderation system that could inspect captions, comments, ads, stories, images, and videos as soon as they were created without delaying uploads or posts for human review or automating every borderline case.

The design we shipped is deliberately split. Text goes through NestJS, Hasura event triggers, BullMQ, and the OpenAI Moderation API. Images and videos go through S3, SNS, SQS, AWS Lambda, and Amazon Rekognition. Both pipelines write the same status model, store the AI evidence, and leave gray-area content for the admin panel.

Moderation is not a feature bolted onto the feed. It is a gate the feed has to pass through.

Why Traditional Moderation Doesn't Scale

As content volume grows, moderation becomes less about identifying whether a single piece of content is safe and more about deciding where each piece should go.

Sending every piece of content to a human does not scale, while automatically approving or rejecting everything based on a single model score introduces its own risks.

The system therefore treats moderation as a triage pipeline: clear cases are automated, ambiguous cases are escalated, and the feed only exposes content that has reached an approved state.

Why Content Moderation Is Required

User-generated content is the product. It is also a risk. Without a consistent review path, unsafe or policy-breaking material can reach other users in seconds.

Protect people on the platform from explicit, violent, hateful, or otherwise harmful material.
Keep posts, ads, and stories out of public feeds until they are safe to show.
Meet App Store, Play Store, and trust-and-safety expectations for UGC products.
Create an audit trail: what the model saw, what score it produced, and who made the final call.
Notify the owner when content is rejected, instead of failing silently.

Manual review of every caption and every frame does not scale.A fully automated approve-or-reject model also creates risk: models are confident on extremes and uncertain in the middle. The system is built around that fact.

What the System Does

Every piece of moderated content starts as pending. Background work then produces a decision: approved, under_review, or rejected.

Approved content can appear in feeds and listings. Rejected content stays out of the public product. Under-review content stays out of the public product until a moderator confirms or overrides the AI decision. That is the contract.

The Architecture: Two Pipelines, One Status Model

We did not force text and media through the same worker. Text is small, frequent, and already in PostgreSQL at the moment it is written. Media is large, arrives in S3, and is expensive to inspect. Each path uses the infrastructure that fits it.

Shared data model content_moderation_status on the source row: pending, approved, under_review, rejected. A content row as the generic moderation record. moderation_results for provider, risk_score, labels, ai_decision, and final_decision. reviewed_by on the result row when a human takes the last step. Feeds that list posts, ads, stories, or media filter on content_moderation_status = approved. The existing status columns (active, trash, uploaded, and so on) still mean what they always meant. Moderation is a separate lifecycle on purpose.

Do not overload “active” to mean “safe.” Safety needs its own column.

Text Moderation: Hasura, BullMQ, and OpenAI

When a user writes text, the row is saved first. The HTTP request does not wait for OpenAI. Hasura fires an event; NestJS creates the content record and enqueues a job; a worker calls the model.

Text request path

Insert / update row
↓
Hasura event trigger
↓
POST /v1/event/content-moderation
↓
Create content row
↓
BullMQ job MODERATE_TEXT
↓
OpenAI /v1/moderations
↓
Save result + update status

The worker

The content-moderation processor runs with limited concurrency and a rate limit so OpenAI traffic stays controlled. Empty text is approved without a call.

Failed jobs go to a dead-letter queue. Hasura retries the webhook (three attempts, ten-second interval, sixty-second timeout) if Nest is briefly unavailable.

How Text Scores Are Calculated

OpenAI returns a flag plus a category_scores map, each value between 0 and 1. We do not treat the boolean flag as the decision. We convert the highest category score into a 0–100 risk score and classify from that number.

Text scoring

OpenAI category_scores
→
Take the maximum score
→
riskScore = round(max × 100)
→
Compare to thresholds
→
approved / under_review / rejected

Risk below 40 → approved. The content can enter the feed.
Risk from 40 up to (but not including) 75 → under_review. A human must decide.
Risk 75 or above → rejected. The content stays hidden and the owner is notified.

Only categories that meet or exceed the approved threshold are stored as labels, as percentages. The admin later sees the violated categories, not a wall of near-zero scores. Thresholds are configurable through MODERATION_APPROVED_THRESHOLD and MODERATION_REJECTED_THRESHOLD; the production defaults are 40 and 75.

How We Calibrated the Thresholds

Scores of 40, 75, or 90 only become production policy after you measure what those cutoffs do to false positives, false negatives, and human workload. The value of these thresholds comes from the calibration loop that produced and maintains them.

Starting points, not final answers

We began with provider semantics and product risk posture, then validated against labelled samples rather than shipping intuition.

Text (OpenAI): start near the middle of the score distribution. Auto-approve only when the max category score is clearly low; auto-reject only when it is clearly high; leave a wide under_review band for the first weeks.
Images/video (Rekognition): start stricter on auto-approve than text (review at 75%, reject at 90%), because a false approval in a feed is costlier than an extra item in the admin queue.
Environment variables keep cutoffs changeable without rewriting the decision engine.

Labelled test set and threshold sweeps

Before locking production defaults, we built a small labelled set covering three buckets: clearly safe, borderline / ambiguous, and clearly violating. For each item we stored the model’s raw scores, the would-be decision at candidate cutoffs, and a human gold label.

Calibration experiment

Collect labelled sample (safe / borderline / violate)
↓
Run OpenAI or Rekognition scores
↓
Sweep candidate thresholds
↓
Measure auto-approve FP, auto-reject FN, review volume
↓
Pick cutoffs that protect the feed without drowning admin
↓
Re-check against human final_decision in production

What “good” looked like for us

Near-zero false auto-approves on the labelled violate set (missed harm is the expensive error).
Bounded false auto-rejects: overturns in admin or appeals should stay rare; if many auto-rejects become approved, raise the reject line or widen review.
A review band that is large enough to catch ambiguity but small enough that queue age stays within SLA.
We treat the AI-human disagreement rate on under_review items as a first-class metric rather than noise.

Text settled at 40 / 75 after sweeps showed that scores below ~40 almost never carried human-confirmed violations in our sample, while scores at or above ~75 almost always did. The middle band produced the majority of human overrides, which is exactly where we want people in the loop.

Media settled at 75 / 90 based on the same calibration process but with a different cost curve: Rekognition confidences in the mid-70s were often ambiguous on a single frame, so we refuse to auto-approve there. Sustained or high-confidence hits (including the video frame policy) justify auto-reject without waiting for a moderator.

Thresholds are hypotheses. Production disagreement rates are the experiment.

We revisit cutoffs when operational signals move: rising appeals after auto-reject, reports against content that was auto-approved, or a review queue that ages past SLA. Calibration is continuous, not a one-time spreadsheet.

Image and Video Moderation: S3, Lambda, SQS, Rekognition

The media never waits in the Nest process for Rekognition. The user uploads to S3 under users/uploads/. An ObjectCreated notification is published to SNS, and the media path continues asynchronously.

Why not two Lambdas on one SQS queue

SQS is a competing-consumer queue: each message is delivered to exactly one consumer. If two Lambda event-source mappings share a single queue and each “filters” by message shape, a message can land on the wrong function. That function may skip and still succeed—deleting the message—or fail and retry forever. Either way, delivery semantics are not safe for mixed payload types.

We therefore do not rely on two independently filtering consumers on one queue. The media pipeline is a two-stage design: an ingest queue owned by the scorer Lambda, and a results queue owned by the worker Lambda. SNS fans events into the correct stage; it is not used as a substitute for competing consumers.

What the first Lambda does

Images (.jpg, .jpeg, .png, .bmp, .webp): DetectModerationLabels and RecognizeCelebrities, then evaluate thresholds and send a result message to SQS.
Videos (.mp4, .mov): StartContentModeration with an SNS notification channel. Rekognition works asynchronously; the worker finishes the job later.
Files matching a thumbnail filename or path pattern are still labeled because ad thumbnails can be the image users see first.

What the worker Lambda does

Look up the media row by S3 key, retrying three times with a three-second delay so the upload API can finish inserting the row.
Inserts content and moderation_results.
Updates media.content_moderation_status.
Propagates the worst status to the parent post, ad, or story.
For videos, pages through GetContentModeration until every label is collected, then applies the frame policy.

SQS is the buffer. Lambda is the computer. Rekognition is the eyes. PostgreSQL is the record.

How Image Scores Are Calculated

Rekognition returns a list of moderation labels, each with a confidence from 0 to 100. We keep every label name mapped to a rounded confidence, then take the maximum confidence as the risk score.

Image scoring

DetectModerationLabels
↓
Build label → confidence map
↓
riskScore = round(max confidence)
↓
Apply 75 / 90 thresholds
↓
Optional celebrity override

Confidence below 75% → approved.
Confidence from 75% up to 90% → under_review.
Confidence 90% or above → rejected.

The policy also calls out hard categories Rekognition can surface—explicit material, sexual activity, violence, hate symbols, drugs and tobacco, visually disturbing content. High confidence in those labels is what the 90% reject line is designed to catch without a human delay.

Video: Async Jobs and Frame Policy

A video cannot be scored from a single still. Rekognition runs a content-moderation job across the file, publishes completion to SNS, and that notification arrives on the same SQS queue the image path already uses.

Video path

S3 video object
↓
StartContentModeration
↓
Rekognition job
↓
SNS success notification
↓
Worker fetches all labels (paginated)
↓
Frame policy + thresholds
↓
Write status

Each label is tied to a timestamp. We still compute riskScore as the maximum confidence, but the decision also depends on how often and how long the violation appears.

Frame policy

A frame is violating if its label confidence is at least the review threshold (75%).
Five or more violating frames, or a cluster lasting three seconds or more, meets the frame policy.
If any label reaches the rejection threshold (90%+), the video is rejected.
If labels are only in the review band and the frame policy is met, the video is also rejected—repeated mid-confidence hits are treated as a pattern, not a coincidence.
If labels are only in the review band and the frame policy is not met, the video goes to under_review.
No qualifying labels → approved.

One risky frame is a question. Five frames or three seconds is an answer.

Parent Aggregation and Feed Visibility

A post, ad, or story can have several media items plus text. The parent must not appear “approved” while one of its images is still pending or rejected.

Worst-status rollup

→ Media A approved
→ Media B under_review
→ Caption approved
→ Parent = under_review

List APIs for the product surface follow one rule: show content only when content_moderation_status is approved. Under-review and rejected items are not a client-side filter. They never qualify for the public listing query.

What Happens When Content Is Rejected

When content_moderation_status becomes rejected, a Hasura event on the content table calls POST /v1/event/content-rejected-notify. That queues an email through SendGrid and an in-app/push notification through Firebase, including the content type and rejection reason. The user is told the content did not pass review. They are not left guessing why it vanished.

Rejection side effects

Status → rejected
↓
Hasura content_rejected_email
↓
Nest notify webhook
↓
Email + push
↓
Still hidden from feed

The Last Step: Admin Panel Review

Automation is the first reviewer, not the last. Anything in under_review—text, image, or video—is a case a person has to close. That is the admin panel’s job.

Human review path

What the moderator needs on screen

A queue, not a scavenger hunt: one list of under_review (and optionally pending) items across posts, comments, ads, stories, and media.
The source itself: caption text, image, or playable video.
The machine evidence: provider (Openai or Rekognition), risk score, labels, celebrity names when present, video frame summary when present.
Actions: approve, reject. Optionally escalate, assign, or add an internal note.
Filters by content type, date, and score so a small team can work the riskiest items first.

On approval, both content_moderation_status and final_decision are set to approve. On rejection, the status becomes rejected, the existing rejection notification path fires, and the item still never reaches the feed. The AI decision remains on the row as ai_decision so we can measure how often humans agree with the model.

under_review is not a soft approve. It is a closed door until a person opens it.

The Production Feedback Loop

Shipping thresholds and queues is not the end of moderation work. The system has to be operated. We treat the following signals as the minimum set for running AI moderation in production—and as the inputs that decide when to retune cutoffs.

Operational signals

Moderation latency: time from enqueue (Hasura event or S3 ObjectCreated) to written decision, tracked as p50/p95 separately for OpenAI text and Rekognition image/video.
Review-queue age and SLA: count and age of under_review items; alert when the oldest item or median wait exceeds the team’s SLA.
Provider failures: OpenAI and Rekognition error rates, throttles, empty results, and dead-letter / retry volume on BullMQ and Lambda.
AI–human disagreement rate: share of rows where ai_decision ≠ final_decision after admin review, broken down by provider and content type.
False positives and appeals: auto-reject later overturned to approve; user appeals or support tickets after rejection notifications.
False negatives and reports: user reports or admin discoveries on content that was auto-approved—these are the most expensive misses.
Threshold drift: weekly or biweekly review of the above rates;if disagreement rates, appeals, or queue age move outside defined bands if disagreement, appeals, or queue age move outside bands, re-run the labelled sweep and adjust env thresholds.

Operate → measure → retune

Ship with calibrated defaults
↓
Watch latency, queue age, provider health
↓
Compare ai_decision to final_decision
↓
Track appeals and reports
↓
Detect drift
↓
Re-sweep thresholds or widen review

Storing ai_decision alongside final_decision and reviewed_by is what makes this loop possible. Without that pair, you can run a model, but you cannot prove whether the model and the product still agree.

A moderation system without operational signals is a deployment, not a practice.

The Deployment Pipeline

The media Lambdas are not updated by hand in the console as a normal workflow. Each function has a GitHub Actions workflow on the develop branch, scoped to its own directory.

Lambda CI/CD

Push to develop
↓
OIDC to AWS
↓
npm ci --omit=dev
↓
Zip the function
↓
Create or update Lambda
↓
Wait + verify

deploy-image-moderation.yml watches lambda/image-moderation/**.
deploy-image-moderation-worker.yml watches lambda/image-moderation-worker/**.
Runtime is Node.js 22. The scorer Lambda is sized for a short Rekognition call (about 60 seconds, 256 MB). The worker is sized for video pagination and database work (about 5 minutes, 512 MB).
Secrets stay in GitHub Secrets: queue URL, database connection, and deployment role. Thresholds and regions live in deploy config.

Text moderation ships with the NestJS service: the Hasura trigger metadata, the webhook, the BullMQ processor, and the OpenAI client. Changing a threshold is a configuration change, not a rewrite of the decision engine.

Edge Cases the Pipeline Had to Absorb

Race with inserts: the worker retries media lookup because S3 can notify Lambda before Nest commits the media row.
Ad thumbnails: if the object is a thumbnail rather than a media row, the worker can still attach the result to hey_ads.
Unsupported types: unknown extensions are skipped rather than auto-approved by accident.
Idempotency: if media is already moderated, a later result may only escalate to a worse status, never quietly downgrade it.
Aggregation after the fact: parent status is recomputed from children so mixed media cannot hide a reject.

What This Changed in the Product

Outcome

Upload / post returns immediately
↓
Background AI inspects content
↓
Clear cases resolve themselves
↓
Gray cases wait for admin
↓
Feed only shows approved
↓
Rejected owners are notified

Text, image, and video share one status vocabulary.
OpenAI and Rekognition are isolated behind queues, so model latency is not user latency.
Scores are explicit numbers with published thresholds, not hidden boolean flags.
The admin panel is the last reviewer for under_review, not an optional extra screen.
Evidence is stored: labels, scores, provider, and the human override.

The important change was not simply adding another moderation API. It was moving moderation from an isolated check into a product lifecycle. Content can return from the upload or post request without waiting for model inference, while remaining outside public listings until the moderation lifecycle resolves. Clear cases move automatically, uncertain cases reach the admin queue, and the final decision is recorded alongside the machine evidence.

This also creates a feedback loop: the AI decision remains stored alongside the human final decision, allowing the team to measure where automated decisions and human decisions agree or differ.

The Bigger Lesson: Automate the Extremes, Staff the Middle

A moderation system that only auto-rejects the obvious still leaks. A system that auto-approves anything the model did not scream about still leaks. The useful design is a three-way split.

The moderation mindset

Ingest without blocking the user
↓
Score with the right model for the medium
↓
Approve the clearly safe
↓
Reject the clearly unsafe
↓
Queue the rest for admin
↓
Keep the feed on approved only

We used OpenAI for language because it already classifies the categories we care about in text. We used Rekognition for pixels because it already understands moderation labels, video timestamps, and celebrity faces. We used Lambda and SQS because media events are bursty and should not live inside the API process. We used Hasura and BullMQ because text events are already database events. We used an admin queue because a score of 62 is not a moral fact. It is a request for a person.

The goal of AI moderation is not to remove humans from the loop. It is to make sure humans spend their time where human judgment matters most.

So before a caption, a photo, or a video is allowed to represent the product, it has to survive the pipeline: pending, scored, decided, and—if the model is unsure—reviewed. That is how content moderation works here: as a path every piece of user content has to follow before it hits the feed, rather than as a slogan in a policy document.

What’s Next? Engineer AI Moderation for Production

The bigger lesson goes beyond content moderation. Production AI features need more than an accurate model. They need clear decision boundaries, asynchronous processing, human review paths, auditability, monitoring, and a way to improve as real-world behavior changes.

The same principle applies whether AI is moderating content, evaluating documents, prioritizing cases, or assisting users inside a product. Automate decisions where confidence is high, route ambiguous cases to people, and retain enough evidence to understand why each decision was made.

That is what turns an AI capability into a system a product team can operate at scale.

At GeekyAnts, our AI Engineering teams design and build production-ready AI systems around real product workflows, combining model integration with scalable architecture, observability, deployment, and human-in-the-loop controls.

Building an AI-powered feature that needs to move from model API to production system? Talk to our AI engineering team.

Automate the Obvious, Review the Rest: Designing AI-Powered Content Moderation

GeekyAnts India Pvt Ltd — Tue, 27 Oct 2026 05:31:27 +0000

This blog explains how to build an AI-powered content moderation system for text, images, and video while keeping human review in the loop.

Introduction

A social product does not fail only when the database is slow. It also fails when the feed becomes a place users no longer trust.

Moderation is not a feature bolted onto the feed. It is a gate the feed has to pass through.

Why Traditional Moderation Doesn't Scale

As content volume grows, moderation becomes less about identifying whether a single piece of content is safe and more about deciding where each piece should go.

Sending every piece of content to a human does not scale, while automatically approving or rejecting everything based on a single model score introduces its own risks.

The system therefore treats moderation as a triage pipeline: clear cases are automated, ambiguous cases are escalated, and the feed only exposes content that has reached an approved state.

Why Content Moderation Is Required

User-generated content is the product. It is also a risk. Without a consistent review path, unsafe or policy-breaking material can reach other users in seconds.

Protect people on the platform from explicit, violent, hateful, or otherwise harmful material.
Keep posts, ads, and stories out of public feeds until they are safe to show.
Meet App Store, Play Store, and trust-and-safety expectations for UGC products.
Create an audit trail: what the model saw, what score it produced, and who made the final call.
Notify the owner when content is rejected, instead of failing silently.

What the System Does

Every piece of moderated content starts as pending. Background work then produces a decision: approved, under_review, or rejected.

The Architecture: Two Pipelines, One Status Model

Shared data model

content_moderation_status on the source row: pending, approved, under_review, rejected.
A content row as the generic moderation record.
moderation_results for provider, risk_score, labels, ai_decision, and final_decision.
reviewed_by on the result row when a human takes the last step.

Feeds that list posts, ads, stories, or media filter on content_moderation_status = approved. The existing status columns (active, trash, uploaded, and so on) still mean what they always meant. Moderation is a separate lifecycle on purpose.

Do not overload "active" to mean "safe." Safety needs its own column.

Text Moderation: Hasura, BullMQ, and OpenAI

When a user writes text, the row is saved first. The HTTP request does not wait for OpenAI. Hasura fires an event; NestJS creates the content record and enqueues a job; a worker calls the model.

Text request path

The worker

The content-moderation processor runs with limited concurrency and a rate limit so OpenAI traffic stays controlled. Empty text is approved without a call.

Failed jobs go to a dead-letter queue. Hasura retries the webhook (three attempts, ten-second interval, sixty-second timeout) if Nest is briefly unavailable.

How Text Scores Are Calculated

Text scoring

OpenAI category_scores
         ↓
Take the maximum score
         ↓
riskScore = round(max × 100)
         ↓
Compare to thresholds
         ↓
approved / under_review / rejected

Risk below 40 → approved. The content can enter the feed.
Risk from 40 up to (but not including) 75 → under_review. A human must decide.
Risk 75 or above → rejected. The content stays hidden and the owner is notified.

How We Calibrated the Thresholds

Starting points, not final answers

We began with provider semantics and product risk posture, then validated against labelled samples rather than shipping intuition.

Text (OpenAI): start near the middle of the score distribution. Auto-approve only when the max category score is clearly low; auto-reject only when it is clearly high; leave a wide under_review band for the first weeks.
Images/video (Rekognition): start stricter on auto-approve than text (review at 75%, reject at 90%), because a false approval in a feed is costlier than an extra item in the admin queue.
Environment variables keep cutoffs changeable without rewriting the decision engine.

Labelled test set and threshold sweeps

Before locking production defaults, we built a small labelled set covering three buckets: clearly safe, borderline / ambiguous, and clearly violating. For each item we stored the model's raw scores, the would-be decision at candidate cutoffs, and a human gold label.

Calibration experiment

What "good" looked like for us

Near-zero false auto-approves on the labelled violate set (missed harm is the expensive error).
Bounded false auto-rejects: overturns in admin or appeals should stay rare; if many auto-rejects become approved, raise the reject line or widen review.
A review band that is large enough to catch ambiguity but small enough that queue age stays within SLA.
We treat the AI-human disagreement rate on under_review items as a first-class metric rather than noise.

Thresholds are hypotheses. Production disagreement rates are the experiment.

Image and Video Moderation: S3, Lambda, SQS, Rekognition

The media never waits in the Nest process for Rekognition. The user uploads to S3 under users/uploads/. An ObjectCreated notification is published to SNS, and the media path continues asynchronously.

Why not two Lambdas on one SQS queue

SQS is a competing-consumer queue: each message is delivered to exactly one consumer. If two Lambda event-source mappings share a single queue and each "filters" by message shape, a message can land on the wrong function. That function may skip and still succeed—deleting the message—or fail and retry forever. Either way, delivery semantics are not safe for mixed payload types.

What the first Lambda does

Images (.jpg, .jpeg, .png, .bmp, .webp): DetectModerationLabels and RecognizeCelebrities, then evaluate thresholds and send a result message to SQS.
Videos (.mp4, .mov): StartContentModeration with an SNS notification channel. Rekognition works asynchronously; the worker finishes the job later.
Files matching a thumbnail filename or path pattern are still labeled because ad thumbnails can be the image users see first.

What the worker Lambda does

Look up the media row by S3 key, retrying three times with a three-second delay so the upload API can finish inserting the row.
Inserts content and moderation_results.
Updates media.content_moderation_status.
Propagates the worst status to the parent post, ad, or story.
For videos, pages through GetContentModeration until every label is collected, then applies the frame policy.

SQS is the buffer. Lambda is the computer. Rekognition is the eyes. PostgreSQL is the record.

How Image Scores Are Calculated

Rekognition returns a list of moderation labels, each with a confidence from 0 to 100. We keep every label name mapped to a rounded confidence, then take the maximum confidence as the risk score.

Image scoring

Confidence below 75% → approved.
Confidence from 75% up to 90% → under_review.
Confidence 90% or above → rejected.

The defaults are MODERATION_REVIEW_THRESHOLD = 75 and MODERATION_REJECT_THRESHOLD = 90. They use a stricter review boundary than the text moderation thresholds because a false approval on a photo is more damaging in a feed than an unnecessary review.
The policy also calls out hard categories Rekognition can surface—explicit material, sexual activity, violence, hate symbols, drugs and tobacco, visually disturbing content. High confidence in those labels is what the 90% reject line is designed to catch without a human delay.

Video: Async Jobs and Frame Policy

Video path

Each label is tied to a timestamp. We still compute riskScore as the maximum confidence, but the decision also depends on how often and how long the violation appears.

Frame policy

A frame is violating if its label confidence is at least the review threshold (75%).
Five or more violating frames, or a cluster lasting three seconds or more, meets the frame policy.
If any label reaches the rejection threshold (90%+), the video is rejected.
If labels are only in the review band and the frame policy is met, the video is also rejected—repeated mid-confidence hits are treated as a pattern, not a coincidence.
If labels are only in the review band and the frame policy is not met, the video goes to under_review.
No qualifying labels → approved.

One risky frame is a question. Five frames or three seconds is an answer.

Parent Aggregation and Feed Visibility

A post, ad, or story can have several media items plus text. The parent must not appear "approved" while one of its images is still pending or rejected.

Worst-status rollup

→ Media A approved
→ Media B under_review
→ Caption approved
→ Parent = under_review

Statuses are ordered pending < approved < under_review < rejected. The database function apply_aggregate_moderation_status applies that order to user_post, hey_ads, and users_story. If any child is rejected, the parent is rejected.
List APIs for the product surface follow one rule: show content only when content_moderation_status is approved. Under-review and rejected items are not a client-side filter. They never qualify for the public listing query.

What Happens When Content Is Rejected

Rejection side effects

The Last Step: Admin Panel Review

Automation is the first reviewer, not the last. Anything in under_review—text, image, or video—is a case a person has to close. That is the admin panel's job.

Human review path

What the moderator needs on screen

A queue, not a scavenger hunt: one list of under_review (and optionally pending) items across posts, comments, ads, stories, and media.
The source itself: caption text, image, or playable video.
The machine evidence: provider (Openai or Rekognition), risk score, labels, celebrity names when present, video frame summary when present.
Actions: approve, reject. Optionally escalate, assign, or add an internal note.
Filters by content type, date, and score so a small team can work the riskiest items first.

under_review is not a soft approve. It is a closed door until a person opens it.

The Production Feedback Loop

Operational signals

Moderation latency: time from enqueue (Hasura event or S3 ObjectCreated) to written decision, tracked as p50/p95 separately for OpenAI text and Rekognition image/video.
Review-queue age and SLA: count and age of under_review items; alert when the oldest item or median wait exceeds the team's SLA.
Provider failures: OpenAI and Rekognition error rates, throttles, empty results, and dead-letter / retry volume on BullMQ and Lambda.
AI–human disagreement rate: share of rows where ai_decision ≠ final_decision after admin review, broken down by provider and content type.
False positives and appeals: auto-reject later overturned to approve; user appeals or support tickets after rejection notifications.
False negatives and reports: user reports or admin discoveries on content that was auto-approved—these are the most expensive misses.
Threshold drift: weekly or biweekly review of the above rates;if disagreement rates, appeals, or queue age move outside defined bands if disagreement, appeals, or queue age move outside bands, re-run the labelled sweep and adjust env thresholds.

Operate → measure → retune

A moderation system without operational signals is a deployment, not a practice.

The Deployment Pipeline

The media Lambdas are not updated by hand in the console as a normal workflow. Each function has a GitHub Actions workflow on the develop branch, scoped to its own directory.

Lambda CI/CD

deploy-image-moderation.yml watches lambda/image-moderation/**.
deploy-image-moderation-worker.yml watches lambda/image-moderation-worker/**.
Runtime is Node.js 22. The scorer Lambda is sized for a short Rekognition call (about 60 seconds, 256 MB). The worker is sized for video pagination and database work (about 5 minutes, 512 MB).
Secrets stay in GitHub Secrets: queue URL, database connection, and deployment role. Thresholds and regions live in deploy config.

Edge Cases the Pipeline Had to Absorb

Race with inserts: the worker retries media lookup because S3 can notify Lambda before Nest commits the media row.
Ad thumbnails: if the object is a thumbnail rather than a media row, the worker can still attach the result to hey_ads.
Unsupported types: unknown extensions are skipped rather than auto-approved by accident.
Idempotency: if media is already moderated, a later result may only escalate to a worse status, never quietly downgrade it.
Aggregation after the fact: parent status is recomputed from children so mixed media cannot hide a reject.

What This Changed in the Product

Outcome

Text, image, and video share one status vocabulary.
OpenAI and Rekognition are isolated behind queues, so model latency is not user latency.
Scores are explicit numbers with published thresholds, not hidden boolean flags.
The admin panel is the last reviewer for under_review, not an optional extra screen.
Evidence is stored: labels, scores, provider, and the human override.

This also creates a feedback loop: the AI decision remains stored alongside the human final decision, allowing the team to measure where automated decisions and human decisions agree or differ.

The Bigger Lesson: Automate the Extremes, Staff the Middle

A moderation system that only auto-rejects the obvious still leaks. A system that auto-approves anything the model did not scream about still leaks. The useful design is a three-way split.

The moderation mindset

The goal of AI moderation is not to remove humans from the loop. It is to make sure humans spend their time where human judgment matters most.

What's Next? Engineer AI Moderation for Production

That is what turns an AI capability into a system a product team can operate at scale.

Building an AI-powered feature that needs to move from model API to production system? Talk to our AI engineering team.

Mobile App Deep Linking With React Native, NestJS and AWS

GeekyAnts India Pvt Ltd — Tue, 27 Oct 2026 05:11:40 +0000

Build a self-hosted deep linking and referral system for React Native without a third-party SDK. This guide covers Android App Links, iOS Universal Links, an S3 and CloudFront setup, and a NestJS backend that routes users to the right app store.

Third-party attribution services like Branch, AppsFlyer, and Firebase Dynamic Links offer turn-key deep linking out of the box. However, they frequently bring vendor lock-in, recurring SaaS subscription overhead, privacy and data residency compliance challenges, and sudden deprecations (such as the sunsetting of Firebase Dynamic Links).

Firebase Dynamic Links is the clearest example: Google shut the service down on August 25, 2025, and every page.link URL stopped working on that date.

This guide walks through building a fully self-hosted, vendor-agnostic deep linking and referral system using native OS domain verification, AWS infrastructure (S3, CloudFront, Application Load Balancer), a NestJS backend, and a React Native mobile client.

What Is Deep Linking?

Deep linking routes a user directly to a specific screen or action within a mobile app when clicking a standard HTTPS URL, bypassing the default home screen.

Tap a normal link on your phone and your browser opens the matching webpage. A deep link works the same way, except it opens a specific screen inside an app instead.

Example: an invite link

A friend sends you a link to join an app.

Normal link: takes you to a generic webpage or app store listing
Deep link: if the app is already installed, it skips the browser completely and drops you straight into the exact screen you need, like a sign-up form with your friend's invite code already filled in

Why this matters

Without deep linking, joining through a link looks like this:

Tap the link
Land on a generic webpage
Manually open the app
Find the right screen
Type in the code by hand

With deep linking, all five steps collapse into one tap.

The hard part: app not installed

If the app isn't on the phone yet, the link still needs to do something useful. Usually that means sending the person to the App Store or Play Store, then remembering their original intent (like that invite code) once they install and open the app for the first time.

This is the harder problem the rest of the guide solves, and it's usually why teams reach for paid third-party tools instead of building it themselves.

Types of Mobile Links

Mobile platforms support more than one kind of link. This guide uses verified HTTPS links because they are the only type that the operating system ties to a domain you own.

Link type	Example	How the OS trusts it	Limitation
Custom URI scheme	`myapp://join?code=ABC123`	No verification. Any app can register the same scheme	Fails with an error when the app is not installed
Android App Links	`https://invite.app.com/join?code=ABC123`	Android checks assetlinks.json on your domain through the Digital Asset Links API	Opens in the browser if verification fails
iOS Universal Links	`https://invite.app.com/join?code=ABC123`	iOS checks apple-app-site-association on your domain	Opens in Safari if the app is missing or the file cannot be fetched

App Links and Universal Links share one URL for both platforms. The same link also works as a normal web page when the app is not installed, which is what makes the store redirect in this guide possible.

Core Use Cases

The rest of this guide builds the referral onboarding case end to end. The same setup supports the other three cases once you add their paths to the manifests and the app router.

Use Case	Example URL	Expected Target Screen
Referral Onboarding	`https://invite.app.com/join?code=ABC123`	Pre-filled Registration Screen
Product Sharing	`https://store.app.com/products/8930`	Specific Product Details Page
Push Notifications	`https://app.com/notifications/102`	Direct Chat / Message Context
Marketing Campaigns	`https://app.com/promos/summer2026`	Discount / Landing Screen

Direct vs. Deferred Deep Linking

Understanding operating system boundaries is essential before designing the architecture.

Direct Deep Linking

The mobile application is already installed. The mobile operating system (Android or iOS) intercepts the HTTP request, matches the domain signature against its installed applications, and directly opens the app without touching the web browser or backend web servers.

The OS only intercepts links that a user taps. On iOS, a Universal Link typed or pasted into the Safari address bar opens the web page instead of the app. A tap on a link to the same domain inside Safari also stays in the browser. Share links from messages, email, notes or other apps to get the expected behavior.

Deferred Deep Linking

The application is not installed. The browser handles the URL, sending the request to your backend. The user is redirected to the Google Play Store or Apple App Store. Because native Android App Links and iOS Universal Links do not natively pass context across an installation boundary, the original URL parameters are lost upon install.

Scenario	Who handles the link	Does the referral code survive?
App installed	Android or iOS opens the app directly	Yes. The app reads it from the URL
App not installed, Android	Browser, then CloudFront, then NestJS, then Play Store	Yes, through the Play Install Referrer
App not installed, iOS	Browser, then CloudFront, then NestJS, then App Store	No. The user enters it in the app during a grace period

Architecture Distinction: Without third-party SDKs, this architecture provides store routing with server-side/manual attribution (via Play Store Install Referrer on Android and manual user/web account grace periods on iOS) rather than seamless, native OS-level deferred deep linking.

How the Deep Linking Architecture Works

One CloudFront distribution sits in front of two origins. Static verification files come from a private S3 bucket, and every dynamic request on the domain goes to the NestJS service behind the Application Load Balancer. This keeps the files that Android and iOS read separate from the redirect logic, on a single domain and a single HTTPS certificate.

Component Responsibilities

AWS / Stack Component	Primary Responsibility
Amazon Route 53	Manages DNS A records (Alias to CloudFront).
Amazon CloudFront	CDN handling HTTPS termination, edge routing, and origin caching.
Amazon S3	Private origin hosting static .well-known OS verification manifests.
AWS ALB	Routes dynamic requests (/join, API routes) to the NestJS cluster.
NestJS API	Evaluates User-Agent headers, processes referral logic, and executes store redirects.
React Native App	Handles incoming deep link events via native Linking listeners.

A request to /.well-known/* is served from S3. Every other path goes to the ALB and then to NestJS, which decides where to send the visitor.

Hosting the Verification Files on Amazon S3

This creates the storage for the two files that Android and iOS download to verify your domain. The bucket stays private, and CloudFront is the only service allowed to read it.

Step 1: Create a Private Amazon S3 Bucket

Open the Amazon S3 Console and click Create bucket.
Name the bucket (e.g., app-domain-verification-prod).
Keep Block all public access enabled.
Set Object Ownership to Bucket owner enforced.

S3 Bucket Structure
└── .well-known/
    ├── assetlinks.json
    └── apple-app-site-association

Step 2: Generate OS Verification Manifests

Create a local directory named .well-known containing the two required manifest files. Each file lists the apps that are allowed to open links on your domain.

Android Verification (.well-known/assetlinks.json)

[
  {
    "relation": [
      "delegate_permission/common.handle_all_urls"
    ],
    "target": {
      "namespace": "android_app",
      "package_name": "com.company.app",
      "sha256_cert_fingerprints": [
        "your-sha-fingerprint-here"
      ]
    }
  }
]

Pro Tip: Retrieve your SHA-256 fingerprint for production builds via Google Play Console under App Signing > App signing key certificate. The same page in Play Console also shows a ready-made Digital Asset Links JSON snippet for your app. The sha256_cert_fingerprints field is an array, so you can add the upload key fingerprint as a second entry if testers install builds outside the Play Store.

iOS Verification (.well-known/apple-app-site-association)

Note: Do not append .json to the file name.

{
  "applinks": {
    "details": [
      {
        "appIDs": [
          "9JA6382026.com.company.app"
        ],
        "components": [
          {
            "/": "/join*"
          }
        ]
      }
    ]
  }
}

The value in appIDs is your Apple Team ID followed by the app's bundle identifier. The components array controls which paths open the app. Here only paths that start with /join are claimed, so every other page on the domain keeps opening in the browser.

Apple sets three hosting rules for this file. It must be served over https:// with a valid certificate, it must return directly with no redirects, and it must sit in the .well-known directory. Since iOS 14, devices do not fetch the file from your server. They fetch it from an Apple-managed CDN, which requests your file within 24 hours. Expect a delay before changes reach devices.

Step 3: Upload Object & Set MIME Types

Upload the .well-known folder to your S3 bucket. Ensure the HTTP Content-Type metadata for both files is explicitly configured as application/json:

aws s3 cp .well-known/ s3://app-domain-verification-prod/.well-known/ \
  --recursive \
  --content-type "application/json"

CloudFront caches these files at the edge. When you change either file later, create an invalidation so the new version is served right away:

aws cloudfront create-invalidation \
  --distribution-id <your-distribution-id> \
  --paths "/.well-known/*"

Configuring CloudFront and Route 53

One CloudFront distribution serves both origins. Static verification files come from S3, and every other request on the domain goes to the NestJS service behind the ALB.

Step 1: Configure Origins & Origin Access Control (OAC)

Navigate to CloudFront > Distributions > Create Distribution.
Origin 1 (S3): Select your S3 bucket endpoint.
- Under Origin Access, select Origin Access Control (OAC) and create an OAC control setting.
- Apply the provided JSON policy to your S3 bucket policy to allow CloudFront access while keeping the bucket completely private.
Origin 2 (ALB): Add your Application Load Balancer domain (e.g., nestjs-alb-12345.ap-south-1.elb.amazonaws.com) using HTTPS.

Important: When CloudFront connects to an origin over HTTPS, the origin's certificate must include a domain name that matches the Origin Domain. If it does not match, CloudFront returns 502 Bad Gateway. You cannot issue a certificate for the default elb.amazonaws.com name. Create a DNS record such as origin-api.app.com that points to the ALB, attach an ACM certificate for that name to the ALB's HTTPS listener, and enter origin-api.app.com as the Origin Domain.

This certificate is created in the ALB's own Region.

Step 2: Configure Path-Based Cache Behaviors

Set up path routing rules to separate static verification from dynamic application requests:

Priority 1 (/.well-known/*): Target: S3 Origin. Caching Enabled (Managed-CachingOptimized).

Priority 2 (/join or Default *): Target: ALB Origin.
Cache Policy: Managed-CachingDisabled (TTL = 0). Dynamic redirects must never be cached by CloudFront edge nodes.
Origin Request Policy: Must forward Query Strings (code) and Headers (User-Agent, or CloudFront Device Detection headers: CloudFront-Is-Mobile-Viewer, CloudFront-Is-IOS-Viewer, CloudFront-Is-Android-Viewer).

Note: By default, CloudFront strips query parameters and overwrites the viewer User-Agent with Amazon CloudFront. Forwarding these headers/queries is required for NestJS platform detection.

Step 3: SSL Certificate & DNS Alias

Request an AWS Certificate Manager (ACM) public certificate for invite.app.com.
- Crucial Rule: The certificate must be created in the us-east-1 (N. Virginia) region to attach to CloudFront.
In CloudFront Settings, add invite.app.com under Alternate domain name (CNAME) and select the ACM certificate.
In Route 53, create an A Record for invite.app.com, choose Alias, and point it to the CloudFront distribution target (dXXXXXX.cloudfront.net).

Step 4: Protect ALB Origin (Direct Access Prevention)

To prevent users or malicious traffic from bypassing CloudFront and hitting the ALB directly:

In CloudFront Origin Settings for the ALB, add a custom HTTP header: X-Origin-Verify: <your-secret-token>.
In your ALB Listener rules (or NestJS middleware), block any incoming request that does not contain this exact X-Origin-Verify header value with a 403 Forbidden.
Restrict the ALB security group so it only accepts traffic from the AWS-managed prefix list com.amazonaws.global.cloudfront.origin-facing.

The prefix list and the header protect against different things. The prefix list blocks traffic that does not come from CloudFront. The header blocks traffic from other CloudFront distributions that could also reach your ALB. AWS recommends using both together.

If you are building a new stack, CloudFront VPC origins are another option. They let CloudFront reach an ALB in a private subnet, so the ALB has no public endpoint at all. AWS announced VPC origins in November 2024 at no additional cost.

Verifying Infrastructure Deployment

Run standard curl checks to verify headers and response bodies before proceeding with client-side integration:

# 1. Verify Android Verification Asset
curl -i https://invite.app.com/.well-known/assetlinks.json

# Expected Output:
# HTTP/2 200
# content-type: application/json
# x-cache: Hit from cloudfront

# 2. Verify iOS Verification Asset
curl -i https://invite.app.com/.well-known/apple-app-site-association

# Expected Output:
# HTTP/2 200
# content-type: application/json

The first request after an upload or invalidation returns x-cache: Miss from cloudfront. Run the command again to see the cached Hit response. A 301 or 302 status on either file is a failure, since both platforms reject redirects.

Two more checks confirm what the platforms themselves see:

# 3. Ask Google's Digital Asset Links API to read your Android file
curl "https://digitalassetlinks.googleapis.com/v1/statements:list?source.web.site=https://invite.app.com&relation=delegate_permission/common.handle_all_urls"

# 4. Check the copy of your iOS file held on Apple's CDN
curl -i https://app-site-association.cdn-apple.com/a/v1/invite.app.com

Check 3 returns your package name and fingerprint when Google can parse the file. Check 4 returns the file once Apple's CDN has fetched it, which can take up to 24 hours after the first upload.

Configuring Android App Links and iOS Universal Links

The server files say which app may open your links. The app must also declare which domain it handles. Both sides have to match before Android or iOS will open the app.

Android Configuration (AndroidManifest.xml)

Add the auto-verify intent filter inside your <activity> block (typically MainActivity):

XML

<activity
  android:name=".MainActivity"
  android:launchMode="singleTask"
  android:exported="true">

  <intent-filter android:autoVerify="true">
      <action android:name="android.intent.action.VIEW" />
      <category android:name="android.intent.category.DEFAULT" />
      <category android:name="android.intent.category.BROWSABLE" />

      <data
          android:scheme="https"
          android:host="invite.app.com"
          android:pathPrefix="/join" />
  </intent-filter>
</activity>

Three attributes matter most here:

android:autoVerify="true" tells Android to check assetlinks.json on install. On Android 6.0 and higher, the system verifies the hosts in your intent filters automatically when the app is installed.
android:launchMode="singleTask" reuses the running app instance. A link tapped while the app is open then reaches the existing Linking listener instead of starting a second copy of the screen.
android:pathPrefix="/join" limits the app to referral paths, which matches the /join* rule in the iOS file.

iOS Configuration (Xcode Capability)

Open your iOS workspace in Xcode.
Under Target > Signing & Capabilities, click + Capability and select Associated Domains.
Add the domain entry using the applinks: prefix:

applinks:invite.app.com

Handling Deep Links in React Native

The app can receive a link in two states. In a cold start, the link launches a closed app. In a warm launch, the link arrives while the app runs in the background. React Native's Linking API covers both: getInitialURL() for cold starts and the url event for warm launches.

Handle cold-start launches (getInitialURL) and background warm launches (addEventListener):

// src/services/DeepLinkHandler.ts
import { useEffect } from 'react';
import { Linking } from 'react-native';
import AsyncStorage from '@react-native-async-storage/async-storage';

export const PENDING_REFERRAL_KEY = '@pending_referral_code';

export function parseReferralCode(url: string | null): string | null {
  if (!url) return null;

  try {
    const parsedUrl = new URL(url);
    if (
      parsedUrl.hostname === 'invite.app.com' &&
      parsedUrl.pathname === '/join'
    ) {
      return parsedUrl.searchParams.get('code');
    }
  } catch (error) {
    console.error('Failed to parse deep link URL:', error);
  }
  return null;
}

export function useDeepLinkHandler() {
  useEffect(() => {
    // 1. Handle background/running app deep links
    const subscription = Linking.addEventListener('url', async ({ url }) => {
      const code = parseReferralCode(url);
      if (code) {
        await AsyncStorage.setItem(PENDING_REFERRAL_KEY, code);
        // Trigger navigation or dispatch action to store
      }
    });

    // 2. Handle cold-start deep links
    async function checkInitialUrl() {
      const initialUrl = await Linking.getInitialURL();
      const code = parseReferralCode(initialUrl);
      if (code) {
        await AsyncStorage.setItem(PENDING_REFERRAL_KEY, code);
      }
    }

    checkInitialUrl();

    return () => {
      subscription.remove();
    };
  }, []);
}

''While iOS requires server-side or grace-period attribution, Android allows us to extract the original referral code automatically from the Play Store install metadata upon first launch.''

Handling Android Deferred Links (Play Store Install Referrer)

On Android, when a user installs the app via a Play Store link containing &referrer=..., the referral code is preserved in the Play Store metadata. Use react-native-play-install-referrer to fetch it on first launch:

import { Platform } from 'react-native';
import AsyncStorage from '@react-native-async-storage/async-storage';
import PlayInstallReferrer from 'react-native-play-install-referrer';
import { PENDING_REFERRAL_KEY } from './DeepLinkHandler';
async function checkAndroidInstallReferrer() {
  if (Platform.OS !== 'android') return;

  const isFirstLaunch = await AsyncStorage.getItem('@first_launch_done');
  if (!isFirstLaunch) {
    PlayInstallReferrer.getInstallReferrer(async (installReferrerInfo, error) => {
      if (!error && installReferrerInfo?.installReferrer) {
        const params = new URLSearchParams(installReferrerInfo.installReferrer);
        const code = params.get('utm_content');
        if (code) {
          await AsyncStorage.setItem(PENDING_REFERRAL_KEY, code);
        }
      }
      await AsyncStorage.setItem('@first_launch_done', 'true');
    });
  }
}

Google keeps the referrer data for 90 days, and it does not change unless the app is reinstalled. Google also advises calling the API only once, during the first run after install. The @first_launch_done flag handles this. The flag is now written inside the callback, so a launch that closes before the Play Store responds will try again on the next start.

Building the NestJS Redirect Backend

When users without the app click the deep link, CloudFront routes the request to your NestJS server behind the ALB. The backend detects the platform and redirects the user to the appropriate App Store.

NestJS Controller (referral.controller.ts)

import { Controller, Get, Query, Req, Res, BadRequestException, Headers } from '@nestjs/common';
import { Request, Response } from 'express';

@Controller('join')
export class ReferralController {
  private readonly PLAY_STORE_URL = 'https://play.google.com/store/apps/details?id=com.company.app';
  private readonly APP_STORE_URL = 'https://apps.apple.com/app/id1234567890';
  private readonly WEB_FALLBACK_URL = 'https://app.com/download';

  @Get()
  handleReferralRedirect(
    @Query('code') code: string,
    @Req() req: Request,
    @Res() res: Response,
    @Headers('cloudfront-is-android-viewer') isAndroidCloudFront?: string,
    @Headers('cloudfront-is-ios-viewer') isIosCloudFront?: string,
  ) {
    if (!code || !/^[A-Za-z0-9_-]{3,16}$/.test(code)) {
      throw new BadRequestException('Invalid or missing referral code.');
    }

    const userAgent = req.headers['user-agent'] || '';

    // Check CloudFront Device Headers first, fallback to User-Agent
    const isAndroid = isAndroidCloudFront === 'true' || /Android/i.test(userAgent);
    const isIOS = isIosCloudFront === 'true' || /iPhone|iPad|iPod/i.test(userAgent);

    if (isAndroid) {
      // Appends Play Store install referrer parameter
      return res.redirect(302, `${this.PLAY_STORE_URL}&referrer=utm_source%3Dreferral%26utm_content%3D${encodeURIComponent(code)}`);
    }

    if (isIOS) {
      return res.redirect(302, this.APP_STORE_URL);
    }

    return res.redirect(302, `${this.WEB_FALLBACK_URL}?code=${encodeURIComponent(code)}`);
  }
}

Production Controls: Abuse & Replay Prevention

Rate Limiting: Protect the /join endpoint using @nestjs/throttler (e.g., max 10 requests per minute per IP) to prevent referral code enumeration attacks.
Referral Claim Limits: Enforce database constraints limiting one claimed referral code per user account and unique device ID.
Fraud Checks: Flag accounts claiming codes where the referrer and referee share the same IP address or payment method.

Fallback Strategies for Uninstalled Users

Because iOS Universal Links and standard App Links do not preserve URL query params across fresh store downloads without a vendor SDK, choose one of these production-ready custom fallbacks:

Option A: 7-Day In-App Grace Period

Allow newly registered users to manually claim or enter a referral code within 7 days of account creation.

Claims Service (referral-claim.service.ts)

import { Injectable, BadRequestException, ConflictException } from '@nestjs/common';
import { PrismaService } from '../prisma/prisma.service';

@Injectable()
export class ReferralClaimService {
  constructor(private readonly prisma: PrismaService) {}

  async claimReferral(userId: string, code: string) {
    const user = await this.prisma.user.findUnique({ where: { id: userId } });
    if (!user) throw new BadRequestException('User not found.');

    // 1. Enforce 7-Day Expiration Policy
    const gracePeriodMs = 7 * 24 * 60 * 60 * 1000;
    const isEligible = Date.now() - new Date(user.createdAt).getTime() <= gracePeriodMs;

    if (!isEligible) {
      throw new BadRequestException('The 7-day grace period for applying a referral code has expired.');
    }

    // 2. Validate Referral Code Existence & Ownership
    const referralCodeOwner = await this.prisma.referralCode.findUnique({ where: { code } });
    if (!referralCodeOwner) {
      throw new BadRequestException('Invalid referral code.');
    }

    if (referralCodeOwner.ownerId === userId) {
      throw new BadRequestException('Users cannot claim their own referral code.');
    }

    // 3. Atomically Record Claim
    return this.prisma.$transaction(async (tx) => {
      const existingClaim = await tx.referralClaim.findUnique({ where: { userId } });
      if (existingClaim) {
        throw new ConflictException('A referral code has already been applied to this account.');
      }

      const claim = await tx.referralClaim.create({
        data: {
          userId,
          referralCode: code,
          referrerId: referralCodeOwner.ownerId,
        },
      });

      // Credit rewards to both parties...
      return claim;
    });
  }
}

Common Pitfalls

Symptom	Primary Cause	Solution
App opens in browser instead of app	Fingerprint mismatch or missing domain configuration.	Check assetlinks.json SHA-256 fingerprint. On iOS, verify applinks: is present in Xcode Capabilities.
CloudFront returns 403 Forbidden	S3 Bucket Policy is blocking CloudFront OAC.	Update S3 bucket policy with proper AWS:SourceArn matching CloudFront distribution ARN.
/join request hits wrong origin	Priority order misconfiguration in CloudFront Cache Behaviors.	Ensure /.well-known/* behavior is listed above Default (*) behavior in CloudFront console.
Universal Link fails on iOS Simulator	Scheme/domain not registered in Simulator environment or bad test invocation.	Test Universal Links on iOS Simulator via CLI using xcrun simctl openurl booted "`https://invite.app.com/join?code=ABC123`". Ensure Associated Domains capability is added to the Xcode scheme.

Production Test Matrix

The Play Store only records the referrer when the install starts from the redirected link. Test the Android path with the full /join redirect instead of opening the store listing directly.

Platform	App Status	Test Trigger URL	Expected Behavior	Verification Step
Android	Installed	`https://invite.app.com/join?code=TEST1`	Intercepted by OS; opens app directly to onboarding screen.	App opens without browser redirect; code=TEST1 present in app state.
Android	Not Installed	`https://invite.app.com/join?code=TEST1`	CloudFront routes to NestJS; 302 redirects to Play Store with &referrer=....	Inspect Network tab; confirm Play Store URL contains referrer query parameter.
iOS	Installed	`https://invite.app.com/join?code=TEST1`	Intercepted by iOS Universal Links; opens app directly.	App opens directly from Safari/Notes/Messages without opening a web page.
iOS	Not Installed	`https://invite.app.com/join?code=TEST1`	CloudFront routes to NestJS; 302 redirects to Apple App Store.	User is redirected to App Store; code applied post-install via 7-day grace period fallback.

Self-Hosted vs Third-Party SDK

The table below compares this self-hosted setup with a third-party SDK. Use it to decide whether the setup covers your attribution needs.

Capability	Custom AWS Architecture	Third-Party SDK (Branch / AppsFlyer)
Direct Deep Linking (App Installed)	Native	Native
Zero Vendor Subscription Cost	Yes	High SaaS Tiers
Data Privacy & GDPR Compliance	100% Data Ownership	Third-party Tracking
Zero External SDK Overhead	Native Modules Only	Large SDK Footprint
Automatic Install Fingerprinting	Requires Web Fallback / Grace Period	Probabilistic / Deterministic Engine

The self-hosted setup suits teams that need branded links, referral onboarding and full control of user data. A third-party SDK is the better fit when marketing needs automatic install attribution on iOS across paid ad networks.

Need Help Building This?

Setting up deep linking infrastructure from scratch takes real engineering time: AWS configuration, platform-specific verification files, backend routing logic, and ongoing maintenance as app store requirements shift.

If your team needs hands-on support, GeekyAnts' Mobile Engineering services cover React Native architecture, backend integration, and infrastructure setup like the one in this guide. Our engineers are active contributors to the React Native ecosystem and have shipped 300+ React Native apps for global enterprises.

Conclusion

This setup gives you deep links that run on your own domain and your own AWS account. Android and iOS verify the domain through two static files on S3, CloudFront splits those files from the dynamic /join route, and NestJS sends each visitor without the app to the correct store. The React Native app reads the code on every launch path, the Play Install Referrer carries it through Android installs, and the grace period, web fallback or invite matching recover it on iOS.

With the checklist complete and the test matrix passing, the same infrastructure can carry product, notification and campaign links by adding their paths to the manifests and the app router.

Build an MCP-Powered QA Agent with Playwright and TypeScript

GeekyAnts India Pvt Ltd — Tue, 27 Oct 2026 05:10:54 +0000

Build an MCP QA agent that runs Playwright tests, validates APIs, and analyzes failures through a controlled, auditable tool layer with human approval.

Connecting AI to Playwright, APIs, test evidence, and CI/CD.

Abstract

AI-assisted software testing is moving beyond code generation toward agents that can inspect applications, execute tests, analyze failures, and interact with engineering tools. Model Context Protocol (MCP) provides a standardized interface for AI applications to discover and invoke tools exposed by external systems. The current MCP TypeScript SDK v2 exposes tools, resources, and prompts.

This article presents a code-first QA architecture using TypeScript, Playwright, Zod, structured outputs, permission boundaries, evidence collection, and human approval. The objective is controlled access to deterministic QA capabilities rather than unrestricted autonomy.

Introduction

A modern QA workflow spans browser automation, API validation, logs, traces, test reports, test data, source control, and CI/CD. An AI model can reason across these systems, but reasoning alone does not give it safe or structured access to them.

MCP provides a standardized protocol boundary between an AI host and external tools and data. The engineering challenge is therefore not simply connecting MCP to Playwright; it is designing a QA tool layer with explicit capabilities, validated inputs, controlled environments, observable outputs, and safe failure behavior.

This article focuses on a practical architecture: an MCP server exposes selected QA operations while Playwright and API clients remain the deterministic execution layer. The AI can request an operation, but the server owns validation and authorization.

Architecture

The MCP TypeScript SDK supports servers exposing tools, resources, and prompts. Tools are appropriate for actions and side effects; resources are appropriate for read-oriented data such as reports or configuration.

The SDK also supports stdio and Streamable HTTP transports. See the official MCP SDK documentation for the current v2 implementation.

Project Structure

qa-mcp-agent/
├── src/
│   ├── server.ts
│   ├── tools/
│   │   ├── playwright.ts
│   │   ├── api.ts
│   │   └── evidence.ts
│   ├── security/
│   │   └── permissions.ts
│   └── utils/
│       └── sanitize.ts
├── tests/
│   └── checkout.spec.ts
├── artifacts/
├── package.json
└── tsconfig.json

Creating the MCP Server

For a current TypeScript implementation, use the v2 server package and Zod for input validation.

npm init -y
npm install @modelcontextprotocol/server zod @playwright/test

import { McpServer } from "@modelcontextprotocol/server";

const server = new McpServer({
  name: "qa-mcp-server",
  version: "1.0.0"
});

The current v2 SDK documents @modelcontextprotocol/server as the server package and uses Zod schemas for typed validation.

Expose a Controlled Playwright Tool

import * as z from "zod/v4";

server.registerTool(
  "run_playwright_test",
  {
    title: "Run Playwright Test",
    description: "Execute an approved Playwright test file",
    inputSchema: { testFile: z.string() },
    outputSchema: {
      passed: z.boolean(),
      output: z.string()
    }
  },
  async ({ testFile }) => {
    validateTestFile(testFile);
    const result = await runPlaywrightTest(testFile);
    const output = { passed: result.passed, output: result.output };
    return {
      content: [{ type: "text", text: JSON.stringify(output) }],
      structuredContent: output
    };
  }
);

MCP tool schemas validate arguments before the handler executes, and structured outputs can be validated against an output schema. Clients can discover tools with listTools and invoke them with callTool.

Keep Playwright Deterministic

import { execFile } from "node:child_process";

function runPlaywrightTest(testFile: string) {
  return new Promise<{ passed: boolean; output: string }>((resolve) => {
    execFile(
      "npx",
      ["playwright", "test", testFile],
      { timeout: 120000 },
      (error, stdout, stderr) => {
        resolve({
          passed: !error,
          output: `${stdout}\n${stderr}`
        });
      }
    );
  });
}

The server should not pass arbitrary shell strings from the model into an operating-system shell. Use an allowlist and validated arguments.

Permission Boundaries

const ALLOWED_TESTS = new Set([
  "tests/login.spec.ts",
  "tests/checkout.spec.ts",
  "tests/cart.spec.ts"
]);

function validateTestFile(testFile: string) {
  if (!ALLOWED_TESTS.has(testFile)) {
    throw new Error(`Test file is not approved: ${testFile}`);
  }
}

The security boundary belongs in deterministic application code, not in the model prompt.

API Testing as Another Tool

server.registerTool(
  "validate_api",
  {
    title: "Validate API",
    description: "Send an HTTP request and validate the response",
    inputSchema: {
      method: z.enum(["GET", "POST", "PUT", "DELETE"]),
      url: z.string(),
      expectedStatus: z.number()
    }
  },
  async ({ method, url, expectedStatus }) => {
    const response = await fetch(url, { method });
    const output = {
      status: response.status,
      passed: response.status === expectedStatus
    };
    return {
      content: [{ type: "text", text: JSON.stringify(output) }],
      structuredContent: output
    };
  }
);

In production, URL allowlists, authentication handling, request-body validation, rate limits, and environment restrictions should be added before exposing mutation-capable API tools.

Use Resources for Test Evidence

MCP resources are a better fit for read-oriented evidence than action tools. A QA server can expose resources such as test-results://latest or trace://checkout.

server.registerResource(
  "latest-results",
  "test-results://latest",
  { title: "Latest Playwright Results", mimeType: "application/json" },
  async (uri) => {
    const results = await readLatestResults();
    return {
      contents: [{
        uri: uri.href,
        mimeType: "application/json",
        text: JSON.stringify(results)
      }]
    };
  }
);

Failure Analysis

server.registerTool(
  "analyze_test_failure",
  {
    title: "Analyze Test Failure",
    inputSchema: {
      testName: z.string(),
      error: z.string(),
      consoleErrors: z.array(z.string()),
      networkFailures: z.array(z.string())
    },
    outputSchema: {
      classification: z.enum([
        "APPLICATION", "TEST", "NETWORK",
        "DATA", "ENVIRONMENT", "UNKNOWN"
      ]),
      evidence: z.array(z.string())
    }
  },
  async (input) => {
    const result = await analyzeFailure(input);
    return {
      content: [{ type: "text", text: JSON.stringify(result) }],
      structuredContent: result
    };
  }
);

Separate observed evidence from AI-generated hypotheses. The AI can classify a failure and recommend an investigation path, but the underlying evidence should remain inspectable.

The UNKNOWN State

type Diagnosis =
  | "APPLICATION" | "TEST" | "NETWORK"
  | "DATA" | "ENVIRONMENT" | "UNKNOWN";

const analysisInstruction = `
Analyze only the supplied evidence.
Do not invent missing information.
If evidence is insufficient, return UNKNOWN.
Separate observed evidence from hypotheses.
`;

Allowing UNKNOWN is an important reliability control. It prevents incomplete logs from being converted into false certainty.

A QA Workflow

Human Approval for High-Risk Operations

type RiskLevel = "LOW" | "MEDIUM" | "HIGH" | "CRITICAL";

const TOOL_RISK: Record<string, RiskLevel> = {
  read_test_results: "LOW",
  run_playwright_test: "LOW",
  validate_api: "LOW",
  create_test_data: "MEDIUM",
  modify_test: "HIGH",
  delete_test_data: "CRITICAL"
};

function requiresApproval(risk: RiskLevel) {
  return risk === "HIGH" || risk === "CRITICAL";
}

The AI can request an action, but the policy layer determines whether it is permitted. Human approval is an explicit control point rather than a suggestion embedded in a prompt.

Auditability

interface AuditEvent {
  timestamp: string;
  agent: string;
  tool: string;
  arguments: unknown;
  environment: string;
  decision: "ALLOWED" | "BLOCKED" | "APPROVAL_REQUIRED";
  approvedBy?: string;
}

Every tool invocation should produce an auditable event so teams can determine which agent called which tool, against which environment, and with what authorization decision.

Connecting the Client

const { tools } = await client.listTools();

const result = await client.callTool({
  name: "run_playwright_test",
  arguments: {
    testFile: "tests/checkout.spec.ts"
  }
});

The MCP client API supports discovery and invocation of tools and access to resources. Playwright also provides an MCP server for browser automation, which can be used when the use case is direct browser interaction.

Testing the MCP Server

test("blocks unapproved test", async () => {
  await expect(
    runPlaywrightTest("tests/../../secret.spec.ts")
  ).rejects.toThrow("Test file is not approved");
});

test("requires approval for data deletion", async () => {
  const decision = authorize({
    tool: "delete_test_data",
    environment: "production"
  });
  expect(decision).toBe("BLOCKED");
});

The MCP server itself is production software. Its schemas, authorization rules, environment restrictions, error handling, and tool contracts should therefore be covered by automated tests.

Evaluation: Avoid Unsupported Benchmark Claims

A strong implementation should be evaluated rather than supported by invented numbers. Compare conventional QA with MCP-assisted QA using the same scenarios and report the actual environment and measurement method.

Metric	Measurement
Execution time	Repeated wall-clock runs
Diagnosis time	Failure to actionable hypothesis
Correct classification	Correct diagnoses / total failures
Human interventions	Manual approvals or interventions
Blocked unsafe actions	Policy violations prevented
Tool-call failures	Invalid or failed calls
Cost	LLM/tool/API cost per workflow

If numerical performance results are reported, include the actual MCP/Node/Playwright versions, machine or CI environment, workload, number of runs, warm-up method, and measurement definition. Do not present illustrative figures as measured evidence.

Production Architecture

Trade-offs

More tools increase capability but also increase the action space the model must select from. More autonomy can reduce manual effort but increases the impact of incorrect decisions. More evidence can improve diagnosis but can increase data exposure and model cost. More guardrails improve safety but can introduce friction for unusual workflows.

The engineering objective is therefore not maximum autonomy. It is useful autonomy within an explicitly defined risk boundary.

Conclusion

MCP gives QA engineers a standardized way to expose testing capabilities to AI systems, but the protocol itself does not define a safe QA architecture. That responsibility belongs to the implementation.

A production-oriented QA MCP server should expose narrow, validated capabilities; keep browser and API execution deterministic; separate tools from read-only evidence; enforce environment and permission boundaries outside the model; require human approval for high-impact actions; and record enough evidence to make agent behavior auditable.

The important shift is from “AI controls the test environment” to “AI operates through a controlled QA interface.” That distinction provides the foundation for building AI-assisted testing systems that remain useful, observable, and governable as their autonomy increases.

References

MCP TypeScript SDK v2 documentation; MCP client tool/resource APIs; MCP resources and tools documentation; Playwright MCP documentation. These are the current primary technical references used for the implementation.

Railway Environment Variable Incident: What Laravel & Cloud Application Teams Should Do Now

GeekyAnts India Pvt Ltd — Tue, 27 Oct 2026 05:10:02 +0000

On October 21, 2026, Railway disclosed that an unauthorized third party may have accessed the systems that store its customers' environment variables. That exposes a problem that rarely comes up when engineering teams discuss cloud security: what happens when the infrastructure you depend on to recover from an incident is itself affected by that incident?

If your team deploys on a platform like Railway, you have probably already been asked the same question internally: what do we do now?

This post is a practical answer. It's written for Laravel and cloud application teams, and it deliberately avoids alarm. Incidents like this are a reasonable moment to take a breather to check your own setup, and most of what follows is good hygiene you would want in place for your team anyway.

What Exactly Happened?

The facts below come from Railway's own disclosure and from press coverage published on the first day after it. Details are still developing, so treat this as a snapshot:

Railway said it detected unusual activity on October 21 in the systems that store customer environment variables.
Its initial assessment was that an unauthorized party may have accessed encrypted variables and, for some of them, obtained the decrypted contents. Reporting describes the affected values as including API keys, database credentials and private registry credentials.
Railway paused deployments while it investigated. The Defiant reported that this left customers unable to push updated applications carrying replacement credentials. Enterprise builds were restored late Wednesday, while customers on other plans were still waiting to redeploy.
Railway's early analysis suggested the impact may be limited to a subset of users, but it still recommended that every customer rotate every stored secret and review service logs for unusual activity. It has not published a detailed scope.
Railway described the incident as part of a series of attacks on platform companies aimed at compromising AI inference providers, and said it had notified upstream providers so they could proactively revoke potentially exposed keys.

One claim is still being examined as this blog was written. A developer has alleged that roughly $370,000 was stolen through secrets stored on Railway. On-chain analysts have tracked funds moving from a suspected attacker's wallet, but the link between that wallet and the Railway incident has not been confirmed.

Why Environment-Variable Exposure Matters?

Environment variables are the standard way to keep configuration out of code. In practice, they're also where applications keep the keys to everything else: database passwords, payment and email credentials, cloud storage keys, signing secrets and third-party API tokens.

That makes them an unusually high-value target. An attacker who reads a single set of variables doesn't need to find a vulnerability in your application. They already hold credentials that your application is trusted to use, and those credentials often work from anywhere on the internet.

Secrets Are Becoming the New Security Perimeter

There is also a structural point here, and it's the main lesson we'd draw from this incident. Secrets are the new perimeter. Attackers increasingly go after the layer that stores and manages credentials, rather than the services those credentials unlock. It's often the softer target, and it can be just as valuable if the stolen keys grant access to paid AI accounts, databases or cloud resources. Your application can be well-written and fully patched, and its security still depends on every platform that holds a copy of its secrets.

The practical conclusion is not "stop using managed platforms." They are vital in removing a great deal of operational burden for growing teams, and most teams are safer for using them. The lesson here is to design in a manner that a single vendor's bad day is survivable, because the credentials it holds are scoped, short-lived and quick to replace.

What Credentials To Review And Rotate Post the Railway Security Incident?

Railway's guidance was to rotate everything stored as an environment variable, and that's the right default for anyone affected. The harder part is knowing what "everything" includes. For a typical Laravel application, work through this list:

Application key (APP_KEY). This encrypts sessions, cookies and any data you encrypt with Laravel's encrypter. Rotating it will invalidate sessions and make previously encrypted data unreadable unless you plan for it. Recent Laravel versions support graceful rotation through APP_PREVIOUS_KEYS, which lets old values keep decrypting while new ones are written.
Database and cache credentials. DB_PASSWORD, any read-replica credentials, and REDIS_PASSWORD. Also check whether the database is reachable from the public internet; if it is, a leaked password is directly usable.
Cloud and storage keys. AWS, GCP or S3-compatible access keys, plus any service-account files stored as variables. Railway noted that it would handle resets for some platform-managed items, such as storage buckets and GitHub connections, but you should confirm that rather than assume it.
Third-party service tokens. Payment gateways (Stripe, Razorpay and similar), email and SMS providers, push and broadcasting services (Pusher, Ably), search and analytics keys.
AI and LLM provider keys. OpenAI, Anthropic and similar keys deserve priority. They're easy to monetize, they can run up large bills quickly, and Railway has said AI inference access appears to be the attackers' target.
Auth and signing material. OAuth client secrets, Passport or Sanctum-related keys, JWT signing secrets, and webhook signing secrets.
Registry and deployment credentials. Private container registry tokens, package registry tokens (for example private Composer repositories), deploy keys and Git provider tokens.
Wallet and signing keys. If any private key that can move money was ever stored as a variable, treat it as the most urgent item and move funds to a new key.

A sensible order is: anything that can spend money or move data first, then anything that grants access to customer data, then everything else. After rotating, check the logs of each connected service for activity you don't recognize in the window around the disclosure, and keep the old credentials revoked rather than just replaced.

Secret-Management Best Practices to Consider

Incident response is the expensive way to learn about secrets. These practices make the next one cheaper.

Keep a secrets inventory. You can't rotate what you can't list. For each secret, record what it unlocks, where it's stored, who owns it and when it was last rotated. Most teams are surprised by how many forgotten keys turn up.

Use a dedicated secrets manager for high-value secrets. Tools such as HashiCorp Vault, AWS Secrets Manager or Doppler give you access control, audit logs and rotation workflows that a plain variable store doesn't. For the most sensitive material, such as signing keys, use a KMS or HSM so the raw key never lands in your application's environment at all.

Prefer short-lived and dynamic credentials. A database password that expires in an hour is a much smaller liability than one that has been valid for three years. Where your stack supports it, issue credentials on demand rather than storing them.

Apply least privilege everywhere. Use separate keys per service and per environment. Give database users only the permissions the application needs. Put spend caps and usage limits on AI and payment keys, so that a leaked key has a ceiling on the damage it can do.

Make a stolen secret less useful. Put databases and caches behind private networking or IP allowlists, so a leaked connection string is not enough on its own. Where providers support it, restrict API keys by IP or referrer.

Keep secrets out of code, images and logs. Check your Git history, build logs, container layers and error reporters. In Laravel, remember that debug pages and verbose exception output can echo environment values if APP_DEBUG is left on in production.

Practice rotation before you need it. Treat rotation as a routine, scripted task. If it only happens during an incident, it will be slow, manual and error-prone.

CI/CD and deployment security

The Railway incident shows a point that is easy to miss: rotation is a deployment problem and not just a missed item on a security checklist. Telling teams to rotate secrets is only useful if they can then ship the new values. When deployments were paused, rotating a key and getting that key into a running application became two separate problems.

A few things make a real difference here:

Keep a path to redeploy that doesn't depend on a single provider. That doesn't require multi-cloud complexity. It can be as simple as a documented, tested way to stand up your application elsewhere, with infrastructure defined as code.
Scope CI/CD secrets tightly. Pipeline variables should be limited per environment, protected so they are only available on trusted branches, and masked in logs. Avoid one broad token that can touch every project.
Prefer short-lived identity over stored keys in pipelines. OIDC-based federation to your cloud provider lets a pipeline obtain temporary credentials without a long-lived key sitting in a variable.
Separate secrets by environment. Production credentials should not be present in staging or preview environments, where controls are usually looser.
Pin and review what your pipeline runs. Lock third-party actions and dependencies, and review changes to workflow files like code, since a modified pipeline is a direct route to your secrets.
Make rotation a one-command or one-pipeline operation. The goal is that rotating a credential and redeploying with it takes minutes, not an afternoon of coordination.
Keep audit trails. Know who changed a secret or a deployment setting and when.

How To Think About Production Security in Light of Security Incidents?

It's tempting to respond to an incident like this in one of two ways: switch vendors, or bring everything in-house. Neither is usually the answer. Switching vendors moves the risk. Running your own infrastructure after a platform breach can make things worse, because you become the next potential breach, often with a smaller security team than the platform you left. A calmer approach is to ask a different set of questions about every system that touches production.

When a credential is exposed, the real concern is how much access that single key was allowed to have. Rotating it is the immediate fix. But if the same key can expose sensitive data or disrupt several services, that points to an architectural problem we should have addressed much earlier.

— Saurabh Sahu, Chief Technology Officer (CTO)

Assume any single platform can have a bad day. The question is not whether your vendors are perfect but what happens to you when one of them isn't. If the answer is "we rotate a few scoped credentials and carry on," you're in good shape. If it's "everything we own is exposed," then the design itself needs a harder assessment.

Shrink the blast radius by design. Scoped keys, private networking, spend limits and separated environments are not exotic. Together they turn a potential catastrophe into a contained, boring incident.

Treat detection as part of the system. Alerts on unusual API usage, billing spikes and unfamiliar logins mean you may notice misuse before any vendor's email arrives. In some reports of this incident, teams had already rolled their keys before the notification reached them.

Do vendor due diligence that matches what you store. What a platform holds should determine how much scrutiny it gets. Ask where secrets are encrypted, who can decrypt them, what the incident communication process looks like, and how quickly you can recover if the platform is unavailable.

Keep your crown jewels elsewhere. Anything that can move large sums of money, sign on your behalf or unlock all customer data should live in a purpose-built secrets system, and ideally not within a general-purpose variable store.

Rehearse. Run a short exercise: pick a key, assume it leaked, and time how long it takes to revoke, replace and redeploy. The gaps you find in an hour of practice are the ones that would otherwise appear during a real incident.

How We Approached It Across 60+ Client Projects

The Railway disclosure wasn't the only upstream problem our teams were tracking this week. A critical PHP vulnerability, a Node.js remote-code-execution issue and a certificate authority key problem were all on the list too. None of it was code we had written, but all of it was ours to handle, because our clients' production systems depended on it.

Here is what that looked like in practice:

We found out what runs where. The hard part wasn't patching. It was answering "which projects are affected?" quickly.
We put everything in one tracker. Every project, its stack, its version and whether it's safe. Our DevOps team owns the sheet and makes sure every upgrade is closed, not just started.
We called the highest-risk client first. They heard it from us before they read about it somewhere else.
We resisted the wrong lesson. Moving to self-managed infrastructure after a platform breach would not have made our clients safer. It would have swapped one dependency for another risk.

You can't stop PHP, Node or a vendor from failing. The only number you own is time-to-safe: how fast you know, how fast you patch, and how fast the client hears from you.

— Saurabh Sahu, Chief Technology Officer (CTO)

That is the idea behind everything in this post. Rotating a secret, patching a runtime and telling a client are all the same measurement: how quickly you go from "something happened" to "we know where we stand."

So here is a question worth asking your own team: how long would it take to answer "what version of Node (or PHP) is every production service on?" If the answer is more than an hour, fix that before you fix anything else.

Where Geekyants Can Help

Most teams are familiar with an ideal security stance. The difficulty is finding the time to do it while also shipping features. This is where we typically work with clients:

Production security reviews. We audit how secrets, access and deployments are handled across your application and infrastructure, and give you a prioritized list of fixes rather than a long report.
Cloud architecture and modernization. We help teams move from single-vendor, static-credential setups to architectures with secrets management, private networking, infrastructure as code and portable deployments.
DevOps and CI/CD hardening. We build pipelines where rotation and redeployment are routine, and where credentials are short-lived and scoped.
Laravel engineering. If your application is built on Laravel and you need senior help with configuration, key rotation, queue and cache setup or a broader hardening pass, you can hire Laravel developers from GeekyAnts who work on production systems every day.

If you're working through your own rotation right now, or you'd like a second pair of eyes on your setup once the dust settles, we're happy to talk it through.

A short checklist for this week

Rotate every secret stored on the affected platform, starting with anything that can spend money or access customer data.
Check the logs and billing of each connected service for unfamiliar activity.
Confirm your database and cache are not reachable from the open internet.
Build a secrets inventory with an owner and a rotation date for each entry.
Move your highest-value secrets into a dedicated secrets manager or KMS.
Add spend caps and usage alerts to AI and payment keys.
Test that you can redeploy quickly with new credentials, and write the steps down.
Keep one tracker of every project, its stack, its versions and its status, so you can answer "what runs where?" in under an hour.