DEV Community: GeekyAnts India Pvt Ltd

Taming Elasticsearch at Scale

GeekyAnts India Pvt Ltd — Fri, 23 Oct 2026 11:03:11 +0000

Optimize Elasticsearch for millions of records with better pagination, leaner payloads, efficient filtering, faster ingestion, and frontend techniques that keep search responsive with increase in data.

Elasticsearch is useful for searching an increasing dataset that starts adding pressure on the database and the application. Instead of checking millions of records one by one, it builds an index that lets the system locate relevant data without scanning the entire dataset for every request. This makes it useful for full-text search, filtering, logs, product catalogs, reports, and other applications where users expect results as they type or filter.

But adding Elasticsearch does not automatically make a search system fast. Query design, index mappings, response size, data ingestion, and frontend rendering can all affect how quickly a search feels to the person using it.

Here is a full-stack guide to searching millions of records without freezing the frontend.

It's a story most of us have lived through in some form.

Your database starts groaning under the weight of a few million records. Every LIKE %query% clause is a small act of violence against your CPU. So the team makes the obvious call: bring in Elasticsearch.

For a while, it's magic. Queries come back in 15 milliseconds. Full-text search just works. Everyone's happy.

Then, six months later, the index has grown into something with millions of dense, deeply nested documents, and the complaints start rolling in: "Every time I search the reports page, the screen freezes for four seconds, my fan spins up, and sometimes the tab just dies."

Which is confusing, because Elasticsearch is built to query petabytes in milliseconds. So why does it feel like the frontend is choking?

In practice, it almost never has anything to do with Elasticsearch's raw power. The real bottleneck is nearly always some combination of how the queries are written, how the data was indexed in the first place, how much JSON gets shipped over the wire, and what the browser is asked to do with it once it arrives.

This is a walk through the places that tend to go wrong — on the backend, at index time, and in the browser — and what to do instead.

Six independent bottlenecks — backend, indexing, and frontend — each with a small, specific fix.

1. The deep pagination trap

Say you're building a paginated table. The natural instinct is page-number pagination:

// Page 100, 50 items per page
{
  "from": 5000,
  "size": 50,
  "query": { "match_all": {} }
}

Looks harmless enough. Here's what it actually does to your cluster.

If your index is spread across 5 shards and you ask for items 5,000–5,050, each shard has to hand its top 5,050 documents to the coordinator node — not just the 50 you want. That's 5 × 5,050 = 25,250 documents pulled into memory, sorted, and then almost entirely thrown away, just so the coordinator can hand you the 50 that were left over.

By the time users are clicking into page 200, your cluster is burning most of its CPU sorting documents it's about to discard. Push it far enough and Elasticsearch will simply refuse — from + size beyond 10,000 hits the index.max_result_window limit by default. (Plenty of teams just raise that limit instead of fixing the underlying issue, which only postpones the memory problem.)

There isn't one correct fix here — it depends on how people actually navigate your UI.

If you have clickable page numbers ("1, 2, 3 ... 25") and people genuinely jump around, from / size is fine. Just cap how deep it goes. Restricting navigation to roughly the first 200 pages (from + size < 10,000) is safe for almost any real-world dataset.

If you have infinite scroll or a "Load more" button, reach for search_after instead:

{
  "size": 50,
  "query": { "match_all": {} },
  "sort": [
    { "created_at": "desc" },
    { "_id": "asc" }
  ],
  "search_after": ["2026-09-20T14:30:00Z", "doc-10482"]
}

Instead of asking for "page 50," you're saying "give me whatever comes right after this document." Each shard jumps straight to that point in its sorted index — no sorting through thousands of discarded documents, no memory spike. Item 50 and item 5,000,000 cost roughly the same.

The catch: search_after only moves forward. There's no jumping to an arbitrary page — it's a cursor, not a page number, and your UI needs to reflect that.

from/size forces every shard to hand over documents it then throws away. search_after skips straight to where it left off.

If you're exporting thousands of rows to CSV or PDF, use a Point-in-Time (PIT) reader combined with search_after. PIT freezes a consistent snapshot of the index, so the export doesn't shift under you if documents get added mid-run.

2. Stop sending the whole warehouse back

Documents in a real system tend to accumulate weight. Customer profiles, nested inspection logs, historical audit trails — it's not unusual for a single document to sit at 50–200 KB.

Now return 50 of those without restricting fields, and you're shipping something like 5 MB of JSON for a single page load. And what actually happens with that 5 MB?

It takes over a second just to download.

The browser has to parse a multi-megabyte string, which locks up the main thread while it does.

The UI stutters, animations skip, and the user assumes something's broken.

Meanwhile, your table is probably showing five columns: ID, code, date, inspector, status.

The fix is boring but effective — tell Elasticsearch exactly which fields you actually need:

{
  "size": 50,
  "_source": [
    "id",
    "report_code",
    "start_date",
    "inspector_name",
    "status"
  ],
  "query": { ... }
}

Strip out the nested arrays and raw submission blobs you're not displaying, and that 5 MB response often drops to something like 40 KB. At that size, parsing takes a couple of milliseconds and the UI never even notices.

3. Don't score things that can't be scored

Elasticsearch draws a real distinction between two kinds of query clauses:

Query context (must, should) — "how well does this match?" It runs BM25 scoring and produces a _score. Filter context (filter, must_not) — "does this match, yes or no?" No scoring. Purely binary.

A mistake that shows up constantly:

// Slow — forces scoring on things that don't have a "degree" of matching
{
  "query": {
    "bool": {
      "must": [
        { "match": { "status": "approved" } },
        { "range": { "start_date": { "gte": "2026-01-01" } } },
        { "match": { "project_name": "Hospital" } }
      ]
    }
  }
}

Is one "approved" report more approved than another? Obviously not — it's binary. But by putting exact-match conditions like status and date inside must, you're asking Elasticsearch to spend CPU scoring things that were never going to have a meaningful score. Worse, clauses inside must skip the filter cache entirely.

Move anything binary into filter:

// Fast — only the full-text part gets scored
{
  "query": {
    "bool": {
      "must": [
        { "match": { "project_name": "Hospital" } }
      ],
      "filter": [
        { "term": { "status.keyword": "approved" } },
        { "range": { "start_date": { "gte": "2026-01-01" } } }
      ]
    }
  }
}

The payoff isn't just skipping scoring — Elasticsearch also caches filter clauses as bitmap structures in memory. The next time anyone filters by status: approved, that part of the query resolves in microseconds without touching the disk.

A decent rule of thumb: if a field is either "yes" or "no," with no concept of how well it matches, it belongs in the filter.

Same conditions, two different execution paths. Binary conditions belong on the right.

4. The leading wildcard problem

When people want search-as-you-type or partial matching, the tempting shortcut looks like this:

{
  "query": {
    "query_string": {
      "query": "*concrete*",
      "fields": ["notes", "sample_description"]
    }
  }
}

A leading wildcard — *concrete, not concrete* — essentially disables the inverted index. Think of the index like the back of a textbook: search for "concrete" and you flip straight to the right page. Search for *concrete* and the book is useless to you — you're back to reading every word on every page, because there's no way to know in advance what might precede "concrete."

Run that across a few million documents and you get exactly the CPU spikes and multi-second delays you'd expect.

A leading wildcard defeats the inverted index. A prefix built at index time turns the same lookup into one hop.

The real fix happens at index time, not query time. The simplest route is Elasticsearch's built-in search_as_you_type field type. First, declare the field this way when you create (or update) your mapping:

PUT /reports
{
  "mappings": {
    "properties": {
      "sample_description": {
        "type": "search_as_you_type"
      }
    }
  }
}

Elasticsearch quietly builds sub-fields behind the scenes (sample_description._2gram, ._3gram, and so on) that store prefix tokens. The word "concrete" ends up indexed as something like ["co", "con", "conc", "concr", "concre", "concret", "concrete"]. Then query with bool_prefix instead of a wildcard:

{
  "query": {
    "multi_match": {
      "query": "conc",
      "type": "bool_prefix",
      "fields": [
        "sample_description",
        "sample_description._2gram",
        "sample_description._3gram"
      ]
    }
  }
}

Now typing "conc" resolves as a direct term lookup against pre-built tokens instead of a full scan of the cluster.

The tradeoff: n-gram indexing inflates the size of the affected fields, typically by something like 2–3x, since you're now storing every prefix variant on disk. For most applications that's a non-issue next to the speed gain — but if storage is tight, you can bound the range (min_gram: 3, max_gram: 10) to keep it in check.

5. The part everyone forgets: how you write data in

Most of the attention in Elasticsearch tuning goes to queries, but a surprising share of real-world slowness starts at ingestion. This pattern shows up in almost every codebase the first time a team wires up Elasticsearch:

# One HTTP request per document — the classic mistake
reports = get_all_pending_reports()
for report in reports:
    document = build_document(report)
    elasticsearch.index(index="reports", body=document)

5,000 reports means 5,000 separate HTTP round-trips, each one negotiating its own connection and potentially triggering a refresh. What should take a few seconds can easily stretch into 10–15 minutes.

The Bulk API exists precisely for this — batch hundreds or thousands of operations into a single request:

reports = get_all_pending_reports()
batch = []
for report in reports:
    document = build_document(report)
    batch.append({ "index": { "_index": "reports" } })
    batch.append(document)
    if len(batch) >= 1000: # 500 docs per flush (action + doc = 2 lines each)
        elasticsearch.bulk(body=batch)
        batch = []
if batch:
    elasticsearch.bulk(body=batch) # flush whatever's left

Batching 500 documents per call turns 5,000 individual requests into 10. The difference is dramatic in practice — bulk indexing typically runs somewhere around 10–50x faster than indexing one document at a time.

Same 5,000 documents, radically fewer round trips

One request per document

Two smaller things worth doing alongside this:

Loosen the refresh interval during heavy writes. By default, Elasticsearch refreshes every index every second so new documents become searchable almost immediately. During a large backfill, that constant refreshing just burns CPU for no benefit. Turn it down while you're bulk-loading:

PUT /reports/_settings
{ "index": { "refresh_interval": "30s" } }

and set it back once you're done:

PUT /reports/_settings
{ "index": { "refresh_interval": "1s" } }

Give documents a stable ID. If a pipeline ever retries after a partial failure, it can easily index the same record twice. Use a consistent ID — your database primary key works well — so Elasticsearch overwrites the old copy instead of quietly duplicating it:

# Passing "id" turns this into an upsert — no duplicate rows
elasticsearch.index(index="reports", id=report.id, body=document)

6. Don't let the browser undo all of that work

Even a 20-millisecond Elasticsearch response can still end in a frozen tab if the frontend isn't careful. Three things worth having in place:

Debounce input. Typing "Bridge Inspection" is 17 keystrokes in about a second and a half. Fire a request per keystroke and you've just sent 17 overlapping searches. A 300ms debounce — wait until the user actually pauses — fixes this outright.

Cancel stale requests. If someone types "B" and then quickly finishes "Bridge," the request for "B" is now irrelevant. If it happens to resolve after the "Bridge" request, your table briefly shows the wrong results. AbortController handles this cleanly:

let currentController = null;
function fetchSearchResults(query) {
  if (currentController) {
    currentController.abort(); // kill the previous request
  }
  currentController = new AbortController();
  fetch(`/api/search?q=${encodeURIComponent(query)}`, {
    signal: currentController.signal
  })
    .then(res => res.json())
    .then(data => renderTable(data.items))
    .catch(err => {
      if (err.name !== 'AbortError') console.error('Fetch error:', err);
    });
}

Batch your DOM writes. Inserting 200 rows one at a time into a live table forces the browser to recalculate layout after each insertion. Build everything off-screen first, then insert it in one shot:

function renderTable(items) {
  const tableBody = document.getElementById('tableBody');
  const fragment = document.createDocumentFragment();
  items.forEach(item => {
    const row = document.createElement('tr');
    row.innerHTML = `
      <td>${item.id}</td>
      <td>${item.report_code}</td>
      <td>${item.inspector_name}</td>
      <td>${item.status}</td>
    `;
    fragment.appendChild(row);
  });
  tableBody.innerHTML = '';
  tableBody.appendChild(fragment); // single repaint
}

Worth flagging: building rows this way with innerHTML is only safe if none of those fields can contain arbitrary user text. If inspector_name or similar could ever hold something a user typed, that's an XSS opening — building rows with textContent instead closes it off.

Past 500 rows or so, consider a virtualized table that only renders the 15–20 rows actually visible in the viewport, rather than materializing the entire result set as DOM nodes.

What it looks like put together

None of these changes are individually dramatic. Together, they compound. For an application with roughly 800,000 indexed documents, applying all six optimizations could produce improvements in this range:

Metric	Before	After
Search response time	~3.2s	~45ms
JSON payload size	~4.8 MB	~38 KB
Frontend render time	~1,400ms	~12ms
Bulk reindex (5,000 docs)	~14 min	~8 sec
Cluster CPU during search	~78% sustained	Single-digit spikes

(These are illustrative figures based on the kind of improvement this set of changes typically produces — worth replacing with your own measured numbers once you've made the changes on your own system.)

Every bar comes from the before/after table above — same numbers, easier to compare at a glance.

A 45ms response with a 38KB payload that renders in 12ms doesn't just feel fast — it feels instant, even on a phone over 4G.

Cheatsheet

Use this cheatsheet to quickly match common Elasticsearch performance issues with their likely cause and fix.

Symptom	What's Actually Happening	Fix
Deep pagination (`from: 5000`)	Coordinator sorts and discards thousands of docs per shard	`search_after` for infinite scroll; cap `from/size` for page-number UIs
Giant payloads (5MB JSON)	Full documents returned when only a few fields are shown	`_source` filtering
Slow filtering	Exact-match fields scored unnecessarily	Move to `bool.filter` for caching
Leading wildcards (`query`)	Inverted index can't be used, full scan instead	`search_as_you_type` + n-grams
Slow bulk imports	One HTTP request per document	Bulk API, batched
UI freezes while typing	No debounce, requests racing	300ms debounce + `AbortController`
Table rendering stutters	DOM reflow on every row insert	`DocumentFragment` or virtualization

The Underlying Idea

Every fix above is really the same move, applied in a different place: figure out where work is happening, and ask whether it's happening at the right time.

Push computation to index time (n-grams, explicit mappings) instead of query time (wildcards). Push filtering into the cache layer (bool.filter) instead of the scoring layer (bool.must). Push payload decisions onto the backend (_source filtering) instead of asking the browser to parse everything and use a fraction of it. Push DOM construction into memory (DocumentFragment) instead of doing it row-by-row on the live page.

If you want a concrete place to start: open your slowest search endpoint right now and add _source filtering so it only returns the fields your UI actually shows. Check the response size before and after. That alone is usually a five-minute change that cuts payload by 90% and makes the page feel noticeably different.

Everything else in this piece can follow from there, one endpoint at a time.

Turn Elasticsearch Performance Into Product Performance

A fast Elasticsearch query means little if the API sends megabytes of JSON or the browser spends a second rendering it. Search performance has to be addressed across indexing, APIs, infrastructure, and the frontend. GeekyAnts helps teams optimize these systems through enterprise backend development and modernization, covering backend architecture, data pipelines, performance tuning, and systems built to handle growing data volumes.

The goal is simple: keep search responsive as the dataset grows, without making every new record or user request more expensive than it needs to be.

Building a Real-Time Photo Sync Video Experience Using 100ms SDK and Reactive State Management

GeekyAnts India Pvt Ltd — Fri, 23 Oct 2026 10:08:14 +0000

This blog explains how to build a real-time video experience with synchronized photo sharing, chat, and reactive state management using the 100ms SDK.

Building real-time video experiences traditionally required deep expertise in WebRTC, signaling infrastructure, media synchronization, and browser-specific handling. With modern SDKs like 100ms, teams can now focus more on product experience and less on low-level media infrastructure.

In this article, we will walk through how to build a real-time video calling experience with synchronized photo sharing and chat, powered by the 100ms JavaScript SDK, a reactive state architecture, and a modular frontend design.

We will cover:

Core concepts of 100ms architecture
Pre-join (green room) experience
Joining and managing a live video room
Real-time photo synchronization across participants
Chat integration using broadcast messaging
State management strategies for responsive UI
Synchronization correctness, conflict resolution, and reconnect handling

High-Level Architecture

Before diving into the implementation, it’s worth understanding how the system components interact.

On the backend, when a meetup is created, the server calls the 100ms Rooms API to provision a room and generate authentication tokens for invited participants. Each participant receives a token tied to the same room ID, ensuring everyone joins the same session.

On the frontend, a single HMSReactiveStore instance acts as the centralized source of truth for media state, participants, tracks, and messages. Both the Green Room and the Live Call screens subscribe to slices of this store rather than maintaining disconnected local state.\
The SDK exposes three key primitives:

Store → Reactive room state
Actions → Methods for operations such as join, mute, or messaging
Notifications → Event listeners for peer updates and room events

This architecture provides a predictable data flow for real-time applications and keeps media state, participant state, and UI subscriptions centralized. The synchronized-photo feature still requires explicit conflict-resolution and lifecycle handling, which we address below.

Why a Pre-Join (Green Room) Matters

A pre-join experience is critical for user confidence and system readiness.\
100ms provides a preview API that allows users to:

Test camera and microphone
Set initial mute/unmute states
Validate network readiness and authentication before joining

Key Implementation Flow

Fetch authentication token from backend
Call preview() with user config
Attach local video track to UI
Allow user to toggle mic and camera

await hmsActions.preview({
 authToken: '<token from backend>',
 userName: 'Alice',
});

Once preview is successful:

Local tracks are created
Permissions are verified
Settings persist into the join phase

This reduces friction during room entry.

Setting Up the 100ms SDK

The 100ms JavaScript SDK ships through @100mslive/hms-video-store.\
Everything flows from a single HMSReactiveStore instance:

// Initialize once — treated as a module-level singleton
import { HMSReactiveStore } from '@100mslive/hms-video-store';
const hmsManager = new HMSReactiveStore();
hmsManager.triggerOnSubscribe(); // fire callbacks immediately with current state
const hmsStore = hmsManager.getStore();
const hmsActions = hmsManager.getActions();
const hmsNotifications = hmsManager.getNotifications();

Calling triggerOnSubscribe() is important because it ensures subscribers immediately receive the current SDK state instead of waiting for the next state update.

Reading State with Selectors

The store exposes state through memoized selectors rather than raw state objects. This enables efficient reactive subscriptions.

import {
  selectPeers,
  selectIsLocalVideoEnabled,
  selectIsLocalAudioEnabled,
  selectLocalVideoTrackID,
  selectLocalPeer,
  selectTrackByID,

  selectPeerByID
} from '@100mslive/hms-video-store';

// One-time read
const isVideoOn = hmsStore.getState(selectIsLocalVideoEnabled);

// Reactive subscription — returns an unsubscribe function
const unsubscribe = hmsStore.subscribe((enabled) => {
  // runs whenever local video enabled/disabled
  updateVideoButton(enabled);
}, selectIsLocalVideoEnabled);

Subscribing to fine-grained selectors helps avoid unnecessary re-renders. In our implementation, this was useful for participant and track updates because changes could be scoped to the affected selector rather than causing broad UI updates.

The Green Room: Pre-Join Experience

The Green Room acts as a staging area before users join the room.

Attaching Local Video

Once preview mode starts, we subscribe to the local video track and attach it to a <video> element.

// Wire local video into a <video> element
hmsStore.subscribe(async (trackId) => {
  if (trackId && videoElement) {
    await hmsActions.attachVideo(trackId, videoElement);
  } else if (videoElement) {
    videoElement.srcObject = null;
  }
}, selectLocalVideoTrackID);

If the user disables their camera, the video track disappears and the UI reflects the change.

Toggling Camera and Microphone

Media controls during preview work the same way as they do during a live call.

// Toggle camera
const next = !hmsStore.getState(selectIsLocalVideoEnabled);
await hmsActions.setLocalVideoEnabled(next);

// Toggle mic
const next = !hmsStore.getState(selectIsLocalAudioEnabled);
await hmsActions.setLocalAudioEnabled(next);

Because the UI subscribes to the store, button states remain synchronized.

Transitioning to the Live Call

When the user clicks "Join", we stop the preview session first by calling hmsActions.leave(). This releases the preview peer from the 100ms session and prevents a ghost local peer from remaining alongside the new peer after the user joins. After leave completes, we navigate to the Live Call route.

The camera/microphone preferences chosen in the Green Room are persisted to a Pinia store (with localStorage persistence) so the Live Call can restore them on mount without asking the user again.\
To join a room, 100ms requires:

Auth token
Username

The join flow is straightforward:

await hmsActions.join({
  authToken: savedToken,
  userName:  'Alice',
});

// Restore preferences from Green Room
await Promise.all([
  hmsActions.setLocalVideoEnabled(preferredCameraOn),
  hmsActions.setLocalAudioEnabled(preferredMicOn),
]);

After joining:

The SDK emits events such as ON_JOIN, ON_PEER_UPDATE, and ON_TRACK_UPDATE
Reactive store subscriptions keep the UI aligned with the resulting SDK state

Rendering the Peer Strip

All connected peers are available via the selectPeers selector. We subscribe to it and re-render the participant strip whenever anyone joins or leaves. The local peer is sorted to the front; remote peers are sorted by join time.

Each peer tile has a <video> element whose DOM reference is stored in a Map keyed by peer ID. When a peer's video track changes (e.g. they turn their camera on), we call attachVideo to bind the new track. When a peer leaves, we call detachVideo to clean up.

// Subscribe to the peer list — fires on every join/leave
hmsStore.subscribe((peers) => {
  renderParticipants(peers);
}, selectPeers);

// Attach video for a specific peer
async function attachPeerVideo(peer, videoEl) {
  if (peer.videoTrack) {
    await hmsActions.attachVideo(peer.videoTrack, videoEl);
  }
}

This provides:

UI updates when participants join or leave
Consistent track state across components

The store acts as a single source of truth for SDK-backed room state, while application-specific state such as the current synchronized photo is handled separately with explicit synchronization rules.

Showing Live Audio and Video Status

For remote peers, track enabled/disabled state is read from the store using selectTrackByID. For the local peer, we use selectIsLocalVideoEnabled and selectIsLocalAudioEnabled directly. This allows the mic icon next to each participant tile to update in real time as participants toggle their devices, without polling.

await hmsActions.attachVideo(trackId, videoElement);

Important considerations:

Handle track changes dynamically
Detach video when track is removed
Retry attaching in case of async delays

Media toggling is handled via actions:

await hmsActions.setLocalVideoEnabled(true);
await hmsActions.setLocalAudioEnabled(false);

State is synced using selectors:

selectIsLocalVideoEnabled
selectIsLocalAudioEnabled

Persisting Preferences

Camera and microphone preferences are stored locally and reused:

During preview
During join

This creates a consistent user experience across sessions.

Real-Time Chat via Broadcast Messages

Rather than spinning up a separate WebSocket or a third-party chat service, we used 100ms's built-in messaging primitives. hmsActions.sendBroadcastMessage(text) delivers a message to every peer in the room. The SDK stores all messages in HMSStore and exposes them via the selectHMSMessages selector.

// Send a chat message
await hmsActions.sendBroadcastMessage(userInput);

// React to new messages
hmsStore.subscribe((messages) => {
  // Filter to 'chat' type only — other types are used internally (e.g. photo sync)
  const chatMessages = messages.filter(m => m.type === 'chat');
  renderMessages(chatMessages);
}, selectHMSMessages);

One important design decision: messages are typed. The default type for a regular chat message is 'chat'. We use a custom type 'PHOTO_SYNC' for the carousel sync messages described in the next section. The chat panel filters to type === 'chat' so photo-sync events never appear as chat bubbles.

Unread Message Badge

We track how many chat messages were visible the last time the panel was open (lastReadCount). When a new message arrives while the panel is closed and the sender isn't the local peer, we set an unread flag that drives a red dot on the chat button. When the panel opens, we reset the count.

Synced Image Carousel Across All Peers

The synchronized carousel required a separate state-management approach. We wanted participants to converge on the same photo when someone navigated the carousel. We built the synchronization on top of 100ms's broadcast message API, with no additional signaling infrastructure.

An important distinction is that these messages are navigation events, not durable shared state. Broadcast messaging works for transient actions such as ‘move to photo 5,’ but it should not serve as the canonical state for late joiners. If the application needs the current photo to survive missed messages or reconnects, the 100ms Session Store is a better fit for a canonical session-level state.

Defining the Sync Message Shape

We define a typed interface for the payload so that both the sender and all receivers agree on the structure:

interface PhotoSyncMessage {
  type:       'navigate_photo';
  photoId:    string; // Primary resolution: unique ID of the photo
  navigatorId: string; // which peer triggered the change
  timestamp:  number; // Diagnostic metadata (clocks may drift, do not use for ordering)
  sequenceId: number; // Monotonic counter to reject stale messages

Note on Synchronization Correctness

To prevent concurrent navigation conflicts (e.g., two users changing photos simultaneously) and distributed clock drift issues, this architecture enforces an authoritative navigator model. The presenter acts as the authoritative navigator. Navigation initiated by other participants is ignored by receivers, preventing simultaneous navigation from producing conflicting shared state. Each accepted navigation event also carries a monotonic sequence ID, allowing receivers to reject stale events.

Sending a Navigation Event

Whenever the local user taps a thumbnail or presses a carousel arrow, we broadcast a PHOTO_SYNC message alongside updating local state:

const AUTHORITATIVE_ROLE = 'presenter';
// Module-level sequence tracker 
let localSequenceId = 0;

async function handlePhotoSelected(photo) {
  // Only the authoritative navigator can initiate synced navigation 
  if (currentUser.roleName !== AUTHORITATIVE_ROLE) { return; }
  localSequenceId++;
  store.setCurrentPhotoId(photo.id);  // local update
  await hmsActions.sendBroadcastMessage(
    JSON.stringify({
      type:        'navigate_photo',
      photoId:     photo.id,
      navigatorId: currentUser.id,
      timestamp:   Date.now(),
      sequenceId:  localSequenceId
    }),
    'PHOTO_SYNC'         // custom type keeps it separate from chat
  );
try { 
  await hmsActions.sessionStore.set('canonical_photo_id', photo.id); 
} catch (error) { 
  console.warn('Failed to update canonical session state', error); }
}

Receiving and Applying Navigation Events

On the receiver side, we hook into hmsNotifications.onNotification(), filter to NEW_MESSAGE events with our custom type, parse and validate the payload, and resolve the target photo by its stable ID before updating local state.

// Track the latest applied sync sequence for the room
let lastSyncSequence = 0;
// Define which role is allowed to navigate to prevent simultaneous conflicts
const AUTHORITATIVE_ROLE = 'presenter'; 

hmsNotifications.onNotification((notification) => {
  if (notification.type !== 'NEW_MESSAGE') return;

  let payload;
  try { payload = JSON.parse(notification.data.message); } 
  catch { return; }

  if (payload.type !== 'navigate_photo') return;

  // --- 1. AUTHORITATIVE NAVIGATOR CHECK ---

// Restrict synchronization to an authoritative peer role.
// This prevents multiple participants from independently
// updating the shared navigation state.

  const senderPeer = hmsStore.getState(selectPeerByID(payload.navigatorId));
  if (senderPeer?.roleName !== AUTHORITATIVE_ROLE) {
    console.warn('Rejected navigation event from non-authoritative peer');
    return; 
  }

  // --- 2. DETERMINISTIC SEQUENCE ORDERING ---
  // Reject out-of-order/stale messages from the authoritative peer
  if (payload.sequenceId <= lastSyncSequence) {
    console.warn('Rejected stale navigation sequence');
    return;
  }

  // --- 3. STABLE ID RESOLUTION (No Index Fallback) ---
  // Match strictly by stable photoId. If lists diverge and ID is unknown, 
  // do not guess using index. Ignore event and rely on Session Store recovery.
  const targetPhoto = photos.find(p => p.id === payload.photoId);
  if (!targetPhoto) {
    console.warn('Photo ID not found in local state, ignoring event.');
    return;
  }

  // Apply state and update tracker
  store.setCurrentPhotoId(targetPhoto.id);
  lastSyncSequence = payload.sequenceId;
});

Syncing New Joiners

A participant who joins mid-session could see the carousel start at index 0 while the rest of the room has moved to another photo. For late joiners, the implementation relies on the 100ms Session Store instead of allowing multiple peers to broadcast competing states. When a user joins, they simply read canonical_photo_id from the Session Store to land on the correct image.

Persisting Media Preferences Between Screens

The user selects their camera and mic state in the Green Room. These preferences need to persist through the route change to the Live Call screen and a page refresh.

We store preferredCameraOn and preferredMicOn in a Pinia store configured with selective localStorage persistence. Only these two fields are persisted; sensitive data like the HMS auth token is kept in memory only and is re-fetched on every session.

// Pinia store — persist only media prefs
persist: {
  key:     'app-meetup',
  storage: localStorage,
  paths:   ['preferredCameraOn', 'preferredMicOn'],
}

When the Live Call screen mount, we read these two values back and call setLocalVideoEnabled / setLocalAudioEnabled immediately after join() resolves. This means the peer publishes with the camera and microphone state configured in the Green Room.

Live Session Teardown

Proper session teardown prevents stale participant and media state.

If a participant disconnects without completing session teardown, other users may see stale or ‘ghost’ participants for a short period.

We handle cleanup in three places:

Explicit leave button
Browser unload events
Component unmount lifecycle hooks

For browser lifecycle handling, do not rely on unload alone. Modern browsers may skip or limit unload processing, so lifecycle handling should account for pagehide, beforeunload, and freeze where appropriate. These events should be treated as best-effort lifecycle signals rather than a guarantee that an awaited asynchronous leave() operation will finish before the page disappears.

async function explicitTeardown() {
  unsubscribers.forEach(unsub => unsub?.());
  // For explicit leave button, we safely await
  await hmsActions.leave();
}

// For browser lifecycle (closing tab, backgrounding, freezing)
function setupBrowserLifecycleTeardown() {
  const bestEffortCleanup = () => {
    unsubscribers.forEach(unsub => unsub?.());
    // Do not await leave() during lifecycle events; it's a best-effort fire-and-forget
    hmsActions.leave().catch(console.error);
  };

  // Modern alternatives to the deprecated 'unload' event
  window.addEventListener('pagehide', bestEffortCleanup);
  window.addEventListener('beforeunload', bestEffortCleanup);
  document.addEventListener('freeze', bestEffortCleanup);
}

For an explicit leave action or route transition, the async leave() call can be awaited normally. For browser termination events, cleanup should be best-effort and should not assume asynchronous work will always complete. The application should also remove subscriptions and detach UI references during component teardown, while relying on the SDK/browser lifecycle for connection termination.

Performance and UX Considerations

1. Avoid Duplicate State

Use the reactive store instead of local component state wherever possible.

2. Lazy Rendering

Attach video tracks only when elements are mounted.

3. Optimistic UI Updates

Update the UI when the user changes a photo, then synchronize the change.

4. Graceful Error Handling

Token failures
Permission issues
Network interruptions
Reconnects and late joiners
Unknown or stale PHOTO_SYNC messages

For synchronized navigation, network interruption is especially important: a participant may miss transient broadcast messages. The design should recover from canonical session-level state when the product requires convergence after reconnect.

Synchronization Validation and Failure Cases

Before describing a real-time synchronization flow as production-ready, it is important to validate failure modes rather than relying only on the happy path. Useful acceptance checks include:

Rapid navigation: repeatedly move forward/backward and verify that older messages do not overwrite the latest accepted sequence.
Simultaneous navigation: have two participants navigate at the same time and verify that the defined authority/conflict-resolution rule produces one deterministic final state.
Late join: join after the room has already moved through several photos and verify that the participant converges to the canonical current photo rather than starting at index 0.
Reconnect: interrupt a participant's network connection, restore it, and verify that the participant recovers the current session state.
Divergent photo ordering: use stable photoId matching rather than assuming every participant's array has identical ordering.
Browser lifecycle: leave through the explicit button, route away, background/freeze the page, and close the tab to verify that subscriptions and media references are cleaned up as far as the browser lifecycle permits.

These checks make the synchronization guarantees explicit. When behavior has not been measured under a defined participant count or network condition, describe the implementation as a maintainable foundation rather than making an absolute scalability or production-readiness claim.

Extensibility Opportunities

This architecture can support additional capabilities:

Screen sharing
Role-based permissions
Recording and streaming
Collaborative annotations
Session-level shared state using the Session Store
Deterministic conflict resolution for collaborative controls

Key Takeaways

Building this feature reinforced several architectural patterns for real-time applications:

1. Centralized reactive state simplifies synchronization

Maintaining a single HMSReactiveStore instance prevents UI desynchronization and keeps media state consistent across screens.

2. Preview mode improves pre-join UX

The Green Room gives users confidence and control before entering the live session.

3. Broadcast messaging can support multiple event types

We used the same messaging layer for chat and photo-navigation events by namespacing message types. For durable shared state, such as the current photo for late joiners, session-level state is a better model.

4. Fine-grained subscriptions improve performance

Track-level subscriptions prevent unnecessary participant list re-renders.

5. Clean teardown logic matters

Explicit transitions should await leave(), while browser lifecycle cleanup must be treated as best-effort and designed around modern lifecycle events rather than assuming unload will complete asynchronous work.

Conclusion

Real-time collaboration is no longer limited to basic video calls.

Combining 100ms’s reactive SDK architecture with a structured frontend state model supports interactive shared experiences without requiring a separate synchronization layer for every feature.

The synchronized-media feature also highlights an important engineering lesson: real-time UX requires more than broadcasting an event. Correctness depends on defining the source of truth, resolving concurrent updates, rejecting stale messages, handling late joiners and reconnects, and managing cleanup across browser lifecycle states.

With those considerations in place, the patterns described here provide a maintainable foundation for collaborative experiences involving shared media, live messaging, and reactive participant management.\
As real-time applications expand beyond traditional conferencing, reactive synchronization, centralized media state, and event-driven collaboration provide a foundation for shared digital experiences.

For teams building real-time video and interactive media products, GeekyAnts provides Media & Entertainment software development services for developing and scaling digital media experiences.

From Test Failure to Root Cause: Building an AI-Assisted Playwright Failure Analysis Pipeline

GeekyAnts India Pvt Ltd — Mon, 12 Oct 2026 10:05:32 +0000

This blog explains how an AI-assisted Playwright pipeline can analyze test failure evidence, identify probable root causes, and support human-validated debugging.

Introduction

Modern applications are increasingly tested through automated CI/CD pipelines. Tools such as Playwright allow teams to execute hundreds or thousands of browser tests consistently across environments. However, automation solves only one part of the problem: detecting that something failed.

The more time-consuming problem often starts after the failure.

A CI pipeline may report:

FAILED checkout.spec.ts 

TimeoutError: 

Expected "Order confirmed" to be visible

But this message does not immediately tell the engineer what actually went wrong.

Was the application broken?
Was the API unavailable?
Was the test data incorrect?
Did a network request fail?
Was the locator outdated?
Was there a race condition?

As automation suites grow, engineers can spend considerable time collecting screenshots, checking logs, opening traces, inspecting network requests, and reproducing failures locally.

This creates an opportunity for AI.

Instead of allowing AI to execute or modify tests autonomously, we can use it as a failure-analysis layer that consumes the evidence already produced by Playwright and generates an evidence-backed diagnosis for an engineer to validate.

The resulting workflow is:

The goal is not to replace the QA engineer, but to reduce the distance between ‘the test failed’ and ‘this is why it probably failed.’

The Problem With Traditional Test Failure Analysis

A typical Playwright test might look like this:

test("user can complete checkout", async ({ page }) => { 

await page.goto("/checkout"); 

await page.getByRole("button", { 

name: "Place Order" 

}).click(); 

await expect( 

page.getByText("Order confirmed") 

).toBeVisible(); 

});

Suppose the test fails. The CI system might only show:

TimeoutError:

"Order confirmed" was not visible

The underlying cause could exist at several layers:

A useful failure-analysis system therefore needs more context than the final assertion message.

Turning Playwright Into an Evidence Source

Playwright already provides many of the artifacts required for deeper diagnosis.

Its Trace Viewer can expose the action timeline, DOM snapshots, screenshots, console output, network activity, source information, and other metadata associated with a test execution.

Instead of treating a failed test as:

PASS / FAIL

we can treat it as:

For CI, Playwright supports collecting traces on retry and screenshots when failures occur, providing useful debugging information without collecting the full set of diagnostic artifacts for every successful test.

A practical configuration is:

import { defineConfig } from "@playwright/test"; 

export default defineConfig({ 

retries: 1, 

use: { 

trace: "on-first-retry", 

screenshot: "only-on-failure", 

video: "retain-on-failure" 

} 

});

The result is a much richer failure package.

The AI-Assisted Failure Analysis Architecture

The proposed architecture separates execution from reasoning.

This architecture keeps Playwright's role deterministic.

AI does not decide whether a test passes or fails. Playwright makes that decision. AI helps explain why the failure may have happened.

Building a Structured Failure Package

Sending an entire CI workspace to an AI model would be inefficient and potentially unsafe. Instead, the pipeline can create a structured failure object:

interface FailureEvidence { 

testName: string; 

file: string; 

error: string; 

url?: string; 

browser?: string; 

duration?: number; 

consoleErrors: string[]; 

networkFailures: string[]; 

screenshotPath?: string; 

tracePath?: string; 

}

For example:

{ 

"testName": "checkout payment", 

"file": "checkout.spec.ts", 

"error": "Order confirmation not visible", 

"url": "/checkout", 

"browser": "chromium", 

"consoleErrors": [], 

"networkFailures": [ 

"POST /api/payment -> 500" 

] 

}

The AI now has structured context instead of receiving only:

Test failed.

Why?

What the AI Should Analyze

The AI layer can classify failures into a controlled set of categories:

Application
Test
Network
Test Data
Environment
Unknown

The analysis flow becomes:

A structured prompt can encourage a consistent output format:

Analyze the Playwright failure using only the supplied evidence. Classify the failure as:

APPLICATION

TEST

NETWORK

DATA

ENVIRONMENT

UNKNOWN

Return:

Classification
Probable root cause
Supporting evidence
Confidence
Recommended next investigation

Do not invent information.

Do not modify the test.

Do not modify application code.

The AI could then return:

{
"classification": "APPLICATION",
"rootCause": "Payment API returned HTTP 500",
"confidence": 0.91, "evidence": [ "POST /api/payment returned 500",
"Checkout page loaded successfully"
],
"nextStep": "Inspect payment service logs"

The distinction between evidence and AI interpretation is important:

Observed Evidence

≠

AI Hypothesis

Example: Diagnosing a Checkout Failure

Imagine that Playwright reports:

Timeout:

"Order confirmed" was not visible

A conventional debugging process might look like:

The AI-assisted workflow can consolidate this:

The engineer still validates the diagnosis, but the investigation starts with a more focused hypothesis.

Why AI Should Not Automatically Fix the Test

An obvious extension would be:

This can create a serious testing risk. Suppose the application introduces a genuine regression, but the AI incorrectly concludes that the locator is outdated. It changes the test. The test passes. The application defect remains.

A safer architecture is:

AI therefore becomes a diagnostic assistant rather than an autonomous test maintainer.

Handling Security and Sensitive Test Data

Failure artifacts can contain sensitive information:

Access tokens
Cookies
User information
API payloads
Authorization headers
Screenshots

Therefore, evidence should be sanitized before it reaches the AI service. A simple example:

function sanitize(value: string): string { 

return value 

.replace( 

/Bearer\s+\S+/g, 

"Bearer [REDACTED]" 

) 

.replace( 

/password=\S+/gi, 

"password=[REDACTED]" 

); 

}

The complete flow becomes:

For production implementations, sanitization should be treated as part of the architecture rather than an optional enhancement.

Measuring Whether AI Actually Helps

A key part of evaluating this system is experimentation.

Producing an AI-generated summary is straightforward. The more useful question is whether the system reduces engineering effort while maintaining acceptable diagnostic accuracy.

A controlled experiment could contain:

10 locator failures
10 API failures
10 network failures
10 data failures
10 environment failures

Then compare:

Metric Traditional AI-Assisted

Mean diagnosis time Measure Measure

Correct classification Measure Measure

Incorrect classification Measure Measure

Human intervention Measure Measure

Analysis latency Measure Measure

One particularly useful metric is Mean Time to Diagnose (MTTD). For example:

Traditional:

Failure → Investigation → Root Cause

18 min

AI-assisted:

Failure → AI Analysis → Validation → Root Cause

5 min 2 min

These numbers should come from the actual experiment rather than assumptions.

Failure Modes of AI-Assisted Diagnosis

AI does not eliminate debugging problems. It introduces another layer of possible failure.

The system may classify an application defect as a test defect.
It may generate a plausible explanation when insufficient evidence exists.
It may infer behavior that was never present in the application.
It may become overconfident when the available evidence is incomplete.

Therefore, the system should support an explicit unknown state:

A system that says “insufficient evidence” is often safer than one that confidently invents a root cause.

Production-Ready Pipeline

A production implementation can evolve into:

This architecture can also be extended with historical test results, defect-management systems, CI metadata, and application logs. As the evidence pipeline becomes more reliable, the AI analysis can provide more useful diagnoses.

Conclusion

Automated testing addresses a major part of software quality workflows by executing tests consistently at scale.

The next challenge is understanding failures efficiently.

An AI-assisted Playwright failure analysis pipeline addresses this by combining:

The key is not to position AI as a replacement for Playwright or QA engineers.

Playwright should remain responsible for deterministic test execution. AI should focus on interpreting evidence, correlating signals, classifying failures, and suggesting where engineers should investigate.

The real measure of success is therefore not:

“Can AI explain a failed test?”

It is:

“Can AI measurably reduce the time engineers spend moving from a test failure to a validated root cause without introducing unacceptable diagnostic errors?”

That is a problem that can be tested, measured, and improved. This makes AI-assisted failure analysis a practical engineering problem rather than another demonstration of generative AI.

From Test Failure to Faster Diagnosis

AI-assisted failure analysis is most useful when it strengthens the testing workflow without replacing deterministic execution or human judgment. By combining Playwright evidence with structured AI analysis and human validation, teams can investigate failures with clearer context and make informed decisions about the next step. For teams looking to extend this approach across automated testing and continuous validation, explore GeekyAnts’ Automation Testing Services.

Stop Automating Everything: A Balanced Quality Engineering Approach to Testing

GeekyAnts India Pvt Ltd — Mon, 12 Oct 2026 08:55:20 +0000

Balanced quality engineering places automation, API testing, exploratory work, and AI where each gives the most value, so teams ship faster without trading away user-perceived quality.

Key Takeaways

Automate by Risk: Focus automation on business-critical, high-risk workflows rather than raw test-case count, since more tests do not mean better quality.
Cheapest Effective Layer: Validate business logic through API tests where possible and reserve UI automation for the user journeys that matter most.
Keep Exploratory Testing: Human testers surface unexpected behaviour, usability gaps, and edge cases that predefined tests never anticipate.
Measure Quality, Not Volume: Track defect detection, leakage, flakiness, maintenance effort, and investigation time instead of counting automated tests.
AI as an Accelerator: Let AI assist with test generation, failure analysis, and maintenance while engineers stay responsible for quality decisions.

Why Passing QA Tests Can Still Ship Poor Experiences

Software teams are expected to release faster while maintaining reliable digital experiences. Automation has become a critical part of achieving this goal.

A typical automated regression workflow looks like:

This provides fast and repeatable feedback, but passing tests do not necessarily mean that users are experiencing a high-quality product.

For example, an automated checkout test may verify that an order is successfully created while missing:

confusing error messages,
poor mobile usability,
inaccessible controls,
unclear navigation,
unexpected user workflows.

This creates an important distinction:

Technical correctness and user-perceived quality are related, but they are not the same thing.

Moving From Test Automation to Quality Engineering

Traditional testing often follows:

A Quality Engineering approach distributes quality throughout the lifecycle:

The goal is not simply to find more bugs. It is to provide fast, reliable quality feedback at the appropriate layer.

What a Balanced Testing Model Looks Like

A balanced QE strategy can combine five capabilities:

Each layer has a different purpose.

Testing approach	Primary purpose
Unit testing	Validate isolated logic
API testing	Validate services and business rules
UI automation	Validate critical user journeys
Exploratory testing	Discover unexpected behavior
UX testing	Validate the user experience
AI assistance	Accelerate testing activities

Deciding What to Automate

Not every scenario should become a UI automation test. Strong automation candidates are generally:

repeatable,
deterministic,
frequently executed,
business-critical,
expensive to execute manually.

For example:

Login
Checkout
Payment
Order creation
Permission validation
API contracts

A useful decision process is:

The objective should not be 100% automation.

It should be meaningful risk coverage.

Validate Business Rules Through APIs, Not the UI

One common mistake is validating every business rule through the UI.

Suppose an application exposes:

POST /api/orders

The business logic can be validated directly:

const response = await request.post("/api/orders", {
  data: {
    productId: 123,
    quantity: 2
  }
});

expect(response.status()).toBe(201);

const body = await response.json();

expect(body.quantity).toBe(2);
expect(body.status).toBe("created");

The UI test can then focus on whether a user can successfully complete the workflow. This creates a useful separation:

This can reduce unnecessary UI automation and make regression suites faster and easier to maintain.

What Human Testers Catch That Automation Misses

Automation executes what engineers explicitly define.

Humans can investigate what was not anticipated.

For example, an exploratory checkout charter might include:

Explore checkout with focus on:

invalid inputs
browser back button
session expiry
duplicate submission
slow network
mobile viewport
unexpected navigation

A tester might discover:

Field	Observation
Action	Submit payment twice quickly.
Expected	One order.
Actual	Two orders created.
Risk	Duplicate order/payment.

A standard happy-path automation test might never discover this scenario. This is why exploratory testing should not disappear when automation increases.

Why a Green Suite Isn't Proof of a Good Experience

Consider:

await page.getByRole("button", {
  name: "Place Order"
}).click();

await expect(
  page.getByText("Order confirmed")
).toBeVisible();

The test proves that the order can be placed.

It does not necessarily prove that:

the button is easy to find,
the error messages are understandable,
the workflow is accessible,
the mobile experience is usable,
the customer understands the payment status.

This is where human-centered testing complements automation.

Metrics That Show Whether the Strategy Works

A balanced QE strategy should be measurable.

Instead of measuring only:

Number of automated tests

track metrics such as:

Execution time
Defects detected
Defect leakage
Flaky tests
Maintenance effort
Failure investigation time
Exploratory findings

A practical experiment can use a fixed set of application workflows:

Registration
Login
Product Search
Cart
Checkout
Payment
Order Confirmation
Invoice Download

Then evaluate them using:

UI Automation
API Testing
Exploratory Testing
UX-focused Testing

The experiment should record the same measures for each testing approach. Until real results are available, it is better to define the measurement plan rather than show placeholder values.

Metric	What to record
Execution time	Time required to complete the selected workflow or test set
Unique defects detected	Distinct defects found by each testing approach
Maintenance effort	Time spent updating or repairing tests as the application changes
Investigation time	Time required to understand and classify a failed test or observed issue
Defect leakage	Defects that escaped earlier test layers and were found later
Flaky-test rate	Intermittent automated failures that do not consistently reproduce
Exploratory findings	Unexpected behaviors, usability problems, or risk scenarios discovered through human investigation

Charts Worth Publishing Once You Have Real Data

Once the experiment has produced real measurements, two graphs can communicate the results clearly without making the article visually heavy:

Execution Time by Testing Approach — compare the measured execution time for UI automation, API testing, exploratory testing, and UX-focused testing.
Unique Defects Detected by Testing Approach — compare the number of distinct defects found by each approach, ideally with defect categories called out where useful.

Do not publish these charts with illustrative or fabricated values. Generate them only from the completed experiment and label the units, sample size, and test conditions so readers can interpret the results correctly.

These graphs should reinforce an important point:

Different testing approaches detect different categories of risk.

Keeping Automation Reliable Enough to Trust

Automation becomes less valuable when teams stop trusting test results.

A test that behaves like this:

Run 1 → PASS
Run 2 → PASS
Run 3 → FAIL
Run 4 → PASS
Run 5 → PASS

is potentially flaky.

Avoid solving this by adding arbitrary waits:

await page.waitForTimeout(5000);

Instead, synchronize with application state:

await expect(
  page.locator('[data-testid="order-confirmation"]')
).toBeVisible();

The principle is:

Wait for conditions, not time.

Automation should be treated as production-quality software, with code review, maintainability, monitoring, and refactoring.

Where AI Helps — and Where It Shouldn't Decide

AI can extend the QE workflow without replacing deterministic testing.

A practical architecture is:

For example, AI can analyze:

Playwright Trace
Screenshot
DOM
Console Logs
Network Errors
Test History

and suggest a failure category:

Test defect
Application defect
Environment issue
Data issue
Network issue

However, the engineer should validate the result.

A useful operating model is:

This keeps AI as an accelerator rather than an uncontrolled decision-maker.

Fitting the Approach Into Your CI/CD Pipeline

The balanced approach fits naturally into a modern CI/CD pipeline:

Different test layers provide feedback at different speeds, allowing teams to avoid using expensive UI tests for problems that can be detected earlier.

Trade-Offs Between Testing Approaches

There is no single testing technique that covers every quality dimension.

Approach	Strength	Limitation
UI Automation	End-to-end regression	Maintenance cost
API Testing	Fast business validation	Doesn't validate UX
Exploratory	Finds unexpected issues	Less repeatable
UX Testing	User experience	Requires human effort
AI Testing	Faster analysis/generation	Requires validation

The right strategy combines these capabilities according to product risk.

A Reference Architecture You Can Adapt

The complete model can be summarized as:

This model does not attempt to eliminate human testing.

Instead, it places automation, APIs, AI, and human reasoning where each provides the most value.

Conclusion: Getting the Balance Right

Modern Quality Engineering should not be a competition between manual testing and automation.

Automation provides speed, repeatability, and scalable regression coverage. API testing provides efficient validation of business behavior. Exploratory and human-centered testing provide contextual reasoning and help identify problems that deterministic automation may miss. AI can further accelerate these activities by assisting with generation, analysis, and maintenance.

The most effective Quality Engineering strategy is one that uses technology to accelerate testing without losing sight of the person ultimately using the software.

The goal is not maximum automation. The goal is maximum meaningful quality feedback for the engineering effort invested.

From Rolling Deployments to Zero-Downtime Releases

GeekyAnts India Pvt Ltd — Wed, 07 Oct 2026 09:17:22 +0000

Executive Summary
In digital banking, deployment availability is a business requirement as much as a technical one. A deployment that interrupts access to online banking can affect customers attempting to log in, view balances, initiate transfers, or perform other critical operations.

Our earlier rolling deployment model introduced problems around version coexistence. Every new frontend build generated build-specific resources such as runtime.1234.js and main.1234.js. During a rolling restart, old and new pods could coexist. A request for a resource from the new build could reach an older pod where that resource did not exist, producing a 404 response.

Static resources were cached at the CDN for about four minutes. We needed to consider application availability and version consistency across the application shell, static resources, CDN, and backend pods.

We moved to a Blue-Green deployment strategy. The new version is deployed into a separate environment, its pods start and pass readiness checks, and the environment is validated before customer traffic is switched. The existing environment continues serving production traffic until the cutover.

In our environment, the observed ALB traffic transition from Blue to Green was about 40–180 ms. The key benefit is not faster application startup. Pod creation, image pulling, startup, and readiness occur outside the customer-facing cutover window. The previous environment also remains available so that traffic can be routed back to it if a rollback is required.

A race condition remained during the ALB transition under high request volumes. A request for a build-specific static resource could reach the previous environment and receive a 404 response. We addressed this with a targeted NGINX retry for the affected static resources, with a delay of about one second before the retry.

Why We Needed a Different Deployment Strategy 1.1 The Rolling Deployment Model A rolling deployment replaces application instances in stages. This avoids replacing every instance at once, but it also means different application versions can coexist during the deployment.

1.2 Build-Specific Static Resources
Every application build generates resources associated with that build. For example:

Build 100:
runtime.100.js
main.100.js
styles.100.css

Build 101:
runtime.101.js
main.101.js
styles.101.css

The browser receives an application shell that references a particular build's resources. If a request for runtime.101.js reaches a Build 100 pod, the resource does not exist there, and the server can return a 404 response.

The pod itself could be healthy. The failure was caused by a version mismatch between the resource requested by the client and the application instance handling the request.

1.3 CDN Caching Added Another Timing Dimension
Static resources were cached at the CDN for about four minutes. The application origin could move to a new version while cached resources from the previous version remained available at the edge.

Deployment correctness had to account for three independent timing domains: application and pod readiness, ALB traffic convergence, and CDN cache lifetime.

Blue-Green Deployment Architecture Blue-Green deployment introduced two complete application environments. Blue is the active version, and Green is the new version.

The principle is simple: deploy and validate Green before allowing it to receive production traffic. This separates deployment preparation from production traffic cutover.

What Happens During a Blue-Green Deployment 3.1 Build and Image Creation The CI/CD pipeline creates an immutable image for the release. The frontend resources generated by that build are part of the same release artifact.

This preserves a clear relationship between the application version and the resources it serves.

3.2 Creating the Green Pods
The new release is deployed to Green without terminating the active Blue environment.

The Green pods go through image pull, container startup, application initialization, and readiness checks before they become eligible to receive traffic.

3.3 Pod Readiness
Readiness is a key mechanism in this architecture. Kubernetes distinguishes a running container from a container that is ready to accept traffic. A failed readiness probe causes the pod to be treated as not ready for normal service traffic. A startup probe can be used when an application needs more initialization time before liveness or readiness checks begin.

For a banking application, readiness should represent the minimum state in which the application can process production requests without creating service risk. Depending on the implementation, this can include application initialization, configuration loading, dependency availability, and other required startup conditions.

3.4 Green Validation
Once the Green pods are Ready, the new environment can be validated before customer traffic is moved.

Confirm that the pods and application are healthy.
Run API smoke tests against the new environment.
Confirm that the required static resources are available.
Check connectivity to critical dependencies.
Test business-critical flows.
Review application and infrastructure metrics.
This allows the team to validate Green while Blue continues serving production traffic.

How the ALB Traffic Switch Works The Application Load Balancer (ALB) provides the boundary for switching production traffic between Blue and Green. An ALB listener evaluates routing rules and forwards traffic to target groups. Target groups contain registered targets and can have their own health checks.

Before the cutover, Blue is the active target group and Green is prepared without receiving normal production traffic. Once Green has passed the deployment gates, the routing configuration changes so that production requests are directed to Green.

The Critical Difference: Deployment Time vs Customer-Impact Time A key distinction in Blue-Green deployment is the difference between deployment time and customer-impact time. The new pods still need to pull the image, start the container, initialize the application, and pass readiness checks.

What changes is where that work occurs.

Blue continues serving customers while Green is being prepared. This removes the time-consuming preparation work from the customer-facing cutover path.

The customer-facing transition is limited to the routing window rather than the full duration required to create and warm the new pods.

The improvement should not be described as “the deployment became X% faster.” The deployment preparation time still exists, but it does not need to translate into customer downtime.

Rollback: Turning Recovery Into a Traffic Switch Blue-Green changes the rollback mechanism.

Normal state:

BLUE = Build 100
GREEN = Build 101
Traffic = 100% Green
If monitoring identifies a production problem after the cutover, Blue can remain available as the previous known-good version.

Instead of rebuilding and redeploying the previous image, the deployment can route traffic back to the running Blue environment.

This changes rollback from a deployment operation into a traffic-routing operation, which can reduce mean time to recovery (MTTR).

The Production Challenge After Blue-Green 7.1 The ALB Transition Window Blue-Green solved the major version-coexistence problem, but production testing exposed a smaller race condition during the traffic switch.

In our environment, the observed ALB transition from Blue to Green was about 40–180 ms.

Before:
Blue = 100%
Green = 0%

Transition:
Blue = 100% -> 0%
Green = 0% -> 100%

After:
Blue = 0%
Green = 100%
Under high request volume, requests can arrive during this short interval. A request that arrives during the transition can encounter the previous routing state.

7.2 Static Resource 404 During the Transition

This was not a persistent application failure. It was a transient race condition caused by the interaction between build-specific resource names and traffic convergence.

NGINX Retry as the Last-Mile Resilience Layer To address this transient condition, a targeted retry mechanism was introduced at NGINX for the affected static resources.

The one-second delay was longer than the observed ALB transition window. This gave the traffic transition time to complete before the retry.

This mechanism was scoped to static-resource requests. A generic retry policy should not be applied to state-changing banking operations such as fund transfers without idempotency and transaction-safety guarantees.

CDN and Resource Versioning Build-specific resource names remain part of the design. Instead of overwriting a stable path such as /main.js, each build produces a unique resource path.

/main.100.js
/main.101.js
/main.102.js
This allows multiple versions of a resource to coexist in the CDN. The application shell should reference the resource set belonging to its build.

The CDN cache lifetime of about four minutes is treated as a separate concern from the ALB transition. Build-specific resources provide coexistence, while the NGINX retry addresses the short traffic-convergence race.

Zero-Downtime as an End-to-End Property The implementation showed that zero-downtime (ZDT) is not a property of Kubernetes or the load balancer alone.

A system can have healthy pods and still return customer-visible errors if resource versioning, caching, or routing transitions are not handled correctly.

Our final approach combines several mechanisms:

Blue-Green deployment
+
Pod readiness
+
Build-specific static resources
+
CDN caching strategy
+
ALB traffic cutover
+
NGINX static-resource retry
+
Fast rollback
=
Zero-Downtime Release Architecture

Business and Operational Impact Before the move toward ZDT, deployments were scheduled during low-traffic periods such as night-time maintenance windows. This was a risk-management decision. If a deployment could cause service interruption, a period with fewer customers using the platform reduced the number of users exposed to that interruption.

Before:
Release Ready
|
v
Wait for Night
|
v
Maintenance Window
|
v
Deployment

Blue-Green changed the release model.

After:
Release Ready
|
v
Deploy Green
|
v
Validate
|
v
Switch Traffic

Because Blue remains available while Green is prepared, a new deployment does not require the application to be taken offline. Subject to change-management, monitoring, and business controls, teams can perform deployments during normal operating hours, including the morning, instead of restricting them to night-time maintenance windows to avoid deployment downtime.

This changes the operational model: application availability no longer dictates the deployment window.

Deployment Sequence
Developer merges release |
CI/CD builds application |
Build-specific static resources generated |
Immutable container image created |
Image pushed to registry |
Green deployment created |
Green pods start |
Startup / initialization |
Readiness probes pass |
Green validation / smoke tests |
Green target group confirmed healthy |
ALB routing switched |
Traffic moves Blue -> Green |
Monitor application + business metrics |
Keep Blue available for rollback |
Decommission Blue after confidence window
Key Engineering Lessons
13.1 Blue-Green Requires Coordinated Deployment Controls
Blue-Green depends on the interaction between deployment orchestration, readiness, traffic management, resource versioning, CDN behavior, observability, and rollback. Two application environments alone do not address the dependencies across the release path.

13.2 Move Preparation Work Outside the Cutover Path
Image pulling, pod creation, startup, and readiness can take minutes. Blue-Green allows the previous version to continue serving traffic while the new environment completes this work.

13.3 Rollback Should Be Designed Up Front
Keeping the previous environment available allows the system to route traffic back to a known-good version without rebuilding and redeploying that version.

13.4 Short Transition Windows Can Expose Race Conditions
A short routing transition can still expose a race condition when request volume is high. Deployment design must account for requests that arrive while traffic moves between environments.

13.5 Static Resources Are Part of the Release Architecture
Frontend resources form part of the production deployment architecture. Their versioning, CDN lifetime, and relationship with the application shell must be considered as part of the release design.

Final Architecture

Conclusion The move from rolling deployments to Blue-Green deployment addressed the version mismatch that could occur when build-specific frontend resources reached pods running an older build. Blue-Green separates release preparation from traffic cutover by allowing the new environment to complete startup, readiness checks, and validation while the existing environment continues serving customers.

The implementation showed that zero-downtime deployment depends on controls across the release path. Resource versioning, CDN caching, traffic switching, readiness checks, retry behavior, and rollback design must work together to protect customer requests during a release.

Keeping the previous environment available also allows recovery through a traffic switch instead of another deployment. This release model reduces the need to schedule deployments around maintenance windows while preserving a path back to the previous version.

Building Release Architectures for Production Reliability
Zero-downtime deployment depends on more than switching traffic between two environments. Pod readiness, static-resource versioning, caching, traffic routing, monitoring, and rollback must work together to protect customer-facing applications during a release. For teams building or modernizing applications with these production requirements, GeekyAnts provides custom software development services covering architecture, CI/CD, deployment, and post-launch support.

References
Kubernetes: Liveness, Readiness, and Startup Probes
https://kubernetes.io/docs/concepts/workloads/pods/probes/
AWS Elastic Load Balancing: Application Load Balancer
https://docs.aws.amazon.com/elasticloadbalancing/latest/application/introduction.html
AWS Elastic Load Balancing: Target Groups
https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-target-groups.html
AWS Elastic Load Balancing: Listener Rules
https://docs.aws.amazon.com/en_en/elasticloadbalancing/latest/application/listener-rules.html

The Deprecation Notice Nobody Reads

GeekyAnts India Pvt Ltd — Wed, 07 Oct 2026 08:49:51 +0000

This blog explains why API deprecation notices often fail and how staged retirement, telemetry, brownouts, and scheduled enforcement can make API migrations more controlled and predictable.

Every API team eventually writes the same message

"This endpoint will be removed on [date]. Please migrate before then." It goes in a changelog, a blog post, maybe an email. Traffic to the deprecated endpoint can show little movement until the day it is switched off, at which point support queues fill up with people who swear they never saw the notice.

They may not have seen it. A warning with no consequence attached is easy for a busy team to defer. The problem with sunset timelines is not communication. Most contain nothing that creates a cost for delay until the exact day they stop being a warning and become an outage.

The flat line before the cliff

The pattern shows up across API providers: deprecation is announced, documentation is updated, and usage of the old endpoint barely dips. One cited account of a typical deprecation window reported that six months after announcing a sunset date, the old version was still carrying 40% of total traffic, with a team left asking why nobody had migrated.

A deprecated endpoint that still returns 200 OK carries no urgency. Migration work competes with every other roadmap item, and "the thing that still technically works" reliably loses that competition until it stops working at all. A blog post is not an effective notice if ignoring it carries no immediate consequence.

Two timelines, two outcomes

Timeline	Without a strategy	With a strategy
Month 0	Endpoint removed, no warning	Deprecation announced; Deprecation/Sunset headers added; migration guide published
Month 3	47 integrations already broken	Migration reminders sent to consumers still calling the endpoint
Month 6	Support queue flooded	First announced brownout window, a scheduled, published maintenance-style window during which the endpoint is intentionally degraded
Month 9	Emergency rollback underway	Quota tightened for remaining consumers; brownout windows lengthened per the published schedule
Month 12	Trust damaged, repeat incidents	Endpoint retired, returns 410 Gone; near-zero breakage

What changes structurally is that, by month 6, the endpoint behaves differently for consumers who have not migrated. That change is published on a schedule before it takes effect rather than discovered by consumers in production.

Why longer notice periods don't fix this alone

Some major API providers use deprecation or version-support windows measured in years, giving customers time to include migration work in planning cycles.

But two years solves a planning problem, not an urgent problem. It does nothing to make migration feel pressing in month three, nine, or eighteen because nothing about the API has changed yet. The calendar moved; the consequence didn't.

Staged, announced retirement

The fix isn't a louder notice, and it isn't quiet degradation either. It's a retirement plan the API enforces on a schedule the consumer knows about. Every stage is disclosed in the changelog and response headers before it takes effect rather than discovered in production.

A staged retirement can follow five stages:

Headers-only window. Deprecation and Sunset headers present on every response. No behavioral change. This window should be long enough to cover at least one full planning cycle for your consumers.
Warnings surfaced in-band. A deprecation notice is added to response bodies, or a Link header points to the migration guide, so tooling rather than memory catches it.
Announced brownout windows. Published, scheduled windows (e.g., "the deprecated endpoint will return elevated latency and occasional 429s every Tuesday 2–4pm UTC starting date") during which the endpoint behaves like it will after full retirement. These are calendared and documented before they start, exactly like a maintenance window.
Progressively tighter quotas for remaining consumers, identified by telemetry, not by chance.
Hard retirement. 410 Gone, with the migration link still present.

At every stage, a consumer hitting the new behavior can look at a public schedule and confirm exactly which stage is in effect and when the next one begins. Nothing about it is a surprise, and nothing about it is silent.

What changes for internal versus public APIs

For an internal API with a handful of known consuming teams, the friction curve can be steep and communication can be direct. A Slack message plus a dashboard showing which service is still calling the old path can help get a fix shipped within a sprint. The consuming teams are known, and a sharper cutoff is more practical because the organization also controls the fallback.

Public and partner-facing APIs are the harder case. The consumer may be unknown until they show up in usage telemetry, may not follow the changelog, and may be running an integration nobody at their organization has touched since it was built. That is where headers, published schedules, and graduated enforcement provide value. They turn an anonymous, unreachable consumer's client code into something that reports its own deprecation status back to whoever looks at the logs.

Comparing three retirement strategies

	Communication-only	Hard sunset	Staged enforcement (brownouts)
What it is	Changelog/email notice, no behavior change until removal day	Fixed date, endpoint simply stops working	Published schedule of increasing, announced friction before final removal
Best fit	Low-risk internal APIs, few known consumers	Emergency/security-driven deprecations	Public and partner APIs with unknown or numerous consumers
Retry-storm risk	Low until cutover, spikes hard at deadline	High; everyone fails at once	Distributed across scheduled windows; retry behavior can be observed before it's catastrophic
SLA / contractual exposure	Rarely satisfies partner deprecation clauses alone	Highest exposure if the date arrives early	Lower; each stage documented and pausable
Rollback strategy	Trivial (nothing changed yet)	Difficult; re-enabling under load is its own incident	Straightforward; each stage is a feature flag
Operational cost	Low upfront, high at cutover	Low upfront, very high at cutover	Higher upfront, spread more evenly
When it fails	Consumers defer migration	Consumers who missed the notice get zero warning	Poorly tuned severity or cadence can trigger retries without Retry-After and backoff

No single strategy is correct in isolation. The choice depends on how well you know your consumers, what your contracts require, and how much operational machinery you're willing to build before you start the clock.

Standards-correct signals

RFC 9745 defines Deprecation as a Structured Fields Date, not an HTTP-date string. That means the value should look like @1735689600 (an @-prefixed Unix timestamp), not the output of toUTCString(). Sunset (RFC 8594) does use an HTTP-date. Mixing the two formats can break client tooling. A malformed value may fail to parse as expected, which can disrupt automated deprecation tracking.

Production-oriented middleware, with per-consumer rollout, a rollback switch, and telemetry hooks built in:

// Formats a JS Date/timestamp as an RFC 9745 Structured Field Date: "@" function toStructuredFieldDate(msTimestamp) { return @${Math.floor(msTimestamp / 1000)}; }

function deprecationMiddleware({ deprecationDate, sunsetDate, migrationLink, stages, // ordered array of { startsAt, mode, config } isEnforced, // (consumerId) => bool — per-consumer rollout / allowlist telemetry, // { recordHit(stage, consumerId), recordOutcome(stage, consumerId, result) } killSwitch, // () => bool — global rollback }) { return function (req, res, next) { const now = Date.now(); const consumerId = req.consumerId; 

res.setHeader('Deprecation', toStructuredFieldDate(deprecationDate));
res.setHeader('Sunset', new Date(sunsetDate).toUTCString());
res.setHeader('Link', `<${migrationLink}>; rel="deprecation"`);

telemetry.recordHit('headers', consumerId);

if (killSwitch()) {
  return next(); // incident-response rollback: headers stay, no enforcement
}

const activeStage = stages
  .filter((s) => now >= s.startsAt)
  .sort((a, b) => b.startsAt - a.startsAt)[0];

if (!activeStage || !isEnforced(consumerId)) {
  return next();
}

switch (activeStage.mode) {
  case 'retired': {
    telemetry.recordOutcome('retired', consumerId, '410');
    return res.status(410).json({
      error: 'gone',
      message: 'This endpoint was retired. See the Link header for the migration guide.',
    });
  }
  case 'brownout': {
    const { rejectRatio, retryAfterSeconds } = activeStage.config;
    if (Math.random() < rejectRatio) {
      telemetry.recordOutcome('brownout', consumerId, '429');
      res.setHeader('Retry-After', String(retryAfterSeconds));
      return res.status(429).json({
        error: 'deprecated_brownout',
        message: `Scheduled brownout in effect per the published retirement schedule. See ${migrationLink}.`,
      });
    }
    telemetry.recordOutcome('brownout', consumerId, 'pass');
    return next();
  }
  case 'quota': {
    req.deprecationQuotaTag = activeStage.config.quotaKey;
    return next();
  }
  default:
    return next();
   }
 }; 
}

Stages here are explicit, scheduled, published data rather than a continuous random function tied to elapsed time. Enforcement is per-consumer and gated by an allowlist so a rollout can ramp in stages instead of switching on for every consumer at the same time. A kill switch supports incident response, and every enforcement decision is recorded to telemetry so it can feed the decision loop below.

Telemetry as the backbone

"Track usage per consumer" isn't a strategy on its own. The strategy is the decision loop that telemetry feeds:

Identify every consumer still hitting the deprecated path, from API keys/auth identity, not just aggregate request counts.
Segment by risk and traffic. A partner integration processing payments and a dormant internal script hitting the endpoint twice a day need different handling and different owners.
Notify owners, not just via the changelog. The segmented list makes targeted outreach possible.
Measure migration after each stage, before deciding to advance. The gate for moving to the next stage should be a measured drop in remaining traffic plus confirmed outreach, not just a date on a calendar.
Enforce the next stage only for consumers who haven't moved. Scope enforcement to the remaining, already-notified cohort rather than applying it to everyone the moment a date arrives.

This loop is also what tells you when to pause: if a critical, contractually-protected partner is still on the old path with an active migration in progress, the gate holds them back from the next enforcement stage while the rest of the cohort proceeds.

Testing the strategy before shipping it

Before rolling out a staged retirement for a public or partner API, test the full sequence of headers, warnings, one brownout window, quota tightening, and retirement against a low-traffic internal endpoint with a small, known set of consumers. Compress the sequence into a shorter window than a real public deprecation would use.

What to Capture at Each Stage

Request volume to the deprecated path, before and after each stage transition
Number of distinct consumers still active, and how that count changes after outreach vs. after the first brownout
Retry behavior during the brownout window, including whether clients back off per Retry-After or retry at a rate that begins to resemble an attack
Support/Slack messages generated per stage
Time from "notified" to "migrated" per consumer, segmented by whether they moved after notice alone or only after a brownout

The test report should show the share of consumers who migrated after headers and direct outreach, the share who required a brownout window, the number of integrations requiring one-on-one follow-up, and any unexpected brownout behavior.

What actually breaks

Staged retirements do not fail cleanly, so teams need to plan for the ways they can fail.

A brownout window can trigger a retry loop that amplifies load instead of shedding it if clients do not respect Retry-After. That is why the header has to be present on every throttled response, not just the first.

A batch job can fail without drawing attention if it does not check for 429 at all. The problem may surface when someone notices missing data downstream.

A dormant integration with no listed owner can remain invisible until an enforcement stage breaks it. That is why the consumer registry needs an ownership field checked before enforcement begins, not after.

None of these are arguments against staged enforcement. They are arguments for building the telemetry and allowlist machinery described above before turning any stage on. If something behaves in an unexpected way, it should show up in a dashboard instead of a support ticket.

Why this matters now

API environments now include internal microservice contracts, partner integrations, and a growing layer of AI agents and tools that can call APIs without a person reading a changelog. A deprecation strategy that depends on a person noticing an email doesn't scale to that world. One built into the protocol through headers, a published schedule, and telemetry-gated enforcement does.

Sunset timelines don't fail because nobody announced them. They fail because, for most of the window, nothing about using the deprecated path changed. The fix isn't writing a better warning, and it isn't making things worse without notice until the deadline. It's a retirement plan the API keeps, announced one stage at a time. By the retirement date, consumers should know it is coming because they have seen it on the schedule, not because they experienced the change without warning.

Build APIs for Change, Not Just Launch

API reliability extends beyond keeping an endpoint available. Teams also need a plan for how APIs evolve, how consumers migrate, and how older versions are retired without creating avoidable production failures. Staged retirement, telemetry, and controlled enforcement make deprecation part of the API lifecycle rather than a last-minute cutoff. Explore how GeekyAnts approaches scalable systems and API architecture through its Backend Engineering capabilities.

From Manual Testing to AI-Assisted Automation with Playwright Agents

GeekyAnts India Pvt Ltd — Wed, 07 Oct 2026 06:01:38 +0000

This blog discusses the value of Playwright Agents in automating workflows. It provides a detailed description of setting up the system, as well as a breakdown of the Playwright Agent’s automation process.

For years, automation engineers have followed a familiar rhythm. Requirements come in, test cases are written, scripts are automated, locators break, scripts fail, and debugging begins. Fix, re-run, repeat.

This cycle hasn’t changed much, even though frameworks have evolved from Selenium to Cypress to modern tools like Playwright.

What if your automation framework didn’t just execute tests — what if it planned them, wrote them, ran them, and even fixed them when they broke? That’s exactly what Playwright Test Agents bring to the table.

Playwright introduced these AI-powered agents in version 1.56 to automate key parts of the testing lifecycle — planning, generating, and healing tests — using an agentic loop that interacts with your live application.

In this blog, we’ll explore what Playwright Agents are, how to set them up, how to use seed tests and prompts, what files they generate (like in your screenshot), and how each agent works in the development lifecycle. We’ll close with practical tips on prompt design and real differences versus generic AI coding tools like Cursor or ChatGPT.

The Evolution of Automation with Playwright

Playwright became popular by addressing common automation challenges like flaky tests and synchronization issues. With features like automatic waiting, semantic locators such as getByRole, and built-in tracing, it reduced the effort required to stabilize tests. This allowed QA engineers to focus more on test coverage rather than debugging framework issues. However, even with these improvements, designing and maintaining test scripts still remained a manual effort.

We still had to:

Translate requirements into scenarios
Convert scenarios into code
Refactor when UI changes
Fix broken locators

Playwright Agents aim to assist in exactly those areas.

Introducing Playwright Test Agents

Playwright Test Agents are AI-assisted automation workflows embedded directly into your Playwright project, designed to help you:

Explore an application and produce a test plan
Transform that plan into executable Playwright test code
Run tests and automatically repair failures

There are three core agents:

The Planner Agent takes natural language input and converts it into structured test scenarios. It uses the seed test as context to explore the application, understand user flows, and identify possible edge cases. The output is a detailed test plan with steps and expected outcomes, similar to how a QA engineer would design test cases.

The Generator Agent takes these structured scenarios and converts them into executable Playwright scripts. While generating code, it interacts with the live application to validate selectors, identify stable locators, and ensure assertions reflect actual UI behavior. It can also follow architectural patterns like Page Object Model, producing manageable and scalable test code.

The Healer Agent focuses on maintaining test stability. When a test fails, it replays the scenario, inspects the DOM, and identifies what caused the failure. It then attempts to fix the issue by updating selectors, adjusting waits, or modifying interaction logic, reducing the manual effort required for test maintenance.

These agents can be invoked independently or chained together in a complete “agentic loop”:

Planner → Generator → Healer

This turns a natural language description of test requirements into a stable test suite with minimal manual coding.

Project Setup — Step by Step (Beginner Friendly)

If you already know basic Playwright automation, this should feel like an extension of that knowledge. If not, stick with it. By the end, you will understand how these agents help even if you’re new to automation.

1. Create a Playwright Project

Start by creating a new directory and initializing a Node project:

mkdir playwright-agent-demo
cd playwright-agent-demo
npm init -y

npm install -D @playwright/test
npx playwright install

Now install Playwright:

You now have a basic Playwright project.

2. Initialize Playwright Agents

To add agent definitions to your project, run:

This command generates agent files in your project — which you’ll recognize in the screenshot below:

A folder named .github/agents contains:

playwright-test-planner.agent.md
playwright-test-generator.agent.md
playwright-test-healer.agent.md

These are the agent definitions that your AI tool (like Claude Code, VS Code Copilot, or OpenCode) uses to understand how to plan, generate, and heal tests.

Under the root folder, you also see:

specs/ – for Markdown plans
tests/ – for generated Playwright test files
seed.spec.ts – a seed test that bootstraps the environment

This exact file structure is aligned with Playwright’s agent conventions: .github/agents, specs/, and tests/

A seed test is essential because it provides a starting context that the planner uses to understand where to begin exploration, including any setup required (like logging in or navigating to a landing page).

Create tests/seed.spec.ts:

Next, add dummy seed data to a JSON file (optional but recommended):

testdata/seed.json

The seed test and seed data help the Planner understand context and scenarios, which makes its output far more relevant and accurate.

How the Planner Agent Works

The Planner Agent is like a QA analyst powered by AI. Rather than immediately writing code, it first produces a structured Markdown test plan that describes required test scenarios, user flows, steps, expected outcomes, and test data.

You can review this file before moving to code generation.

Generator Agent: Turning Plans into Code

Once you have a test plan, it’s time to generate actual automation scripts.

Switch your AI assistant to Generator mode and provide a prompt such as:

The Generator reads the Markdown plan, actively interacts with the browser to verify selectors and assertions, and produces test scripts under the tests/ directory. Similar to this:

For example, provide a prompt like:

“Create a test plan for login functionality with valid and invalid user scenarios using the seed test context.”

The Planner will explore your live application (through the seed test) and generate a Markdown file under s*pecs/* such as:

specs/login-plan.md

This file contains detailed, human-readable test plans, not code, but instructions for how you want the generator to build tests.

This step mirrors the typical QA process of writing test case documentation, except that the agent generates it automatically.

Each test should mirror a scenario from the plan.

Because the Generator interacts directly with the live app and evaluates selectors as it writes code, the tests it generates are often more stable and accurate than typical prompt-only AI output.

Healer Agent: Using AI to Fix Failing Tests

Inevitably, tests fail. It might be due to UI changes, such as an updated locator or changed button label.

Traditionally, you would open your editor, inspect the DOM, update selectors, and re-run tests. With Playwright’s Healer Agent, this can be assisted by AI.

Invoke the healer with a prompt like:

“Run and fix the failing test tests/amazon-search-add-to-cart-edge.spec.spec.ts.”

The healer will:

Replay the failing test in debug mode
Inspect the DOM to find equivalent elements or flows
Propose updates to locators or waits
Re-run until the test passes, or decide that the test really reflects a broken feature.

This reduces repeated manual debugging cycles, especially for tests that only break due to minor UI refactors.

From Prompt to Execution: Inside the Agent Workflow

While using Playwright Test Agents feels simple from a user perspective, there is significant processing happening in the background.

The agents operate through an agentic loop where they can read project files, execute tests, interact with the browser, and inspect the live DOM. For example, the Planner uses the seed test to explore the application and understand flows, the Generator validates selectors in real time while generating scripts, and the Healer replays failing tests to identify and fix issues.

In the foreground, this complexity is abstracted into simple inputs and outputs. Users provide prompts and receive structured test plans, executable test scripts, or suggested fixes without directly interacting with the underlying processes. This separation is what makes Playwright Agents both powerful and easy to use.

Structuring Tests with Page Object Model

One of the biggest benefits of designer prompts is instructing the generator to produce maintainable code, and that starts with architecture.

If you prompt:

“Use Page Object Model and store locators in separate page files.”

The generator will output something like:

pages/login.page.ts

tests/login/login.spec.ts

Where:

login.page.ts contains locator definitions and reusable page actions
login.spec.ts uses the page object and seed data for test logic

This results in a clean, maintainable automation framework that scales well.

The Agentic Loop: From Plan to Stable Tests

When you use all three agents together, you get:

Seed Test + Prompt

↓

Planner → Create Markdown Plan

↓

Generator → Create Tests

↓

Healer → Fix Failures

↓

Stable Automation Suite

This mirrors a full human automation lifecycle, except now it is assisted by AI and deeply integrated with Playwright’s tooling.

How Playwright Agents Differ from Generic AI Tools

Feature	Regular AI Code Generation	Playwright Agents
Generates code based on prompts	Yes	Yes
Runs tests	No	Yes
Fixes failing tests autonomously	No	Yes
Understands live DOM while generating code	No	Yes
Integrated into the Playwright ecosystem	No	Yes

It’s easy to confuse Playwright Agents with other AI coding tools, such as:

Cursor AI
ChatGPT code generation
Generic AI assistants

But there is a fundamental difference:

Playwright Agents integrate with MCP (Model Context Protocol) and interact with your application and tests as part of the lifecycle. This makes them far more context-aware and useful than simple prompt-to-code generation.

Best Practices for Using Playwright Test Agents

Here are some practical tips based on real usage trends:

Provide Good Context

Always include a clear seed test
Use structured seed data
Reference environment details

Write Clear Prompts

Make sure your prompts include architecture preferences, test data references, and expected outputs.

Review Generated Tests

AI can generate great boilerplate, but human review is still important.

Integrate into CI Carefully

Treat healed and generated tests as drafts until fully reviewed.

Conclusion

Playwright Agents, Planner, Generator, and Healer bring AI directly into the automation lifecycle. They:

Plan test scenarios from natural language
Generate well-structured automation code
Check and repair failing tests
Help QA teams move faster with less manual overhead

For any QA engineer with basic Playwright knowledge, these agents unlock productivity leaps, from planning without code to generating and healing tests with AI.

If you want to experiment with this in your own project, run the agent setup, build a seed test, and start with simple prompts. You will be amazed at how much of the automation lifecycle can now be AI-assisted.

From Prompting to Process: What Changed When Flutter Shipped Agent Skills

GeekyAnts India Pvt Ltd — Mon, 28 Sep 2026 11:44:01 +0000

Google is starting to ship Flutter's engineering workflows as machine-readable guidance for AI agents. It may look like another AI feature, but it hints at a much bigger shift in how teams build with Flutter.

The Consistency Problem

Over the past few years, AI coding tools like Cursor, Claude Code, Copilot, and OpenCode have become part of many developers' daily workflows. They produce code, explain unfamiliar APIs, write tests, and navigate large codebases with impressive accuracy.

But if you have been using them on a reasonably large project, you have probably noticed something.

They are not very consistent.

Ask an AI to implement the same feature in two different sessions, and there's a good chance you will get two different approaches. Switch models, and the implementation changes again. Sometimes it follows the latest framework recommendations. Other times, it relies on outdated patterns.

Most teams respond the same way: write better prompts, add repository rules, or create a custom skill.md files to steer the agent toward the right decisions.

We took the same approach by documenting our architecture, coding conventions, review expectations, and engineering practices as custom Skills. They made the agent noticeably more consistent.

So when Flutter announced official Agent Skills, my first reaction was not: "How do we use them?" It was: "If we already have our own Skills, what problem are Flutter's official Skills actually solving?"

That question turned into a couple of experiments.

Three Layers of AI-Assisted Development

Looking back, Flutter's recent AI investments weren't isolated features. They were building on each other.

First came Rules, giving teams a way to define project-specific conventions and preferences.

Then came Model Context Protocol (MCP), allowing AI agents to inspect, debug, and interact with running Flutter applications instead of reasoning purely from static code.

And then came Agent Skills.

If Rules tell an agent how your team works, and MCP tells it what's happening inside your application, Agent Skills answer a different question: how does Flutter itself recommend solving this problem?

That is the important shift.

Flutter is now versioning its engineering workflows alongside the framework itself. Instead of relying entirely on what an AI model happened to learn during training, agents can follow workflows maintained by the Flutter team.

Today, those workflows cover areas like:

Localization
Responsive layouts
Routing
Widget and unit testing
Static analysis
JSON serialization
Platform integration
Architecture best practices

In other words, Flutter is shipping its engineering knowledge as structured workflows.

That naturally led to the next question: does this actually change how an AI agent behaves?

I ran two experiments to find out.

First Experiment: Declarative Routing

It is one of those features where there is not just one thing to do. Depending on the prompt, an AI could jump straight into writing routes, miss platform-specific configuration, skip deep linking altogether, or recommend an approach based on what it learned during training rather than Flutter's latest guidance.

So I kept the prompt intentionally simple:

"Set up declarative routing for this Flutter application."

I wanted to see how the agent would approach the problem before I told it how to solve it.

The first thing it did caught my attention.

Before generating an implementation plan, it explicitly selected the flutter-setup-declarative-routing skill.

Show Image

From that point on, it was not about figuring out a solution; it was following Flutter's own workflow.

That was the interesting part.

Without Agent Skills, the implementation depends on the model's reasoning and whatever Flutter knowledge it has internalized. With Agent Skills, the framework itself becomes the starting point.

What happens when Flutter's Skills and our own Skills are both available?

I tested this with a login screen.

It needed:

Flutter's localization workflow
Our project conventions for authentication, repositories, dependency injection, and state management

I kept the prompt simple and let the agent decide how to approach it -

"Implement a login screen for this application. Follow the existing project architecture and add localization for all user-facing strings.

It picked the right source every time.

For localization, it used Flutter's official Skill. For everything related to our application architecture, it followed our custom Skills. I did not have to tell it which one to use or write a carefully engineered prompt.

The two sets of Skills worked together naturally. Flutter handled the framework guidance, while our repository continued to define how our application was built.

Show Image

Why This Matters: A Floor, Not a Finish Line

So, what actually changed?

Flutter is taking ownership of its engineering expertise.

Engineering teams no longer need to teach AI how Flutter expects routing, localization, responsive layouts, or testing to be implemented. Flutter now ships that knowledge itself.

That removes a significant amount of duplication. Instead of every team maintaining its own version of Flutter best practices inside prompts or custom Skills, the official workflows become the source of truth.

Even better, those workflows evolve alongside Flutter. As recommendations change, the official Skills change too, and every compatible AI agent benefits automatically.

Of course, that's only half the story.

Flutter's Agent Skills provide a strong foundation, but they do not replace your organization's architecture, coding standards, or business-specific workflows. Those remain your responsibility, and that is exactly how it should be.

Final Thoughts

Looking back, Flutter's recent AI features fit together surprisingly well.

Rules capture how your team works. MCP gives agents runtime context. Agent Skills teach them how Flutter itself expects problems to be solved. Custom Skills layer on everything unique to your organization.

Together, they reduce the amount of engineering knowledge an AI has to infer. That is the real significance of Flutter's recent AI investments. Engineering knowledge is becoming explicit, versioned, reusable, and maintained by the people best positioned to own it.

Today's skill library covers foundational workflows, but it already hints at what's possible. Imagine Skills for performance profiling, DevTools workflows, accessibility audits, plugin development, migrations, or advanced rendering patterns. Every new skill moves another piece of framework knowledge out of documentation and into a reusable workflow.

The answer to how much of Flutter an agent should really guess keeps shrinking with every release.

Feature Flags as Technical Debt: The Cleanup Nobody Schedules

GeekyAnts India Pvt Ltd — Mon, 28 Sep 2026 08:46:47 +0000

Feature flags are one of the cheapest tools in engineering to adopt and one of the most expensive to leave unmanaged. Teams use them for gradual rollouts, A/B tests, kill switches, and gating unfinished work, but few teams have a matching process for removing them once they have served their purpose. As a result, flags that were meant to be temporary become permanent, adding unnecessary complexity to the codebase.

This article explains why flag cleanup gets skipped and why it creates a real engineering cost. It also walks through a staleness-detection implementation, a flag-removal exercise, and a practical checklist for managing flags across their full lifecycle, from creation to removal.

Why Flags Are Easy to Add and Hard to Remove

Adding a flag is a small, fast PR. Wrapping a block of code in a conditional and wiring it to a config value takes minutes, and it ships in the same PR as the feature it gates — there's no separate approval step, no extra review, no reason for anyone to push back. Removing a flag involves a different kind of work. You have to find every place it is checked, including references in logging, analytics, and monitoring code, confirm which branch is now permanently "on" or "off," delete the dead branch, update or delete the tests that covered it, and verify that nothing downstream depends on the old behavior. This multi-step effort competes with new feature work for the same sprint capacity, so cleanup often gets pushed to a later sprint until the flag becomes a permanent part of the codebase.

There's also a confidence problem. Once a flag has been live for months, the person who added it may have moved teams, changed roles, or forgotten why it was added. Removing the conditional can feel risky when its dependencies are unclear, so teams may leave it in place. That decision can turn a two-week release flag into a permanent part of the codebase.

Underneath both of these is a structural gap: a flag rarely comes with a ticket, an expiry date, or a named owner responsible for its removal. By default, creating a flag does not create a corresponding task to revisit it later. Without documented future work, the flag can remain in place. Meanwhile, teams add new flags every sprint while removing few of the old ones, so the backlog grows and cleanup becomes more complex as additional flags enter the same code paths.

The Flag Lifecycle

The diagram below lays out the full path a flag should take, from creation to retirement. The critical fork sits in the middle of the diagram: once a flag reaches "stable at 100%," it either gets caught by an automated staleness check and routed to a removal PR, or it gets ignored and drifts into permanent technical debt. Most flags fail at exactly this point — not because removal is technically difficult, but because nothing in the default engineering workflow forces the question to be asked. Without an automated trigger, a flag can sit at "stable" for years with nobody ever explicitly deciding to leave it that way; it simply never comes up.

Figure 1: The feature flag lifecycle, from creation through rollout to either scheduled removal or stale limbo.

Why This Is a Real Cost, Not Just Clutter

Each independent binary flag can double the number of possible runtime states. A function gated by three independent flags can have up to eight possible states, but teams may have designed or tested only a subset of those combinations. The remaining states can introduce interactions that were not considered during development or testing.

Stale flags can create bugs through untested interactions. Two flags that are each considered "basically always on" can still interact in a combination the team did not anticipate. If the team assumes that only one meaningful state is live, that interaction may not be covered during testing or code review and can surface only under production traffic.

New engineers may avoid code affected by unfamiliar flags. When it is unclear which flags are critical and which can be removed, team members may work around that code rather than risk breaking an unknown dependency. This can slow down simple changes and introduce workarounds that add further complexity.

The number of dependencies can grow the longer a flag remains in place. Flag checks can extend into related systems, including analytics events tied to flag state, log lines that reference the flag, monitoring dashboards built around it, and configurations in other services. As these dependencies accumulate, removing the flag can require changes across several parts of the system.

Not All Flags Are the Same

A major reason cleanup gets mishandled is that teams treat every flag the same, even though the appropriate lifespan depends on why the flag exists. Classifying a flag at creation establishes its expected lifespan and removal requirements before the original context is lost.

A release flag usually lasts from a few days to a few weeks and is used to gate an in-progress feature during development and rollout. Its cleanup urgency is high, so it should be removed once the feature has reached 100% rollout and the release has stabilized. An experiment flag typically remains active only for the duration of a test, such as an A/B experiment or a gradual, data-driven rollout. Its cleanup urgency is also high, and it should be removed as soon as the experiment concludes, regardless of the outcome. An ops or kill-switch flag, by contrast, is designed to remain in place indefinitely. It acts as a manual override for risky dependencies or emergency controls, so it does not need immediate removal. Instead, it should be reviewed periodically to confirm that it is still necessary and working as intended.

This highlights a common cleanup problem: a release or experiment flag with a short intended lifespan can receive the same caution as an ops kill switch designed to remain in place. That mismatch can turn a two-week flag into a permanent one.

Building the Staleness Detector

The checklist below recommends automating staleness detection, so it helps to show what that implementation looks like. The following example presents the core logic, simplified for readability. In practice, it receives data from the flag provider in use, such as LaunchDarkly, Unleash, or a homegrown configuration table, with each flag providing a rollout percentage, type, and the length of time its state has remained unchanged.

Two design choices here matter more than the code itself:

Thresholds are per-type, not global. A single "flag unchanged for 90 days" rule either fires constantly on ops flags that are correctly untouched, or lets release flags rot for far too long. Splitting the threshold by type — pulled directly from the classification the team should already be doing at creation — makes the report something people trust instead of something they learn to ignore.

The report identifies UNASSIGNED owners. An unowned stale flag has no person or team responsible for acting on the report, which can leave it in the codebase without a clear path to removal. Surfacing that ownership gap instead of defaulting to "team lead" makes responsibility for cleanup visible.

Wiring this into a weekly Slack post or a lightweight internal dashboard, including a spreadsheet as a starting point, turns staleness into something the team reviews on a fixed cadence rather than discovers during an unrelated investigation.

Removing a Stale Flag: A Worked Example

Detection is only half the problem. Deleting a flag safely requires a defined procedure because missed dependencies can create production issues. The sequence below shows how to remove a release flag that has remained at 100% for several weeks, using a checkout-discount flag as an example.

Confirm the resolved state, not just the current one. Check the flag's rollout history, not just its current value — a flag sitting at 100% today that was flipped back to 0% twice in the last month is not actually stable, regardless of what the staleness report says this week.
Search for every reference, not just the obvious one. A project-wide search for the flag's key (checkout_discount_v2) can reveal references beyond the if branch in the checkout service, including a log line that prints the flag's value, an analytics event property, and a conditional in a monitoring dashboard's alert query. All of these references need to be accounted for so the cleanup does not leave dead references behind.
Delete the dead branch, not just the flag check. Before removal, the code checks the flag and branches between apply_discount_v2(cart) and apply_discount_legacy(cart). After removal, it calls apply_discount_v2(cart) directly, with both the flag check and the unused branch removed. Leaving both branches as dead code "just in case" preserves unnecessary code outside the flag inventory. If apply_discount_legacy has no other callers, remove it as well.
Remove the tests for the dead branch, not just add tests for the surviving one. The legacy-path test coverage is now testing code that no longer exists in any reachable state; leaving it in the suite either silently rots (mocking a function that's been deleted) or keeps a maintenance burden alive for behavior nobody can trigger anymore.
Ship it as its own PR, reviewed as a deletion. Bundling flag removal into an unrelated feature PR can reduce the attention given to the search results from step 2. A standalone "remove checkout_discount_v2" PR keeps the review focused on deletion. Any addition in that diff warrants additional review. This procedure is often undocumented, which can make cleanup PRs feel riskier than they are.

When Cleanup Doesn't Happen: A Technical Walkthrough

The scenario below illustrates a failure pattern common enough that most teams running flags at any scale will recognize a version of it, even if the specifics differ. A team adds checkout_discount_v2 to gate a new checkout flow. The rollout succeeds within two weeks and reaches 100% — by every functional measure, the flag has done its job. But it stays in the code for over a year, because removal was never written into any ticket and nobody was assigned to come back to it.

Months later, a second flag, checkout_pricing_experiment, is added to the same checkout path for an unrelated pricing test. It runs after the discount flag: it takes whatever total the discount logic produced and applies an experimental pricing adjustment on top. Individually, each flag was tested and behaved correctly. However, apply_experimental_pricing was written and reviewed under the assumption that total came from apply_discount_legacy. By that point, checkout_discount_v2 had remained at 100% long enough that engineers working on the checkout path no longer considered it a meaningful variable, even though it remained in the code and continued to execute.

apply_discount_v2 returned a total that had already been floored to two decimal places; apply_discount_legacy had not. apply_experimental_pricing applied a percentage multiplier and then rounded. This worked with apply_discount_legacy's unrounded output but could produce an off-by-one-cent total with apply_discount_v2's pre-rounded output for a narrow set of cart values. This type of bug can escape testing when no test covers the combination of two flags operating on the same code path. The fix required a two-line change to apply rounding consistently in one place. The greater cost came from the bug reaching production and requiring someone to identify a cent-level discrepancy in reconciliation data and trace it through a code path that was not expected to contain two active flags. This is the pattern the checklist below is designed to prevent: the gradual accumulation of untracked complexity that can lead to production issues.

Build vs. Buy: Tooling Trade-offs

Teams generally use one of three approaches for flag management, and the right choice depends less on team size than on how comfortable the organization is with an external SaaS dependency in the request path.

A managed feature flag platform, such as LaunchDarkly, provides rich targeting rules, built-in audit logs and change history, and features for identifying stale flags or monitoring usage with relatively little setup. The tradeoff is recurring cost, which can increase with seats or monthly active users. It also introduces another network dependency into the request path, and while the platform may identify stale flags, someone still needs to own and act on that cleanup.

An open-source, self-hosted platform, such as Unleash, avoids per-seat licensing costs and gives teams greater control over data residency and customization. However, the engineering team becomes responsible for operating the flag service, including uptime, upgrades, maintenance, and scaling. These platforms may also provide fewer built-in insights than managed alternatives unless additional monitoring and reporting are configured.

A homegrown configuration table keeps the setup simple because it does not require introducing a separate feature flag platform. Teams can query the data directly and build custom staleness reports, such as the automated cleanup check described above. The downside is that there is usually no dedicated interface or audit trail unless those capabilities are built intentionally. Targeting logic can also spread across the codebase over time if it is not kept centralized.

The staleness detector shown earlier works with all three approaches because it requires only a list of flags with a rollout percentage and a last-modified timestamp. This keeps the implementation compatible with each approach. Managed platforms may expose this information through a report or webhook, while the other two approaches can use the script above or a similar implementation. The tooling decision affects how much the team needs to build, but each approach still requires a process for detecting stale flags and assigning cleanup work.

A Practical Checklist for Managing Flag Lifecycle

Each step below maps directly to a stage in the feature flag lifecycle. Together, they turn the lifecycle from a diagram into a process that teams can follow consistently.

1. Classify at creation. Every flag should be tagged as a release, experiment, or ops flag when it is created. This establishes the expected lifespan and removal requirements from the beginning.

2. Assign an owner and expiry. Each flag should have a named person or team responsible for it, along with a target removal date, even if that date is approximate. Without clear ownership or a deadline, a flag can remain in the codebase indefinitely without a defined cleanup path.

3. Bake removal into the ticket. Flag removal should be included in the original story's definition of done. This prevents cleanup from becoming a separate task that has to compete for priority later.

4. Automate staleness detection. Teams should run scheduled checks for flags that have remained at 0% or 100% rollout beyond their type-specific staleness threshold. This makes stale flags visible without relying on someone to remember them manually.

5. Maintain a flag inventory. A central dashboard should track every live flag, including its type, owner, and age. This gives teams a quick answer to questions such as how many flags are active and which ones may need attention.

6. Establish a recurring review cadence. Teams should review flag ownership and staleness data monthly or quarterly. Regular reviews help catch neglected flags before they accumulate into a larger cleanup backlog.

7. Prioritize removal PRs. Pull requests that remove obsolete flags should be treated as focused cleanup work. Because these changes primarily remove code rather than add new behavior, they can often have a narrower review scope while reducing unnecessary complexity in the codebase.

A note on step 2: individual ownership can become outdated when the named owner changes teams or leaves the company. Tying ownership to a service or feature area rather than a specific person provides continuity when personnel change and reduces the risk of flags becoming orphaned.

Closing Thought

Feature flags are useful, but flag creation represents only part of their lifecycle. A temporary flag without a removal plan can become a permanent addition to the codebase and increase its complexity over time.

Closing that gap requires a per-type staleness threshold, a script that checks it on a schedule, and a removal procedure that treats deletion PRs as part of planned engineering work. This approach builds removal into the same process as creation, classifies flags by type when they are created, and uses automated staleness checks to identify flags that require attention. Together, these changes make flag removal part of the same engineering process as flag creation.

Make Feature Flag Cleanup Part of the Engineering Lifecycle

Feature flags work best when removal is treated as part of the same engineering lifecycle as creation and rollout. Clear ownership, type-specific staleness checks, scheduled reviews, and focused removal PRs help teams prevent temporary controls from becoming permanent technical debt. For teams looking to strengthen these practices across deployment, automation, monitoring, and production operations, GeekyAnts’ DevOps consulting services provide support across the software delivery lifecycle.

The Bug That Doesn't Show Up in Code Review: Why Your Flutter Web App Reloads on Safari

GeekyAnts India Pvt Ltd — Mon, 28 Sep 2026 06:46:46 +0000

Demo example app*:* github.com/manuindersekhon/flutter-image-memory-demo

AI-assisted teams now ship correct code faster than ever. But this article is about a class of failure that is not in the code at all. It lives in the gap between what the code says and what the device actually does. And that gap is exactly where all the velocity we gained gets eaten back.

The Bug Report That Made No Sense

Some time back, one of our production apps had a leaderboard. A simple, scrollable list of 100+ players, each row with a name, a score, and a 50x50 profile picture. Nothing fancy.

Then the bug reports started coming in.

“The leaderboard page keeps refreshing on Safari.” “The app closed by itself on my iPhone while scrolling.” No error in the console. No crash log with a stack trace. Nothing reproducible on our development machines. The page would simply reload on Safari as if the user had pressed refresh, and on iOS the app would just disappear.

Two different platforms, two different symptoms, and as it turned out, one single bug. One that no code review could have caught.

The Code That Passed Review

Here is the code at the centre of it. This is roughly what our leaderboard row looked like:

ListTile(
  leading: CircleAvatar(
    radius: 25,
    backgroundImage: NetworkImage(user.avatarUrl),
  ),
  title: Text(user.name),
)

Take a moment and review it. It compiles. It is an idiomatic Flutter, straight from the documentation. The analyzer is happy. It renders perfectly on the MacBook and in every demo. Whether a teammate wrote it or an AI agent generated it, any of us would have approved this diff.

Nothing in the code is wrong. The defect only exists at the intersection of three layers that no code-level reviewer is looking at: what the backend serves, what the image decoder does with it, and what the device does when memory runs out.

Layer One: What the URL Actually Returns

Our users uploaded their profile pictures from their phones. A modern phone camera produces a 12-megapixel image, around 4032x3024 pixels, a few MB of JPEG. Our backend stored it exactly as uploaded, and the avatar URL served it exactly as stored.

The UI contract says “50 pixel avatar”. The API contract says “whatever the user uploaded”. There is no type system, no lint rule, and no review checklist that connects these two, and that mismatch travels silently all the way to the user's device.

Layer Two: File Size Is Not Memory Size

This is the part of image handling that is easy to forget. A JPEG is a compressed format, but a screen cannot draw compressed bytes. Before anything is rendered, the image is decoded into a raw bitmap: four bytes for every pixel.

So that 3 MB JPEG from the user's phone becomes 4032 x 3024 x 4 bytes, which is about 46.5 MB of memory. For one avatar. In a 50-pixel circle.

And here is the key Flutter behaviour: unless you tell it otherwise, Flutter decodes an image at its full intrinsic size, not at the size of the widget displaying it. The CircleAvatar's 50-pixel constraint never reaches the decoder. The full bitmap is decoded, kept, and scaled down on every frame.

To verify this, we built a small demo app that recreates the leaderboard and prints what the engine actually holds for each image. This is not theory, this is the app reporting its own image cache:

The demo leaderboard in Chrome. The bar at the bottom is the app reporting its own decoded image memory: avatar #1 is held at 4032x3024, 46.51 MB, for a 50px circle.

Multiply that by a leaderboard. Scrolling through 100+ entries asks the engine for several gigabytes of decoded bitmaps. Flutter's built-in image cache has a 100 MB budget, but that budget only applies to images that are no longer on screen. Images that are currently visible, or kept alive by the layout, are held regardless of it. The cap we were all silently relying on was never going to save us.

Layer Three: Every Platform Has a Ceiling, and They Are All Different

Here is where the story splits by platform, and where “works on my machine” stops being a joke and becomes the actual mechanism of the bug.

Chrome survives this abuse. It decodes images through a dedicated browser API into memory it can discard and re-decode at will. When we scrolled our naive leaderboard in Chrome, the tab stayed around a few hundred MB and nothing bad happened. This is exactly why the bug never appeared on our development machines.

Safari has no such decode path. The decoded bitmaps land inside the tab's own web content process, and WebKit enforces a hard memory budget on that process. On the same page, same scroll, Safari's process climbed to four times what Chrome used. Push further, and WebKit does exactly what its source code says it will do: it kills the web content process and reloads the page.

In our stress test, the process crossed 7 GB and was terminated, and the page came back fresh, exactly the “random refresh” our users reported. Safari even tells the user politely: “This webpage was reloaded because it was using significant memory.”

There is also a subtler symptom on iPads and iPhones. Before killing the page, WebKit fights back by purging decoded images. We watched the iOS Safari process balloon, get purged, and keep running with the leaderboard text intact but every avatar blank:

iOS Safari after WebKit's memory purge: the page survives, but the avatars are silently gone.

And the native iOS app? Same bug, different executioner. iOS enforces a per-app memory budget through a system called Jetsam. The budget depends on the device, community measurements put it under 1 GB on older iPhones and around 2 GB on mid-range ones. A leaderboard holding a few dozen 46 MB bitmaps alive walks into that limit within a couple of screen-heights of scrolling. The app is killed by the OS, not by your code. There is no Dart exception and no useful stack trace, and your crash reporting tool shows an “out of memory session” at best. That was our mysterious iOS crash.

One bug. Three ceilings. Three completely different symptoms.

Why Review Never Had a Chance

Now go back to that CircleAvatar snippet and ask: where in the diff is this bug?

It is not there. The reviewer sees idiomatic widget code. The tests see an image that renders. CI is green. The 46.5 MB number exists only at runtime, on a real device, with real production images. The defect is spread across three systems whose owners never appear in the same pull request: the upload pipeline that stores 12-megapixel photos, the framework default that decodes at intrinsic size, and the platform policy that kills the process.

This is also why the AI angle matters to us. An agent will write you this exact code, and it will be right by every static standard. An AI reviewer will approve it for the same reason a human does: the evidence is simply not in the artifact being reviewed. As AI compresses the cost of writing code, the defects that survive migrate to the layers that neither the agent nor the reviewer can see. The cost does not disappear. It moves to a production incident three weeks later, on a device you do not own.

The Fix Is One Parameter (and a Better One Upstream)

The immediate fix is embarrassingly small. Flutter lets you tell the decoder what size you actually need:

Image.network(
  url,
  width: 50,
  height: 50,
  fit: BoxFit.cover,
  cacheWidth: (50 * MediaQuery.devicePixelRatioOf(context)).round(),
)

// CircleAvatar has no sizing hook, so wrap the provider:
CircleAvatar(
  backgroundImage: ResizeImage(
    NetworkImage(url),
    width: 150,
  ),
)

With cacheWidth set, the same avatar decodes at 150 pixels instead of 4032. In our demo, that took each image from 46.51 MB down to 0.06 MB, roughly 775 times less memory, with zero visible difference in a 50-pixel circle. The whole leaderboard now fits in less than one megabyte of decoded images:

The fixed build: same leaderboard, same photos, 0.8 MB of decoded images in total. The probe shows avatar #1 now decodes at 150x112, 0.06 MB.

But honestly, the client-side parameter is the band-aid. The real fix is to never ship a 12-megapixel file to a 50-pixel widget in the first place: serve resized images from your CDN or an image proxy. That also fixes what cacheWidth cannot, because the full-size download and the browser-side decode of the original file still happen either way. Fix it at the source and every client, including the ones you have not written yet, gets it for free.

Making Sure It Never Comes Back

A bug that review cannot catch needs guardrails that do not depend on review.

The one we wish we had turned on earlier is built into Flutter itself. Set debugInvertOversizedImages to true in your debug builds, and the framework will flip and invert the colours of any image that was decoded significantly larger than its display size, and log the wasted bytes. Our leaderboard lit up like this:

debugInvertOversizedImages in action: the framework flags every avatar that was decoded far larger than the size it is displayed at.

Beyond that flag, a few habits close the loop:

Wrap remote images in one shared widget (an AppAvatar of your own) that requires a decode size, so the naive version cannot be written casually.
Test on the smallest-RAM device you actually support, with production images, not neat little asset placeholders.
When a “random refresh on Safari” or an “app just closed” report comes in, put memory on the suspect list before routing and state management.

Final Thoughts

The most interesting thing about this bug is how ordinary the code was. No clever trick went wrong, no obscure API was misused. A default did exactly what it was documented to do, on inputs nobody in the pull request could see, on devices with limits nobody in the pull request was thinking about.

AI has made the code layer cheap. It has not made the other layers cheap. The asset pipeline, the decoder, the memory ceilings of a five-year-old iPhone, someone on the team still has to own those. The teams that move fastest with AI will not be the ones that generate the most code. They will be the ones who know exactly which questions the diff cannot answer, and go looking for the evidence themselves.

The demo app used for every number and screenshot in this article is open source, and reproduces the whole story, including the Safari reload: github.com/manuindersekhon/flutter-image-memory-demo

What a PHP-to-NestJS Banking Migration Taught Us About Architecture, Security, and Trust

GeekyAnts India Pvt Ltd — Mon, 28 Sep 2026 06:12:26 +0000

Every digital banking platform eventually hits the same wall: the monolith that shipped a decade ago still works, still processes real money for real account holders every day, and still cannot be touched without fear. Ours was a PHP application built with Laravel, serving multiple financial institutions from a single, tightly coupled codebase. It had grown feature by feature, bank by bank, for years. The rewrite wasn’t optional. Every new institution we onboarded meant another branch of conditional logic, another set of hacks specific to one bank bolted onto shared controllers, and another opportunity for one institution’s change to break another’s production environment.

This article covers the architecture decisions we made while rewriting that platform as a multitenant system driven by configuration, and what the migration process taught us about verifying legacy behavior instead of assuming it.

The core decision: separate “what the UI looks like” from “what the business logic does”

The biggest architectural decision was splitting the platform into three layers instead of one: a thin frontend driven by configuration; a backend for frontend (BFF) that owns all business logic, session state, and vendor orchestration; and a config service that serves server-driven UI (SDUI) definitions per institution.

In the legacy system, “multitenant” meant if ($bank === 'X') scattered across views and controllers. Every institution’s quirks lived inside the same PHP files as everyone else’s, which meant every deployment carried the blast radius of every tenant at once. The new architecture inverts that. The mobile and web clients render forms, layouts, and copy from JSON served by the config service, merged through three layers: platform defaults, core banking provider defaults, then tenant-specific overrides. A new institution’s branding, field visibility, or feature flags become a config change, not a code change. The frontend contains no logic specific to an institution. It renders what the configuration defines.

That decision to push all business logic into a backend service and make the frontend a renderer let us migrate in stages. We could stand up new BFF endpoints one feature at a time behind the same configuration-driven client without a single cutover event. Legacy and new systems ran side by side for months, split by institution and feature, and nobody outside the engineering team could tell which backend was serving a given screen.

The security decision: stop letting the client hold the keys

The second major decision came from an audit of the tokens flowing between the client and the backend. Like many systems that grow over time, the legacy platform’s API responses included internal identifiers and session tokens issued by the core banking system for a given member or account. These appeared in payloads that the client held and replayed on subsequent calls. The approach worked, but it meant an artifact sitting on the client carried a token minted by the core banking system itself, and every downstream service had to trust whatever the client sent back.

We restructured this so the BFF never lets a core banking session token leave the server. On initial resolution, the BFF holds the real token on the server, keyed by its own session identifier, and the client sees only that opaque session UUID. Every subsequent request resolves the real token from server-side session state rather than trusting anything replayed from the client.

The mechanical part of this change was straightforward. The challenge was scope: identifying every place where a value shaped like a real token was flowing, not just fields with an obvious name such as memberToken. We found aliased fields carrying the same sensitive value under a different property name. A search based on field names missed them because the value mattered, not the label attached to it.

We also found flows that broke a simpler mental model. A couple of endpoints cached resolved session context across two separate HTTP requests: an MFA challenge and its resumption. A migration that always resolved the token from server-side session state would have broken authentication resumption the next time that challenge fired. Unit tests would not have exposed the failure because they typically mocked the layer where the gap lived.

The fix in those cases was to carry an explicit internal object, never exposed to the client, through the cached challenge state and reseed server-side session context from it before resuming. This was a deliberate design decision rather than a mechanical rename, and one worth flagging for review rather than treating as a routine migration change.

The performance decision: let Redis carry the weight of session state

Moving the real token off the client solved the trust problem, but it introduced a new one. The BFF now had to resolve that token, the member’s profile, and account-level context on every request that needed them. Calling the core banking system each time would have made the new platform slower than the one it was replacing. The third decision was to cache resolved session state rather than fetch it on every request.

Once a session is established, the BFF writes the real token and the resolved member and account context into Redis under the opaque session identifier, with a time to live tied to the permitted session duration. Every later request in that session reads from Redis instead of paying the round-trip cost of a call to a core banking system that, in most deployments, is the slowest and most rate-limited dependency in the request path. What used to require a network call to an external system on every screen became an in-memory lookup, reducing fetch times for requests that depended on member or account context.

The cache keys are never a flat string. Each one is built from three parts: what is being cached, which tenant it belongs to, and an optional subkey for the specific record. This structure prevents a cached profile for one institution from colliding with an identically shaped record for another. It also means an entire tenant’s cache can be cleared or inspected without touching anyone else’s data, which matters when a single Redis instance serves every institution on the platform.

The same pattern caches the merged three-layer configuration for a tenant because recomputing that merge on every request would reduce the latency gains Redis provides. The result for account holders was fewer round trips per screen and lower dependence on fresh core banking lookups for each action.

The methodology lesson: documentation is something you build, not something you inherit

One constraint shaped the migration process: there was no migration specification handed down from the old system. Nobody had written a document describing what the legacy application did feature by feature. The old codebase itself was the only complete record of the business rules. Institutional memory had faded, the original authors were gone, and the Blade templates and controllers were the closest thing to a specification that existed.

The practice became clear: for every feature we migrated, we read the legacy implementation rather than relying on a summary and documented what we learned as we went. That growing set of documents became the migration’s specification, built one feature at a time instead of inherited on day one. The documentation became valuable, but we learned that it was not equivalent to the source itself.

During one debugging session, three separate “bug fixes” based on our migration notes turned out to be wrong once checked against the legacy source. Each one moved behavior away from what the old system did in production rather than back toward it. In each case, the notes had conflated two separate legacy checks or described a distinction that sounded reasonable but that the original code didn’t make.

The documentation was written in good faith by someone summarizing a page of PHP in a sentence, but the summary lost a detail that mattered.

The corrective habit was clear but easy to skip under time pressure: treat every internal document, including the ones we wrote ourselves, as a working hypothesis about the legacy system rather than a fact about it. Before relying on a written description of “what the old system does” to justify a fix, reread the legacy implementation. Plausibility is not evidence. A behavior that “seems right” for a banking flow is worth only as much as the source code confirms.

Every time the source and documentation disagreed, we corrected the documentation, so the record improved as the migration progressed. It never replaced the source as the final authority. The principle also works in the other direction. Not every divergence from legacy behavior is a bug. Some are intentional improvements that have received explicit approval. The discipline lies in knowing which is which before acting rather than assuming either case.

What carried over

None of these decisions were exotic. UI driven by configuration, a BFF that owns business logic, and server-side token indirection are well-established patterns. What made them work in a live, regulated, multitenant banking system was discipline in the implementation details: treating the legacy source, not a document about it, as the final authority; scoping security fixes by the value flowing through a field rather than its name; and building the migration documentation alongside the code instead of waiting for a finished specification that was never coming.

The architecture gave us the seams to migrate safely. The process discipline kept the migration honest.

Building a Safer Path for Banking Modernization

Legacy banking systems can be modernized without forcing a single high-risk replacement. Phased migration, clear service boundaries, secure session management, and careful verification of existing behavior can help teams modernize while protecting critical banking operations. For teams working through similar architecture challenges, explore our core banking modernization services.

Why Legacy Systems Make Business Growth More Expensive: Navigate A Smarter Path to Legacy Modernization

GeekyAnts India Pvt Ltd — Wed, 23 Sep 2026 11:53:55 +0000

Learn how legacy systems make business growth more expensive and how edge-first modernization can remove constraints without replacing the existing system.

Six Brands, Six Rebuilds, Until We Changed the Equation

A North American restaurant group came to GeekyAnts with six casual-dining brands and a digital estate. The system encountered challenges every time it needed to change as per the business requirements.

There was an aging JSP web application, a separate native mobile app for each brand, and separate build and release pipelines for each of the systems. Each brand also had its own web and mobile development teams. A change that should have been made once had to be implemented six times.

Adding another brand took two to three months, and most of that time was going into creating another app, another pipeline, and duplicated work across the web and mobile teams. The core systems were not the issue since some of the routine tasks like ordering, menus, pricing, and availability were already working in a functional manner.

The cost was actually spent in the existing digital layer around the system, where every additional brand meant another application, another pipeline, and more duplicated work. We started with a two-page proof of concept. The question that needed to be answered was if the same components could genuinely be shared between web and native mobile. Once this assumption was tested, it gave us the basis to replace the edge while keeping the core systems in place.

Bringing Changes to One Digital Platform Instead of Adding Six New Rebuilds

We focused on the layer where the duplication was happening and built one platform for web and mobile, so that the changes could be made once instead of across six separate brand applications. A shared component library removed repeated development work, while a single mobile app shell let the team configure and roll out new brands instead of building another app from scratch. Over-the-air updates also meant changes could reach users without repeating the full app release cycle for every brand.

The first brand went live in 6-8 months. After that, a new brand was onboarded in 2-3 weeks instead of 2-3 months. The per-brand teams were consolidated into one web team and one mobile team, freeing them from rebuilding the same screen and flow.

The seventh brand was added after the platform existed, without recreating the old setup. The goal was to remove the multiplication that came with every new brand.

When Does Legacy Infrastructure Become a Business Problem?

Aging infrastructure alone is not the best reason to replace a system. In actuality, business constraints are one of the crucial reasons why legacy infrastructure needs to be replaced.

McKinsey estimates that technical debt can account for 20-40% of the value of an organization’s technology estate. So when a system becomes a problem, the business starts paying for its limitations. A release that once took two weeks can stretch into a quarter. Adding a new brand, region, or product can mean adding another technology stack. An integration task that should be a simple process becomes a whole project if the system doesn’t connect with it the right way.

The same pattern shows up in other business operations where a compliance change that should take weeks takes months, engineering teams spend more time keeping duplicated experiences running than building the next product, and each new unit of growth costs more as the business has to repeat the work it has already done.

The better question ideally would be “What is this system preventing the business from doing, and what is that constraint costing us?” because it changes the modernization conversation from replacing old technology to removing the things that hold business growth back.

Why Did We Modernize Edge Before the Core System?

When a core system starts holding the business back, the first instinct would be to replace it, which may be a risky process.

Most of the business processes may not be documented in one place, as some may exist in code, some in old integrations, and some may only be known by the people who have worked with the system for years.

The edge is different. It is where customers, employees, and partners interact with the business: web and mobile experiences, APIs, integrations, workflows, automation, and deployment pipelines.

Modernizing the edge gives the businesses enough room to be adaptive to changes without immediately disrupting the logic on which the systems run. You can improve how customers place an order, connect a new channel, or streamline an employee workflow without interrupting the system that handles ordering or pricing tasks.

The risk, however, changes as you move closer to the core system. The more business logic and dependencies you have to uncover, the more you risk finding rules nobody remembered were there.

Needless to say, modernization risk increases with the amount of business logic you have to rediscover.

Seven Decisions To Be Considered for Modernization of Legacy Systems

Every part of the legacy system can have a different path forward. Depending on what it does and where it creates friction, you can:

Retire the system when nothing depends on it anymore.
Retain when it works and creates no business constraint.
Expose when the capability is useful but trapped behind poor interfaces.
Augment when the core works but the surrounding experience does not.
Re-platform when the application works but the underlying runtime has become the problem.
Refactor when the business logic is worth keeping but the architecture prevents change or scale.
Replace when the economics no longer make sense or the system's business logic no longer matches how the company operates.

Modernization Should Create Room for Business Growth

The best place to begin the modernization process often starts from navigating which business process is actually hindering its growth. The real opportunity in legacy modernization lies in making room for business’s growth without creating unnecessary risks.

Planning a legacy modernization initiative? Talk to GeekyAnts about identifying what is actually constraining growth and modernizing it without replacing the existing system.